Commit 530df3c2 authored by Sean McGivern's avatar Sean McGivern

Merge branch 'jv-enable-workhorse-sidechannel-2' into 'master'

Enable workhorse_use_sidechannel by default

See merge request gitlab-org/gitlab!73536
parents 23b10a21 3c9766d6
......@@ -5,4 +5,4 @@ rollout_issue_url: https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1
milestone: '14.4'
type: development
group: 'group::scalability'
default_enabled: false
default_enabled: true
......@@ -250,12 +250,15 @@ concurrent = 4
This makes the cloning configuration to be part of the given runner
and does not require us to update each `.gitlab-ci.yml`.
## Pre-clone step
> [An issue exists](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/463) to remove the need for this optimization.
For very active repositories with a large number of references and files, you can also
optimize your CI jobs by seeding repository data with GitLab Runner's [`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section).
See [our development documentation](../../development/pipelines.md#pre-clone-step) for
an overview of how we implemented this approach on GitLab.com for the main GitLab repository.
## Git fetch caching or pre-clone step
For very active repositories with a large number of references and files, you can either (or both):
- Consider using the [Gitaly pack-objects cache](../../administration/gitaly/configure_gitaly.md#pack-objects-cache) instead of a
pre-clone step. This is easier to set up and it benefits all repositories on your GitLab server, unlike the pre-clone step that
must be configured per-repository. The pack-objects cache also automatically works for forks. For `gitlab-org/gitlab` development
on GitLab.com, we stopped using a pre-clone step.
- Optimize your CI/CD jobs by seeding repository data in a pre-clone step with the
[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) of GitLab Runner. See our
[development documentation](../../development/pipelines.md#pre-clone-step) for an overview of how we used to implement this approach on
GitLab.com for the main GitLab repository.
......@@ -793,19 +793,30 @@ request, be sure to start the `dont-interrupt-me` job before pushing.
We limit the artifacts that are saved and retrieved by jobs to the minimum in order to reduce the upload/download time and costs, as well as the artifacts storage.
### Pre-clone step
### Git fetch caching
Because GitLab.com uses the [pack-objects cache](../administration/gitaly/configure_gitaly.md#pack-objects-cache),
concurrent Git fetches of the same pipeline ref are deduplicated on
the Gitaly server (always) and served from cache (when available).
This works well for the following reasons:
The `gitlab-org/gitlab` project on GitLab.com uses a [pre-clone step](https://gitlab.com/gitlab-org/gitlab/-/issues/39134)
to seed the project with a recent archive of the repository. This is done for
several reasons:
- The pack-objects cache is enabled on all Gitaly servers on GitLab.com.
- The CI/CD [Git strategy setting](../ci/pipelines/settings.md#choose-the-default-git-strategy) for `gitlab-org/gitlab` is **Git clone**,
causing all jobs to fetch the same data, which maximizes the cache hit ratio.
- We use [shallow clone](../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) to avoid downloading the full Git
history for every job.
### Pre-clone step
- It speeds up builds because a 800 MB download only takes seconds, as opposed to a full Git clone.
- It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`.
NOTE:
We no longer use this optimization for `gitlab-org/gitlab` because the [pack-objects cache](../administration/gitaly/configure_gitaly.md#pack-objects-cache)
allows Gitaly to serve the full CI/CD fetch traffic now. See [Git fetch caching](#git-fetch-caching).
The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable
[defined by GitLab.com shared runners](../ci/runners/build_cloud/linux_build_cloud.md#pre-clone-script).
The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD variable:
The `CI_PRE_CLONE_SCRIPT` is defined as a project CI/CD variable:
```shell
(
......
......@@ -305,10 +305,14 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap
### 14.5.0
When `make` is run, Gitaly builds are now created in `_build/bin` and no longer in the root directory of the source directory. If you
- When `make` is run, Gitaly builds are now created in `_build/bin` and no longer in the root directory of the source directory. If you
are using a source install, update paths to these binaries in your [systemd unit files](upgrading_from_source.md#configure-systemd-units)
or [init scripts](upgrading_from_source.md#configure-sysv-init-script) by [following the documentation](upgrading_from_source.md).
- Connections between Workhorse and Gitaly use the Gitaly `backchannel` protocol by default. If you deployed a gRPC proxy between Workhorse and Gitaly,
Workhorse can no longer connect. As a workaround, [disable the temporary `workhorse_use_sidechannel`](../administration/feature_flags.md#enable-or-disable-the-feature)
feature flag. If you need a proxy between Workhorse and Gitaly, use a TCP proxy.
### 14.4.0
Git 2.33.x and later is required. We recommend you use the
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment