Commit b5c5b319 authored by Kushal Pandya's avatar Kushal Pandya

Merge branch 'selhorn-runner-reorg-1' into 'master'

Reorganized runner content to bring SaaS shared runners to main page

See merge request gitlab-org/gitlab!63572
parents 503533a3 1bb271df
...@@ -5,16 +5,16 @@ ...@@ -5,16 +5,16 @@
= s_('Runners|This runner is available to all groups and projects in your GitLab instance.') = s_('Runners|This runner is available to all groups and projects in your GitLab instance.')
.gl-alert-body .gl-alert-body
= s_('Runners|Shared runners are available to every project in a GitLab instance. If you want a runner to build only specific projects, restrict the project in the table below. After you restrict a runner to a project, you cannot change it back to a shared runner.') = s_('Runners|Shared runners are available to every project in a GitLab instance. If you want a runner to build only specific projects, restrict the project in the table below. After you restrict a runner to a project, you cannot change it back to a shared runner.')
= link_to _('Learn more.'), help_page_path('ci/runners/README', anchor: 'shared-runners'), target: '_blank', rel: 'noopener noreferrer' = link_to _('Learn more.'), help_page_path('ci/runners/runners_scope', anchor: 'shared-runners'), target: '_blank', rel: 'noopener noreferrer'
- elsif runner.group_type? - elsif runner.group_type?
%h4.gl-alert-title %h4.gl-alert-title
= s_('Runners|This runner is available to all projects and subgroups in a group.') = s_('Runners|This runner is available to all projects and subgroups in a group.')
.gl-alert-body .gl-alert-body
= s_('Runners|Use Group runners when you want all projects in a group to have access to a set of runners.') = s_('Runners|Use Group runners when you want all projects in a group to have access to a set of runners.')
= link_to _('Learn more.'), help_page_path('ci/runners/README', anchor: 'group-runners'), target: '_blank', rel: 'noopener noreferrer' = link_to _('Learn more.'), help_page_path('ci/runners/runners_scope', anchor: 'group-runners'), target: '_blank', rel: 'noopener noreferrer'
- else - else
%h4.gl-alert-title %h4.gl-alert-title
= s_('Runners|This runner is associated with specific projects.') = s_('Runners|This runner is associated with specific projects.')
.gl-alert-body .gl-alert-body
= s_('Runners|You can set up a specific runner to be used by multiple projects but you cannot make this a shared runner.') = s_('Runners|You can set up a specific runner to be used by multiple projects but you cannot make this a shared runner.')
= link_to _('Learn more.'), help_page_path('ci/runners/README', anchor: 'specific-runners'), target: '_blank', rel: 'noopener noreferrer' = link_to _('Learn more.'), help_page_path('ci/runners/runners_scope', anchor: 'specific-runners'), target: '_blank', rel: 'noopener noreferrer'
...@@ -172,7 +172,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ...@@ -172,7 +172,7 @@ Learn how to install, configure, update, and maintain your GitLab instance.
- [External Pipeline Validation](external_pipeline_validation.md): Enable, disable, and configure external pipeline validation. - [External Pipeline Validation](external_pipeline_validation.md): Enable, disable, and configure external pipeline validation.
- [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully).
- [Job logs](job_logs.md): Information about the job logs. - [Job logs](job_logs.md): Information about the job logs.
- [Register runners](../ci/runners/README.md#types-of-runners): Learn how to register and configure runners. - [Register runners](../ci/runners/runners_scope.md): Learn how to register and configure runners.
- [Shared runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota): Limit the usage of pipeline minutes for shared runners. - [Shared runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota): Limit the usage of pipeline minutes for shared runners.
- [Enable/disable Auto DevOps](../topics/autodevops/index.md#enable-or-disable-auto-devops): Enable or disable Auto DevOps for your instance. - [Enable/disable Auto DevOps](../topics/autodevops/index.md#enable-or-disable-auto-devops): Enable or disable Auto DevOps for your instance.
......
...@@ -43,9 +43,9 @@ can't link to files outside it. ...@@ -43,9 +43,9 @@ can't link to files outside it.
To ensure maximum availability of the cache, when you declare `cache` in your jobs, To ensure maximum availability of the cache, when you declare `cache` in your jobs,
use one or more of the following: use one or more of the following:
- [Tag your runners](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) and use the tag on jobs - [Tag your runners](../runners/configure_runners.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) and use the tag on jobs
that share their cache. that share their cache.
- [Use runners that are only available to a particular project](../runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects). - [Use runners that are only available to a particular project](../runners/runners_scope.md#prevent-a-specific-runner-from-being-enabled-for-other-projects).
- [Use a `key`](../yaml/README.md#cachekey) that fits your workflow (for example, - [Use a `key`](../yaml/README.md#cachekey) that fits your workflow (for example,
different caches on each branch). For that, you can take advantage of the different caches on each branch). For that, you can take advantage of the
[predefined CI/CD variables](../variables/README.md#predefined-cicd-variables). [predefined CI/CD variables](../variables/README.md#predefined-cicd-variables).
......
...@@ -25,9 +25,8 @@ To enable Docker commands for your CI/CD jobs, you can use: ...@@ -25,9 +25,8 @@ To enable Docker commands for your CI/CD jobs, you can use:
If you don't want to execute a runner in privileged mode, If you don't want to execute a runner in privileged mode,
but want to use `docker build`, you can also [use kaniko](using_kaniko.md). but want to use `docker build`, you can also [use kaniko](using_kaniko.md).
If you are using shared runners on GitLab.com, see If you are using shared runners on GitLab.com,
[GitLab.com shared runners](../../user/gitlab_com/index.md#shared-runners) [learn more about how these runners are configured](../runners/README.md).
to learn more about how these runners are configured.
### Use the shell executor ### Use the shell executor
...@@ -91,7 +90,7 @@ The Docker image has all of the `docker` tools installed and can run ...@@ -91,7 +90,7 @@ The Docker image has all of the `docker` tools installed and can run
the job script in context of the image in privileged mode. the job script in context of the image in privileged mode.
We recommend you use [Docker-in-Docker with TLS enabled](#docker-in-docker-with-tls-enabled), We recommend you use [Docker-in-Docker with TLS enabled](#docker-in-docker-with-tls-enabled),
which is supported by [GitLab.com shared runners](../../user/gitlab_com/index.md#shared-runners). which is supported by [GitLab.com shared runners](../runners/README.md).
You should always specify a specific version of the image, like `docker:19.03.12`. You should always specify a specific version of the image, like `docker:19.03.12`.
If you use a tag like `docker:stable`, you have no control over which version is used. If you use a tag like `docker:stable`, you have no control over which version is used.
......
...@@ -133,7 +133,7 @@ The [Least Privilege Container Builds with Kaniko on GitLab](https://www.youtube ...@@ -133,7 +133,7 @@ The [Least Privilege Container Builds with Kaniko on GitLab](https://www.youtube
video is a walkthrough of the [Kaniko Docker Build](https://gitlab.com/guided-explorations/containers/kaniko-docker-build) video is a walkthrough of the [Kaniko Docker Build](https://gitlab.com/guided-explorations/containers/kaniko-docker-build)
Guided Exploration project pipeline. It was tested on: Guided Exploration project pipeline. It was tested on:
- [GitLab.com shared runners](../../user/gitlab_com/index.md#shared-runners) - [GitLab.com shared runners](../runners/README.md)
- [The Kubernetes runner executor](https://docs.gitlab.com/runner/executors/kubernetes.html) - [The Kubernetes runner executor](https://docs.gitlab.com/runner/executors/kubernetes.html)
The example can be copied to your own group or instance for testing. More details The example can be copied to your own group or instance for testing. More details
......
...@@ -117,7 +117,7 @@ The other pipelines don't get the protected variable. You can also ...@@ -117,7 +117,7 @@ The other pipelines don't get the protected variable. You can also
[scope variables to specific environments](../variables/where_variables_can_be_used.md#variables-with-an-environment-scope). [scope variables to specific environments](../variables/where_variables_can_be_used.md#variables-with-an-environment-scope).
We recommend that you use protected variables on protected environments to make sure that the We recommend that you use protected variables on protected environments to make sure that the
secrets aren't exposed unintentionally. You can also define production secrets on the secrets aren't exposed unintentionally. You can also define production secrets on the
[runner side](../runners/README.md#prevent-runners-from-revealing-sensitive-information). [runner side](../runners/configure_runners.md#prevent-runners-from-revealing-sensitive-information).
This prevents other maintainers from reading the secrets and makes sure that the runner only runs on This prevents other maintainers from reading the secrets and makes sure that the runner only runs on
protected branches. protected branches.
......
...@@ -409,7 +409,7 @@ The job with [`action: stop` might not run](#the-job-with-action-stop-doesnt-run ...@@ -409,7 +409,7 @@ The job with [`action: stop` might not run](#the-job-with-action-stop-doesnt-run
if it's in a later stage than the job that started the environment. if it's in a later stage than the job that started the environment.
If you can't use [pipelines for merge requests](../merge_request_pipelines/index.md), If you can't use [pipelines for merge requests](../merge_request_pipelines/index.md),
set the [`GIT_STRATEGY`](../runners/README.md#git-strategy) to `none` in the set the [`GIT_STRATEGY`](../runners/configure_runners.md#git-strategy) to `none` in the
`stop_review` job. Then the [runner](https://docs.gitlab.com/runner/) doesn't `stop_review` job. Then the [runner](https://docs.gitlab.com/runner/) doesn't
try to check out the code after the branch is deleted. try to check out the code after the branch is deleted.
......
...@@ -215,7 +215,7 @@ configured: ...@@ -215,7 +215,7 @@ configured:
(or above) to the top-level group. They can maintain CI/CD configurations for (or above) to the top-level group. They can maintain CI/CD configurations for
the higher environments (such as production) in the group-level settings page, the higher environments (such as production) in the group-level settings page,
wnich includes group-level protected environments, wnich includes group-level protected environments,
[group-level runners](../runners/README.md#group-runners), [group-level runners](../runners/runners_scope.md#group-runners),
[group-level clusters](../../user/group/clusters/index.md), etc. Those [group-level clusters](../../user/group/clusters/index.md), etc. Those
configurations are inherited to the child projects as read-only entries. configurations are inherited to the child projects as read-only entries.
This ensures that only operators can configure the organization-wide This ensures that only operators can configure the organization-wide
......
...@@ -53,7 +53,7 @@ To make submodules work correctly in CI/CD jobs: ...@@ -53,7 +53,7 @@ To make submodules work correctly in CI/CD jobs:
1. Make sure you use [relative URLs](#configure-the-gitmodules-file) 1. Make sure you use [relative URLs](#configure-the-gitmodules-file)
for submodules located in the same GitLab server. for submodules located in the same GitLab server.
1. You can set the `GIT_SUBMODULE_STRATEGY` variable to either `normal` or `recursive` 1. You can set the `GIT_SUBMODULE_STRATEGY` variable to either `normal` or `recursive`
to tell the runner to [fetch your submodules before the job](runners/README.md#git-submodule-strategy): to tell the runner to [fetch your submodules before the job](runners/configure_runners.md#git-submodule-strategy):
```yaml ```yaml
variables: variables:
......
...@@ -16,7 +16,7 @@ is deployed, some [security precautions](../../administration/integration/termin ...@@ -16,7 +16,7 @@ is deployed, some [security precautions](../../administration/integration/termin
taken to protect the users. taken to protect the users.
NOTE: NOTE:
[Shared runners on GitLab.com](../runners/README.md#shared-runners) do not [Shared runners on GitLab.com](../runners/README.md) do not
provide an interactive web terminal. Follow [this provide an interactive web terminal. Follow [this
issue](https://gitlab.com/gitlab-org/gitlab/-/issues/24674) for progress on issue](https://gitlab.com/gitlab-org/gitlab/-/issues/24674) for progress on
adding support. For groups and projects hosted on GitLab.com, interactive web adding support. For groups and projects hosted on GitLab.com, interactive web
......
...@@ -56,7 +56,7 @@ test: ...@@ -56,7 +56,7 @@ test:
> Introduced in GitLab Runner 8.9. > Introduced in GitLab Runner 8.9.
By default, GitLab is configured to use the [`fetch` Git strategy](../runners/README.md#git-strategy), By default, GitLab is configured to use the [`fetch` Git strategy](../runners/configure_runners.md#git-strategy),
which is recommended for large repositories. which is recommended for large repositories.
This strategy reduces the amount of data to transfer and This strategy reduces the amount of data to transfer and
does not really impact the operations that you might do on a repository from CI. does not really impact the operations that you might do on a repository from CI.
...@@ -65,7 +65,7 @@ does not really impact the operations that you might do on a repository from CI. ...@@ -65,7 +65,7 @@ does not really impact the operations that you might do on a repository from CI.
> Introduced in GitLab Runner 11.10. > Introduced in GitLab Runner 11.10.
[`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) allows you to [`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) allows you to
control where you clone your sources. This can have implications if you control where you clone your sources. This can have implications if you
heavily use big repositories with fork workflow. heavily use big repositories with fork workflow.
...@@ -77,7 +77,7 @@ In such cases, ideally you want to make the GitLab Runner executor be used only ...@@ -77,7 +77,7 @@ In such cases, ideally you want to make the GitLab Runner executor be used only
for the given project and not shared across different projects to make this for the given project and not shared across different projects to make this
process more efficient. process more efficient.
The [`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) has to be The [`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) has to be
within the `$CI_BUILDS_DIR`. Currently, it is impossible to pick any path within the `$CI_BUILDS_DIR`. Currently, it is impossible to pick any path
from disk. from disk.
...@@ -85,12 +85,12 @@ from disk. ...@@ -85,12 +85,12 @@ from disk.
> Introduced in GitLab Runner 11.10. > Introduced in GitLab Runner 11.10.
[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags) allows you to control [`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) allows you to control
whether or not you require the `git clean` command to be executed for each CI whether or not you require the `git clean` command to be executed for each CI
job. By default, GitLab ensures that you have your worktree on the given SHA, job. By default, GitLab ensures that you have your worktree on the given SHA,
and that your repository is clean. and that your repository is clean.
[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags) is disabled when set [`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) is disabled when set
to `none`. On very big repositories, this might be desired because `git to `none`. On very big repositories, this might be desired because `git
clean` is disk I/O intensive. Controlling that with `GIT_CLEAN_FLAGS: -ffdx clean` is disk I/O intensive. Controlling that with `GIT_CLEAN_FLAGS: -ffdx
-e .build/` (for example) allows you to control and disable removal of some -e .build/` (for example) allows you to control and disable removal of some
...@@ -99,7 +99,7 @@ the incremental builds. This has the biggest effect if you re-use existing ...@@ -99,7 +99,7 @@ the incremental builds. This has the biggest effect if you re-use existing
machines and have an existing worktree that you can re-use for builds. machines and have an existing worktree that you can re-use for builds.
For exact parameters accepted by For exact parameters accepted by
[`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags), see the documentation [`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags), see the documentation
for [`git clean`](https://git-scm.com/docs/git-clean). The available parameters for [`git clean`](https://git-scm.com/docs/git-clean). The available parameters
are dependent on Git version. are dependent on Git version.
...@@ -107,7 +107,7 @@ are dependent on Git version. ...@@ -107,7 +107,7 @@ are dependent on Git version.
> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1. > [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1.
[`GIT_FETCH_EXTRA_FLAGS`](../runners/README.md#git-fetch-extra-flags) allows you [`GIT_FETCH_EXTRA_FLAGS`](../runners/configure_runners.md#git-fetch-extra-flags) allows you
to modify `git fetch` behavior by passing extra flags. to modify `git fetch` behavior by passing extra flags.
For example, if your project contains a large number of tags that your CI jobs don't rely on, For example, if your project contains a large number of tags that your CI jobs don't rely on,
...@@ -119,7 +119,7 @@ tags, `--no-tags` can [make a big difference in some ...@@ -119,7 +119,7 @@ tags, `--no-tags` can [make a big difference in some
cases](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/746). cases](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/746).
If your CI builds do not depend on Git tags it is worth trying. If your CI builds do not depend on Git tags it is worth trying.
See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../runners/README.md#git-fetch-extra-flags) See the [`GIT_FETCH_EXTRA_FLAGS` documentation](../runners/configure_runners.md#git-fetch-extra-flags)
for more information. for more information.
## Fork-based workflow ## Fork-based workflow
......
...@@ -128,16 +128,16 @@ agents you were using. ...@@ -128,16 +128,16 @@ agents you were using.
There are some important differences in the way runners work in comparison to agents: There are some important differences in the way runners work in comparison to agents:
- Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/README.md#types-of-runners). - Runners can be set up as [shared across an instance, be added at the group level, or set up at the project level](../runners/runners_scope.md).
They self-select jobs from the scopes you've defined automatically. They self-select jobs from the scopes you've defined automatically.
- You can also [use tags](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) for finer control, and - You can also [use tags](../runners/configure_runners.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) for finer control, and
associate runners with specific jobs. For example, you can use a tag for jobs that associate runners with specific jobs. For example, you can use a tag for jobs that
require dedicated, more powerful, or specific hardware. require dedicated, more powerful, or specific hardware.
- GitLab has [autoscaling for runners](https://docs.gitlab.com/runner/configuration/autoscale.html). - GitLab has [autoscaling for runners](https://docs.gitlab.com/runner/configuration/autoscale.html).
Use autoscaling to provision runners only when needed, and scale down when not needed. Use autoscaling to provision runners only when needed, and scale down when not needed.
This is similar to ephemeral agents in Jenkins. This is similar to ephemeral agents in Jenkins.
If you are using `gitlab.com`, you can take advantage of our [shared runner fleet](../../user/gitlab_com/index.md#shared-runners) If you are using `gitlab.com`, you can take advantage of our [shared runner fleet](../runners/README.md)
to run jobs without provisioning your own runners. We are investigating making them to run jobs without provisioning your own runners. We are investigating making them
[available for self-managed instances](https://gitlab.com/groups/gitlab-org/-/epics/835) [available for self-managed instances](https://gitlab.com/groups/gitlab-org/-/epics/835)
as well. as well.
...@@ -230,7 +230,7 @@ and is meant to be a mapping of concepts there to concepts in GitLab. ...@@ -230,7 +230,7 @@ and is meant to be a mapping of concepts there to concepts in GitLab.
The agent section is used to define how a pipeline executes. For GitLab, we use [runners](../runners/README.md) The agent section is used to define how a pipeline executes. For GitLab, we use [runners](../runners/README.md)
to provide this capability. You can configure your own runners in Kubernetes or on any host, or take advantage to provide this capability. You can configure your own runners in Kubernetes or on any host, or take advantage
of our shared runner fleet (note that the shared runner fleet is only available for GitLab.com users). of our shared runner fleet (note that the shared runner fleet is only available for GitLab.com users).
We also support using [tags](../runners/README.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) to direct different jobs We also support using [tags](../runners/configure_runners.md#use-tags-to-limit-the-number-of-jobs-using-the-runner) to direct different jobs
to different runners (execution agents). to different runners (execution agents).
The `agent` section also allows you to define which Docker images should be used for execution, for which we use The `agent` section also allows you to define which Docker images should be used for execution, for which we use
......
...@@ -35,7 +35,7 @@ There are two options. Using: ...@@ -35,7 +35,7 @@ There are two options. Using:
back to clone if it doesn't exist). back to clone if it doesn't exist).
This is recommended, especially for [large repositories](../large_repositories/index.md#git-strategy). This is recommended, especially for [large repositories](../large_repositories/index.md#git-strategy).
The configured Git strategy can be overridden by the [`GIT_STRATEGY` variable](../runners/README.md#git-strategy) The configured Git strategy can be overridden by the [`GIT_STRATEGY` variable](../runners/configure_runners.md#git-strategy)
in `.gitlab-ci.yml`. in `.gitlab-ci.yml`.
## Git shallow clone ## Git shallow clone
...@@ -66,7 +66,7 @@ if the job surpasses the threshold, it is marked as failed. ...@@ -66,7 +66,7 @@ if the job surpasses the threshold, it is marked as failed.
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17221) in GitLab 10.7. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17221) in GitLab 10.7.
Project defined timeout (either specific timeout set by user or the default Project defined timeout (either specific timeout set by user or the default
60 minutes timeout) may be [overridden for runners](../runners/README.md#set-maximum-job-timeout-for-a-runner). 60 minutes timeout) may be [overridden for runners](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner).
## Maximum artifacts size **(FREE SELF)** ## Maximum artifacts size **(FREE SELF)**
......
...@@ -41,7 +41,7 @@ The job results [are displayed in a pipeline](#view-the-status-of-your-pipeline- ...@@ -41,7 +41,7 @@ The job results [are displayed in a pipeline](#view-the-status-of-your-pipeline-
In GitLab, runners are agents that run your CI/CD jobs. In GitLab, runners are agents that run your CI/CD jobs.
You might already have runners available for your project, including You might already have runners available for your project, including
[shared runners](../runners/README.md#shared-runners), which are [shared runners](../runners/runners_scope.md), which are
available to all projects in your GitLab instance. available to all projects in your GitLab instance.
To view available runners: To view available runners:
......
...@@ -5,847 +5,288 @@ info: To determine the technical writer assigned to the Stage/Group associated w ...@@ -5,847 +5,288 @@ info: To determine the technical writer assigned to the Stage/Group associated w
type: reference type: reference
--- ---
# Configuring runners in GitLab # Runners
In GitLab CI/CD, runners run the code defined in [`.gitlab-ci.yml`](../yaml/README.md). In GitLab CI/CD, runners run the code defined in [`.gitlab-ci.yml`](../yaml/README.md).
A runner is a lightweight, highly-scalable agent that picks up a CI job through A runner is a lightweight, highly-scalable agent that picks up a CI job through
the coordinator API of GitLab CI/CD, runs the job, and sends the result back to the GitLab instance. the coordinator API of GitLab CI/CD, runs the job, and sends the result back to the GitLab instance.
Runners are created by an administrator and are visible in the GitLab UI. If you are using GitLab SaaS (GitLab.com), your CI jobs automatically run on shared runners. No configuration is required.
Runners can be specific to certain projects or available to all projects.
This documentation is focused on using runners in GitLab. If you are using self-managed GitLab or you want to use your own runners on GitLab.com, you can
If you need to install and configure GitLab Runner, see [install and configure your own runners](https://docs.gitlab.com/runner/install/).
[the GitLab Runner documentation](https://docs.gitlab.com/runner/).
## Types of runners ## Runners on GitLab.com
In the GitLab UI there are three types of runners, based on who you want to have access: On GitLab.com, your jobs can run on [Linux](#linux-shared-runners) or [Windows](#windows-shared-runners-beta).
- [Shared runners](#shared-runners) are available to all groups and projects in a GitLab instance. ### Linux shared runners
- [Group runners](#group-runners) are available to all projects and subgroups in a group.
- [Specific runners](#specific-runners) are associated with specific projects.
Typically, specific runners are used for one project at a time.
### Shared runners Linux shared runners on GitLab.com run in autoscale mode and are powered by Google Cloud Platform.
*Shared runners* are available to every project in a GitLab instance. Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each project, thus maximizing security. These shared runners are available for users and customers on GitLab.com.
Use shared runners when you have multiple jobs with similar requirements. Rather than GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier.
having multiple runners idling for many projects, you can have a few runners that handle
multiple projects.
If you are using a self-managed instance of GitLab: All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, CoreOS and the latest Docker Engine
installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default
region of the VMs is US East1.
Each instance is used only for one job, this ensures any sensitive data left on the system can't be accessed by other people their CI jobs.
- Your administrator can install and register shared runners by The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times.
going to your project's **Settings > CI/CD**, expanding the **Runners** section,
and clicking **Show runner installation instructions**.
These instructions are also available [in the documentation](https://docs.gitlab.com/runner/install/index.html).
- The administrator can also configure a maximum number of shared runner [pipeline minutes for
each group](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota).
If you are using GitLab.com: Jobs handled by the shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`),
**time out after 3 hours**, regardless of the timeout configured in a
project. Check the issues [4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference.
- You can select from a list of [shared runners that GitLab maintains](../../user/gitlab_com/index.md#shared-runners). Below are the shared runners settings.
- The shared runners consume the [pipelines minutes](../../subscriptions/gitlab_com/index.md#ci-pipeline-minutes)
included with your account.
#### How shared runners pick jobs | Setting | GitLab.com | Default |
| ----------- | ----------------- | ---------- |
| [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) | [Runner versions dashboard](https://dashboards.gitlab.com/d/000000159/ci?from=now-1h&to=now&refresh=5m&orgId=1&panelId=12&fullscreen&theme=light) | - |
| Executor | `docker+machine` | - |
| Default Docker image | `ruby:2.5` | - |
| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true` | `false` |
Shared runners process jobs by using a fair usage queue. This queue prevents #### Pre-clone script
projects from creating hundreds of jobs and using all available
shared runner resources.
The fair usage queue algorithm assigns jobs based on the projects that have the Linux shared runners on GitLab.com provide a way to run commands in a CI
fewest number of jobs already running on shared runners. job before the runner attempts to run `git init` and `git fetch` to
download a GitLab repository. The
[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section)
can be used for:
**Example 1** - Seeding the build directory with repository data
- Sending a request to a server
- Downloading assets from a CDN
- Any other commands that must run before the `git init`
If these jobs are in the queue: To use this feature, define a [CI/CD variable](../../ci/variables/README.md#custom-cicd-variables) called
`CI_PRE_CLONE_SCRIPT` that contains a bash script.
- Job 1 for Project 1 [This example](../../development/pipelines.md#pre-clone-step)
- Job 2 for Project 1 demonstrates how you might use a pre-clone step to seed the build
- Job 3 for Project 1 directory.
- Job 4 for Project 2
- Job 5 for Project 2
- Job 6 for Project 3
The fair usage algorithm assigns jobs in this order: #### `config.toml`
1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects). The full contents of our `config.toml` are:
1. Job 4 is next, because 4 is now the lowest job number from projects with no running jobs (Project 1 has a job running).
1. Job 6 is next, because 6 is now the lowest job number from projects with no running jobs (Projects 1 and 2 have jobs running).
1. Job 2 is next, because, of projects with the lowest number of jobs running (each has 1), it is the lowest job number.
1. Job 5 is next, because Project 1 now has 2 jobs running and Job 5 is the lowest remaining job number between Projects 2 and 3.
1. Finally is Job 3... because it's the only job left.
---
**Example 2**
If these jobs are in the queue:
- Job 1 for Project 1
- Job 2 for Project 1
- Job 3 for Project 1
- Job 4 for Project 2
- Job 5 for Project 2
- Job 6 for Project 3
The fair usage algorithm assigns jobs in this order:
1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects).
1. We finish Job 1.
1. Job 2 is next, because, having finished Job 1, all projects have 0 jobs running again, and 2 is the lowest available job number.
1. Job 4 is next, because with Project 1 running a Job, 4 is the lowest number from projects running no jobs (Projects 2 and 3).
1. We finish Job 4.
1. Job 5 is next, because having finished Job 4, Project 2 has no jobs running again.
1. Job 6 is next, because Project 3 is the only project left with no running jobs.
1. Lastly we choose Job 3... because, again, it's the only job left.
#### Enable shared runners
On GitLab.com, [shared runners](#shared-runners) are enabled in all projects by
default.
On self-managed instances of GitLab, an administrator must [install](https://docs.gitlab.com/runner/install/index.html)
and [register](https://docs.gitlab.com/runner/register/index.html) them.
You can also enable shared runners for individual projects.
To enable shared runners:
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section.
1. Select **Enable shared runners for this project**.
#### Disable shared runners
You can disable shared runners for individual projects or for groups.
You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the project
or group.
To disable shared runners for a project:
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section.
1. In the **Shared runners** area, select **Enable shared runners for this project** so the toggle is grayed-out.
Shared runners are automatically disabled for a project:
- If the shared runners setting for the parent group is disabled, and
- If overriding this setting is not permitted at the project level.
To disable shared runners for a group:
1. Go to the group's **Settings > CI/CD** and expand the **Runners** section.
1. In the **Shared runners** area, turn off the **Enable shared runners for this group** toggle.
1. Optionally, to allow shared runners to be enabled for individual projects or subgroups,
click **Allow projects and subgroups to override the group setting**.
NOTE:
To re-enable the shared runners for a group, turn on the
**Enable shared runners for this group** toggle.
Then, an owner or maintainer must explicitly change this setting
for each project subgroup or project.
### Group runners
Use *Group runners* when you want all projects in a group
to have access to a set of runners.
Group runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue.
#### Create a group runner
You can create a group runner for your self-managed GitLab instance or for GitLab.com.
You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group.
To create a group runner:
1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/).
1. Go to the group you want to make the runner work for.
1. Go to **Settings > CI/CD** and expand the **Runners** section.
1. Note the URL and token.
1. [Register the runner](https://docs.gitlab.com/runner/register/).
#### View and manage group runners
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/37366/) in GitLab 13.2.
You can view and manage all runners for a group, its subgroups, and projects.
You can do this for your self-managed GitLab instance or for GitLab.com.
You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group.
1. Go to the group where you want to view the runners.
1. Go to **Settings > CI/CD** and expand the **Runners** section.
1. The following fields are displayed.
| Attribute | Description |
| ------------ | ----------- |
| Type | Displays the runner type: `group` or `specific`, together with the optional states `locked` and `paused` |
| Runner token | Token used to identify the runner, and that the runner uses to communicate with the GitLab instance |
| Description | Description given to the runner when it was created |
| Version | GitLab Runner version |
| IP address | IP address of the host on which the runner is registered |
| Projects | The count of projects to which the runner is assigned |
| Jobs | Total of jobs run by the runner |
| Tags | Tags associated with the runner |
| Last contact | Timestamp indicating when the GitLab instance last contacted the runner |
From this page, you can edit, pause, and remove runners from the group, its subgroups, and projects.
#### Pause or remove a group runner
You can pause or remove a group runner for your self-managed GitLab instance or for GitLab.com.
You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group.
1. Go to the group you want to remove or pause the runner for.
1. Go to **Settings > CI/CD** and expand the **Runners** section.
1. Click **Pause** or **Remove runner**.
- If you pause a group runner that is used by multiple projects, the runner pauses for all projects.
- From the group view, you cannot remove a runner that is assigned to more than one project.
You must remove it from each project first.
1. On the confirmation dialog, click **OK**.
### Specific runners
Use *Specific runners* when you want to use runners for specific projects. For example,
when you have:
- Jobs with specific requirements, like a deploy job that requires credentials.
- Projects with a lot of CI activity that can benefit from being separate from other runners.
You can set up a specific runner to be used by multiple projects. Specific runners
must be enabled for each project explicitly.
Specific runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue.
NOTE: NOTE:
Specific runners do not get shared with forked projects automatically. Settings that are not public are shown as `X`.
A fork *does* copy the CI/CD settings of the cloned repository.
**Google Cloud Platform**
#### Create a specific runner
```toml
You can create a specific runner for your self-managed GitLab instance or for GitLab.com. concurrent = X
You must have the [Owner role](../../user/permissions.md#project-members-permissions) for the project. check_interval = 1
metrics_server = "X"
To create a specific runner: sentry_dsn = "X"
1. [Install runner](https://docs.gitlab.com/runner/install/). [[runners]]
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. name = "docker-auto-scale"
1. Note the URL and token. request_concurrency = X
1. [Register the runner](https://docs.gitlab.com/runner/register/). url = "https://gitlab.com/"
token = "SHARED_RUNNER_TOKEN"
#### Enable a specific runner for a specific project pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\""
executor = "docker+machine"
A specific runner is available in the project it was created for. An administrator can environment = [
enable a specific runner to apply to additional projects. "DOCKER_DRIVER=overlay2",
"DOCKER_TLS_CERTDIR="
- You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the ]
project. limit = X
- The specific runner must not be [locked](#prevent-a-specific-runner-from-being-enabled-for-other-projects). [runners.docker]
image = "ruby:2.5"
To enable or disable a specific runner for a project: privileged = true
volumes = [
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. "/certs/client",
1. Click **Enable for this project** or **Disable for this project**. "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP.
]
#### Prevent a specific runner from being enabled for other projects [runners.machine]
IdleCount = 50
You can configure a specific runner so it is "locked" and cannot be enabled for other projects. IdleTime = 3600
This setting can be enabled when you first [register a runner](https://docs.gitlab.com/runner/register/), MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused.
but can also be changed later. MachineName = "srm-%s"
MachineDriver = "google"
To lock or unlock a runner: MachineOptions = [
"google-project=PROJECT",
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section. "google-disk-size=25",
1. Find the runner you want to lock or unlock. Make sure it's enabled. "google-machine-type=n1-standard-1",
1. Click the pencil button. "google-username=core",
1. Check the **Lock to current projects** option. "google-tags=gitlab-com,srm",
1. Click **Save changes**. "google-use-internal-ip",
"google-zone=us-east1-d",
## Manually clear the runner cache "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928
"google-machine-image=PROJECT/global/images/IMAGE",
Read [clearing the cache](../caching/index.md#clearing-the-cache). "engine-opt=ipv6", # This will create IPv6 interfaces in the containers.
"engine-opt=fixed-cidr-v6=fc00::/7",
## Set maximum job timeout for a runner "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600
]
For each runner, you can specify a *maximum job timeout*. This timeout, [[runners.machine.autoscaling]]
if smaller than the [project defined timeout](../pipelines/settings.md#timeout), takes precedence. Periods = ["* * * * * sat,sun *"]
Timezone = "UTC"
This feature can be used to prevent your shared runner from being overwhelmed IdleCount = 70
by a project that has jobs with a long timeout (for example, one week). IdleTime = 3600
[[runners.machine.autoscaling]]
When not configured, runners do not override the project timeout. Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"]
Timezone = "UTC"
On GitLab.com, you cannot override the job timeout for shared runners and must use the [project defined timeout](../pipelines/settings.md#timeout). IdleCount = 700
IdleTime = 3600
To set the maximum job timeout: [runners.cache]
Type = "gcs"
1. In a project, go to **Settings > CI/CD > Runners**. Shared = true
1. Select your specific runner to edit the settings. [runners.cache.gcs]
1. Enter a value under **Maximum job timeout**. CredentialsFile = "/path/to/file"
1. Select **Save changes**. BucketName = "bucket-name"
How this feature works:
**Example 1 - Runner timeout bigger than project timeout**
1. You set the _maximum job timeout_ for a runner to 24 hours
1. You set the _CI/CD Timeout_ for a project to **2 hours**
1. You start a job
1. The job, if running longer, times out after **2 hours**
**Example 2 - Runner timeout not configured**
1. You remove the _maximum job timeout_ configuration from a runner
1. You set the _CI/CD Timeout_ for a project to **2 hours**
1. You start a job
1. The job, if running longer, times out after **2 hours**
**Example 3 - Runner timeout smaller than project timeout**
1. You set the _maximum job timeout_ for a runner to **30 minutes**
1. You set the _CI/CD Timeout_ for a project to 2 hours
1. You start a job
1. The job, if running longer, times out after **30 minutes**
## Be careful with sensitive information
With some [runner executors](https://docs.gitlab.com/runner/executors/README.html),
if you can run a job on the runner, you can get full access to the file system,
and thus any code it runs as well as the token of the runner. With shared runners, this means that anyone
that runs jobs on the runner, can access anyone else's code that runs on the
runner.
In addition, because you can get access to the runner token, it is possible
to create a clone of a runner and submit false jobs, for example.
The above is easily avoided by restricting the usage of shared runners
on large public GitLab instances, controlling access to your GitLab instance,
and using more secure [runner executors](https://docs.gitlab.com/runner/executors/README.html).
### Prevent runners from revealing sensitive information
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13194) in GitLab 10.0.
You can protect runners so they don't reveal sensitive information.
When a runner is protected, the runner picks jobs created on
[protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md) only,
and ignores other jobs.
To protect or unprotect a runner:
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section.
1. Find the runner you want to protect or unprotect. Make sure it's enabled.
1. Click the pencil button.
1. Check the **Protected** option.
1. Click **Save changes**.
![specific runners edit icon](img/protected_runners_check_box.png)
### Forks
Whenever a project is forked, it copies the settings of the jobs that relate
to it. This means that if you have shared runners set up for a project and
someone forks that project, the shared runners serve jobs of this project.
### Attack vectors in runners
Mentioned briefly earlier, but the following things of runners can be exploited.
We're always looking for contributions that can mitigate these
[Security Considerations](https://docs.gitlab.com/runner/security/).
### Reset the runner registration token for a project
If you think that a registration token for a project was revealed, you should
reset it. A token can be used to register another runner for the project. That new runner
may then be used to obtain the values of secret variables or to clone project code.
To reset the token:
1. Go to the project's **Settings > CI/CD**.
1. Expand the **General pipelines settings** section.
1. Find the **Runner token** form field and click the **Reveal value** button.
1. Delete the value and save the form.
1. After the page is refreshed, expand the **Runners settings** section
and check the registration token - it should be changed.
From now on the old token is no longer valid and does not register
any new runners to the project. If you are using any tools to provision and
register new runners, the tokens used in those tools should be updated to reflect the
value of the new token.
## Determine the IP address of a runner
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17286) in GitLab 10.6.
It may be useful to know the IP address of a runner so you can troubleshoot
issues with that runner. GitLab stores and displays the IP address by viewing
the source of the HTTP requests it makes to GitLab when polling for jobs. The
IP address is always kept up to date so if the runner IP changes it
automatically updates in GitLab.
The IP address for shared runners and specific runners can be found in
different places.
### Determine the IP address of a shared runner
To view the IP address of a shared runner you must have admin access to
the GitLab instance. To determine this:
1. Visit **Admin Area > Overview > Runners**.
1. Look for the runner in the table and you should see a column for **IP Address**.
![shared runner IP address](img/shared_runner_ip_address.png)
### Determine the IP address of a specific runner
To can find the IP address of a runner for a specific project,
you must have the [Owner role](../../user/permissions.md#project-members-permissions) for the
project.
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section.
1. On the details page you should see a row for **IP Address**.
![specific runner IP address](img/specific_runner_ip_address.png)
## Use tags to limit the number of jobs using the runner
You must set up a runner to be able to run all the different types of jobs
that it may encounter on the projects it's shared over. This would be
problematic for large amounts of projects, if it weren't for tags.
GitLab CI tags are not the same as Git tags. GitLab CI tags are associated with runners.
Git tags are associated with commits.
By tagging a runner for the types of jobs it can handle, you can make sure
shared runners will [only run the jobs they are equipped to run](../yaml/README.md#tags).
For instance, at GitLab we have runners tagged with `rails` if they contain
the appropriate dependencies to run Rails test suites.
When you [register a runner](https://docs.gitlab.com/runner/register/), its default behavior is to **only pick**
[tagged jobs](../yaml/README.md#tags).
To change this, you must have the [Owner role](../../user/permissions.md#project-members-permissions) for the project.
To make a runner pick untagged jobs:
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section.
1. Find the runner you want to pick untagged jobs and make sure it's enabled.
1. Click the pencil button.
1. Check the **Run untagged jobs** option.
1. Click the **Save changes** button for the changes to take effect.
NOTE:
The runner tags list can not be empty when it's not allowed to pick untagged jobs.
Below are some example scenarios of different variations.
### runner runs only tagged jobs
The following examples illustrate the potential impact of the runner being set
to run only tagged jobs.
Example 1:
1. The runner is configured to run only tagged jobs and has the `docker` tag.
1. A job that has a `hello` tag is executed and stuck.
Example 2:
1. The runner is configured to run only tagged jobs and has the `docker` tag.
1. A job that has a `docker` tag is executed and run.
Example 3:
1. The runner is configured to run only tagged jobs and has the `docker` tag.
1. A job that has no tags defined is executed and stuck.
### runner is allowed to run untagged jobs
The following examples illustrate the potential impact of the runner being set
to run tagged and untagged jobs.
Example 1:
1. The runner is configured to run untagged jobs and has the `docker` tag.
1. A job that has no tags defined is executed and run.
1. A second job that has a `docker` tag defined is executed and run.
Example 2:
1. The runner is configured to run untagged jobs and has no tags defined.
1. A job that has no tags defined is executed and run.
1. A second job that has a `docker` tag defined is stuck.
## Configure runner behavior with variables
You can use [CI/CD variables](../variables/README.md) to configure runner Git behavior
globally or for individual jobs:
- [`GIT_STRATEGY`](#git-strategy)
- [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy)
- [`GIT_CHECKOUT`](#git-checkout)
- [`GIT_CLEAN_FLAGS`](#git-clean-flags)
- [`GIT_FETCH_EXTRA_FLAGS`](#git-fetch-extra-flags)
- [`GIT_DEPTH`](#shallow-cloning) (shallow cloning)
- [`GIT_CLONE_PATH`](#custom-build-directories) (custom build directories)
You can also use variables to configure how many times a runner
[attempts certain stages of job execution](#job-stages-attempts).
### Git strategy
> - Introduced in GitLab 8.9 as an experimental feature.
> - `GIT_STRATEGY=none` requires GitLab Runner v1.7+.
You can set the `GIT_STRATEGY` used to fetch the repository content, either
globally or per-job in the [`variables`](../yaml/README.md#variables) section:
```yaml
variables:
GIT_STRATEGY: clone
```
There are three possible values: `clone`, `fetch`, and `none`. If left unspecified,
jobs use the [project's pipeline setting](../pipelines/settings.md#git-strategy).
`clone` is the slowest option. It clones the repository from scratch for every
job, ensuring that the local working copy is always pristine.
If an existing worktree is found, it is removed before cloning.
`fetch` is faster as it re-uses the local working copy (falling back to `clone`
if it does not exist). `git clean` is used to undo any changes made by the last
job, and `git fetch` is used to retrieve commits made after the last job ran.
However, `fetch` does require access to the previous worktree. This works
well when using the `shell` or `docker` executor because these
try to preserve worktrees and try to re-use them by default.
This has limitations when using the [Docker Machine executor](https://docs.gitlab.com/runner/executors/docker_machine.html).
It does not work for [the `kubernetes` executor](https://docs.gitlab.com/runner/executors/kubernetes.html),
but a [feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3847) exists.
The `kubernetes` executor always clones into an temporary directory.
A Git strategy of `none` also re-uses the local working copy, but skips all Git
operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped,
if present. This strategy could mean you need to add `fetch` and `checkout` commands
to [your `.gitlab-ci.yml` script](../yaml/README.md#script).
It can be used for jobs that operate exclusively on artifacts, like a deployment job.
Git repository data may be present, but it's likely out of date. You should only
rely on files brought into the local working copy from cache or artifacts.
### Git submodule strategy
> Requires GitLab Runner v1.10+.
The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git
submodules are included when fetching the code before a build. You can set them
globally or per-job in the [`variables`](../yaml/README.md#variables) section.
There are three possible values: `none`, `normal`, and `recursive`:
- `none` means that submodules are not included when fetching the project
code. This is the default, which matches the pre-v1.10 behavior.
- `normal` means that only the top-level submodules are included. It's
equivalent to:
```shell
git submodule sync
git submodule update --init
```
- `recursive` means that all submodules (including submodules of submodules)
are included. This feature needs Git v1.8.1 and later. When using a
GitLab Runner with an executor not based on Docker, make sure the Git version
meets that requirement. It's equivalent to:
```shell
git submodule sync --recursive
git submodule update --init --recursive
```
For this feature to work correctly, the submodules must be configured
(in `.gitmodules`) with either:
- the HTTP(S) URL of a publicly-accessible repository, or
- a relative path to another repository on the same GitLab server. See the
[Git submodules](../git_submodules.md) documentation.
### Git checkout
> Introduced in GitLab Runner 9.3.
The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either
`clone` or `fetch` to specify whether a `git checkout` should be run. If not
specified, it defaults to true. You can set them globally or per-job in the
[`variables`](../yaml/README.md#variables) section.
If set to `false`, the runner:
- when doing `fetch` - updates the repository and leaves the working copy on
the current revision,
- when doing `clone` - clones the repository and leaves the working copy on the
default branch.
If `GIT_CHECKOUT` is set to `true`, both `clone` and `fetch` work the same way.
The runner checks out the working copy of a revision related
to the CI pipeline:
```yaml
variables:
GIT_STRATEGY: clone
GIT_CHECKOUT: "false"
script:
- git checkout -B master origin/master
- git merge $CI_COMMIT_SHA
``` ```
### Git clean flags ### Windows shared runners (beta)
> Introduced in GitLab Runner 11.10 The Windows shared runners are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta)
and shouldn't be used for production workloads.
The `GIT_CLEAN_FLAGS` variable is used to control the default behavior of During this beta period, the [shared runner pipeline quota](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota)
`git clean` after checking out the sources. You can set it globally or per-job in the applies for groups and projects in the same manner as Linux runners. This may
[`variables`](../yaml/README.md#variables) section. change when the beta period ends, as discussed in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30834).
`GIT_CLEAN_FLAGS` accepts all possible options of the [`git clean`](https://git-scm.com/docs/git-clean) Windows shared runners on GitLab.com autoscale by launching virtual machines on
command. the Google Cloud Platform. This solution uses an
[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md)
developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html).
Windows shared runners execute your CI/CD jobs on `n1-standard-2` instances with
2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in
the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/master/cookbooks/preinstalled-software/README.md).
`git clean` is disabled if `GIT_CHECKOUT: "false"` is specified. We want to keep iterating to get Windows shared runners in a stable state and
[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga).
You can follow our work towards this goal in the
[related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162).
If `GIT_CLEAN_FLAGS` is: #### Configuration
- Not specified, `git clean` flags default to `-ffdx`. The full contents of our `config.toml` are:
- Given the value `none`, `git clean` is not executed.
For example:
```yaml
variables:
GIT_CLEAN_FLAGS: -ffdx -e cache/
script:
- ls -al cache/
```
### Git fetch extra flags
> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1.
The `GIT_FETCH_EXTRA_FLAGS` variable is used to control the behavior of
`git fetch`. You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section.
`GIT_FETCH_EXTRA_FLAGS` accepts all options of the [`git fetch`](https://git-scm.com/docs/git-fetch) command. However, `GIT_FETCH_EXTRA_FLAGS` flags are appended after the default flags that can't be modified.
The default flags are:
- [GIT_DEPTH](#shallow-cloning).
- The list of [refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec).
- A remote called `origin`.
If `GIT_FETCH_EXTRA_FLAGS` is:
- Not specified, `git fetch` flags default to `--prune --quiet` along with the default flags.
- Given the value `none`, `git fetch` is executed only with the default flags.
For example, the default flags are `--prune --quiet`, so you can make `git fetch` more verbose by overriding this with just `--prune`:
```yaml
variables:
GIT_FETCH_EXTRA_FLAGS: --prune
script:
- ls -al cache/
```
The configuration above results in `git fetch` being called this way:
```shell
git fetch origin $REFSPECS --depth 50 --prune
```
Where `$REFSPECS` is a value provided to the runner internally by GitLab. NOTE:
Settings that aren't public are shown as `X`.
### Shallow cloning
```toml
> Introduced in GitLab 8.9 as an experimental feature. concurrent = X
check_interval = 3
You can specify the depth of fetching and cloning using `GIT_DEPTH`.
`GIT_DEPTH` does a shallow clone of the repository and can significantly speed up cloning. [[runners]]
It can be helpful for repositories with a large number of commits or old, large binaries. The value is name = "windows-runner"
passed to `git fetch` and `git clone`. url = "https://gitlab.com/"
token = "TOKEN"
In GitLab 12.0 and later, newly-created projects automatically have a executor = "custom"
[default `git depth` value of `50`](../pipelines/settings.md#git-shallow-clone). builds_dir = "C:\\GitLab-Runner\\builds"
cache_dir = "C:\\GitLab-Runner\\cache"
If you use a depth of `1` and have a queue of jobs or retry shell = "powershell"
jobs, jobs may fail. [runners.custom]
config_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe"
Git fetching and cloning is based on a ref, such as a branch name, so runners config_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "config"]
can't clone a specific commit SHA. If multiple jobs are in the queue, or prepare_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe"
you're retrying an old job, the commit to be tested must be within the prepare_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "prepare"]
Git history that is cloned. Setting too small a value for `GIT_DEPTH` can make run_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe"
it impossible to run these old commits and `unresolved reference` is displayed in run_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "run"]
job logs. You should then reconsider changing `GIT_DEPTH` to a higher value. cleanup_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe"
cleanup_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "cleanup"]
Jobs that rely on `git describe` may not work correctly when `GIT_DEPTH` is
set since only part of the Git history is present.
To fetch or clone only the last 3 commits:
```yaml
variables:
GIT_DEPTH: "3"
``` ```
You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section. The full contents of our `autoscaler/config.toml` are:
### Custom build directories ```toml
Provider = "gcp"
> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2211) in GitLab Runner 11.10. Executor = "winrm"
OS = "windows"
By default, GitLab Runner clones the repository in a unique subpath of the LogLevel = "info"
`$CI_BUILDS_DIR` directory. However, your project might require the code in a LogFormat = "text"
specific directory (Go projects, for example). In that case, you can specify LogFile = "C:\\GitLab-Runner\\autoscaler\\autoscaler.log"
the `GIT_CLONE_PATH` variable to tell the runner the directory to clone the VMTag = "windows"
repository in:
[GCP]
```yaml ServiceAccountFile = "PATH"
variables: Project = "some-project-df9323"
GIT_CLONE_PATH: $CI_BUILDS_DIR/project-name Zone = "us-east1-c"
MachineType = "n1-standard-2"
test: Image = "IMAGE"
script: DiskSize = 50
- pwd DiskType = "pd-standard"
Subnetwork = "default"
Network = "default"
Tags = ["TAGS"]
Username = "gitlab_runner"
[WinRM]
MaximumTimeout = 3600
ExecutionMaxRetries = 0
[ProviderCache]
Enabled = true
Directory = "C:\\GitLab-Runner\\autoscaler\\machines"
``` ```
The `GIT_CLONE_PATH` has to always be within `$CI_BUILDS_DIR`. The directory set in `$CI_BUILDS_DIR` #### Example
is dependent on executor and configuration of [runners.builds_dir](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section)
setting.
This can only be used when `custom_build_dir` is enabled in the
[runner's configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscustom_build_dir-section).
This is the default configuration for the `docker` and `kubernetes` executors.
#### Handling concurrency
An executor that uses a concurrency greater than `1` might lead
to failures. Multiple jobs might be working on the same directory if the `builds_dir`
is shared between jobs.
The runner does not try to prevent this situation. It's up to the administrator
and developers to comply with the requirements of runner configuration.
To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because runner
exposes two additional variables that provide a unique `ID` of concurrency:
- `$CI_CONCURRENT_ID`: Unique ID for all jobs running within the given executor. Below is a simple `.gitlab-ci.yml` file to show how to start using the
- `$CI_CONCURRENT_PROJECT_ID`: Unique ID for all jobs running within the given executor and project. Windows shared runners:
The most stable configuration that should work well in any scenario and on any executor
is to use `$CI_CONCURRENT_ID` in the `GIT_CLONE_PATH`. For example:
```yaml ```yaml
variables: .shared_windows_runners:
GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/project-name tags:
- shared-windows
test: - windows
- windows-1809
stages:
- build
- test
before_script:
- Set-Variable -Name "time" -Value (date -Format "%H:%m")
- echo ${time}
- echo "started by ${GITLAB_USER_NAME}"
build:
extends:
- .shared_windows_runners
stage: build
script: script:
- pwd - echo "running scripts in the build job"
```
The `$CI_CONCURRENT_PROJECT_ID` should be used in conjunction with `$CI_PROJECT_PATH`
as the `$CI_PROJECT_PATH` provides a path of a repository. That is, `group/subgroup/project`. For example:
```yaml
variables:
GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH
test: test:
extends:
- .shared_windows_runners
stage: test
script: script:
- pwd - echo "running scripts in the test job"
```
#### Nested paths
The value of `GIT_CLONE_PATH` is expanded once and nesting variables
within is not supported.
For example, you define both the variables below in your
`.gitlab-ci.yml` file:
```yaml
variables:
GOPATH: $CI_BUILDS_DIR/go
GIT_CLONE_PATH: $GOPATH/src/namespace/project
```
The value of `GIT_CLONE_PATH` is expanded once into
`$CI_BUILDS_DIR/go/src/namespace/project`, and results in failure
because `$CI_BUILDS_DIR` is not expanded.
### Job stages attempts
> Introduced in GitLab, it requires GitLab Runner v1.9+.
You can set the number of attempts that the running job tries to execute
the following stages:
| Variable | Description |
|---------------------------------|--------------------------------------------------------|
| `ARTIFACT_DOWNLOAD_ATTEMPTS` | Number of attempts to download artifacts running a job |
| `EXECUTOR_JOB_SECTION_ATTEMPTS` | [In GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) and later, the number of attempts to run a section in a job after a [`No Such Container`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) error ([Docker executor](https://docs.gitlab.com/runner/executors/docker.html) only). |
| `GET_SOURCES_ATTEMPTS` | Number of attempts to fetch sources running a job |
| `RESTORE_CACHE_ATTEMPTS` | Number of attempts to restore the cache running a job |
The default is one single attempt.
Example:
```yaml
variables:
GET_SOURCES_ATTEMPTS: 3
```
You can set them globally or per-job in the [`variables`](../yaml/README.md#variables) section.
## System calls not available on GitLab.com shared runners
GitLab.com shared runners run on CoreOS. This means that you cannot use some system calls, like `getlogin`, from the C standard library.
## Artifact and cache settings
> Introduced in GitLab Runner 13.9.
Artifact and cache settings control the compression ratio of artifacts and caches.
Use these settings to specify the size of the archive produced by a job.
- On a slow network, uploads might be faster for smaller archives.
- On a fast network where bandwidth and storage are not a concern, uploads might be faster using the fastest compression ratio, despite the archive produced being larger.
For [GitLab Pages](../../user/project/pages/index.md) to serve
[HTTP Range requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests), artifacts
should use the `ARTIFACT_COMPRESSION_LEVEL: fastest` setting, as only uncompressed zip archives
support this feature.
A meter can also be enabled to provide the rate of transfer for uploads and downloads.
```yaml
variables:
# output upload and download progress every 2 seconds
TRANSFER_METER_FREQUENCY: "2s"
# Use fast compression for artifacts, resulting in larger archives
ARTIFACT_COMPRESSION_LEVEL: "fast"
# Use no compression for caches
CACHE_COMPRESSION_LEVEL: "fastest"
``` ```
| Variable | Description | #### Limitations and known issues
|---------------------------------|--------------------------------------------------------|
| `TRANSFER_METER_FREQUENCY` | Specify how often to print the meter's transfer rate. It can be set to a duration (for example, `1s` or `1m30s`). A duration of `0` disables the meter (default). When a value is set, the pipeline shows a progress meter for artifact and cache uploads and downloads. | - All the limitations mentioned in our [beta
| `ARTIFACT_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | definition](https://about.gitlab.com/handbook/product/#beta).
| `CACHE_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. | - The average provisioning time for a new Windows VM is 5 minutes.
This means that you may notice slower build start times
on the Windows shared runner fleet during the beta. In a future
release we intend to update the autoscaler to enable
the pre-provisioning of virtual machines. This is intended to significantly reduce
the time it takes to provision a VM on the Windows fleet. You can
follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32).
- The Windows shared runner fleet may be unavailable occasionally
for maintenance or updates.
- The Windows shared runner virtual machine instances do not use the
GitLab Docker executor. This means that you can't specify
[`image`](../../ci/yaml/README.md#image) or [`services`](../../ci/yaml/README.md#services) in
your pipeline configuration.
- For the beta release, we have included a set of software packages in
the base VM image. If your CI job requires additional software that's
not included in this list, then you must add installation
commands to [`before_script`](../../ci/yaml/README.md#before_script) or [`script`](../../ci/yaml/README.md#script) to install the required
software. Note that each job runs on a new VM instance, so the
installation of additional software packages needs to be repeated for
each job in your pipeline.
- The job may stay in a pending state for longer than the
Linux shared runners.
- There is the possibility that we introduce breaking changes which will
require updates to pipelines that are using the Windows shared runner
fleet.
---
stage: Verify
group: Runner
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
type: reference
---
# Configuring runners
If you have installed your own runners, you can configure and secure them in GitLab.
If you need to configure runners on the machine where you installed GitLab Runner, see
[the GitLab Runner documentation](https://docs.gitlab.com/runner/configuration).
## Manually clear the runner cache
Read [clearing the cache](../caching/index.md#clearing-the-cache).
## Set maximum job timeout for a runner
For each runner, you can specify a *maximum job timeout*. This timeout,
if smaller than the [project defined timeout](../pipelines/settings.md#timeout), takes precedence.
This feature can be used to prevent your shared runner from being overwhelmed
by a project that has jobs with a long timeout (for example, one week).
When not configured, runners do not override the project timeout.
On GitLab.com, you cannot override the job timeout for shared runners and must use the [project defined timeout](../pipelines/settings.md#timeout).
To set the maximum job timeout:
1. In a project, go to **Settings > CI/CD > Runners**.
1. Select your specific runner to edit the settings.
1. Enter a value under **Maximum job timeout**.
1. Select **Save changes**.
How this feature works:
**Example 1 - Runner timeout bigger than project timeout**
1. You set the _maximum job timeout_ for a runner to 24 hours
1. You set the _CI/CD Timeout_ for a project to **2 hours**
1. You start a job
1. The job, if running longer, times out after **2 hours**
**Example 2 - Runner timeout not configured**
1. You remove the _maximum job timeout_ configuration from a runner
1. You set the _CI/CD Timeout_ for a project to **2 hours**
1. You start a job
1. The job, if running longer, times out after **2 hours**
**Example 3 - Runner timeout smaller than project timeout**
1. You set the _maximum job timeout_ for a runner to **30 minutes**
1. You set the _CI/CD Timeout_ for a project to 2 hours
1. You start a job
1. The job, if running longer, times out after **30 minutes**
## Be careful with sensitive information
With some [runner executors](https://docs.gitlab.com/runner/executors/README.html),
if you can run a job on the runner, you can get full access to the file system,
and thus any code it runs as well as the token of the runner. With shared runners, this means that anyone
that runs jobs on the runner, can access anyone else's code that runs on the
runner.
In addition, because you can get access to the runner token, it is possible
to create a clone of a runner and submit false jobs, for example.
The above is easily avoided by restricting the usage of shared runners
on large public GitLab instances, controlling access to your GitLab instance,
and using more secure [runner executors](https://docs.gitlab.com/runner/executors/README.html).
### Prevent runners from revealing sensitive information
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/13194) in GitLab 10.0.
You can protect runners so they don't reveal sensitive information.
When a runner is protected, the runner picks jobs created on
[protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md) only,
and ignores other jobs.
To protect or unprotect a runner:
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section.
1. Find the runner you want to protect or unprotect. Make sure it's enabled.
1. Click the pencil button.
1. Check the **Protected** option.
1. Click **Save changes**.
![specific runners edit icon](img/protected_runners_check_box.png)
### Forks
Whenever a project is forked, it copies the settings of the jobs that relate
to it. This means that if you have shared runners set up for a project and
someone forks that project, the shared runners serve jobs of this project.
### Attack vectors in runners
Mentioned briefly earlier, but the following things of runners can be exploited.
We're always looking for contributions that can mitigate these
[Security Considerations](https://docs.gitlab.com/runner/security/).
### Reset the runner registration token for a project
If you think that a registration token for a project was revealed, you should
reset it. A token can be used to register another runner for the project. That new runner
may then be used to obtain the values of secret variables or to clone project code.
To reset the token:
1. Go to the project's **Settings > CI/CD**.
1. Expand the **General pipelines settings** section.
1. Find the **Runner token** form field and click the **Reveal value** button.
1. Delete the value and save the form.
1. After the page is refreshed, expand the **Runners settings** section
and check the registration token - it should be changed.
From now on the old token is no longer valid and does not register
any new runners to the project. If you are using any tools to provision and
register new runners, the tokens used in those tools should be updated to reflect the
value of the new token.
## Determine the IP address of a runner
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17286) in GitLab 10.6.
It may be useful to know the IP address of a runner so you can troubleshoot
issues with that runner. GitLab stores and displays the IP address by viewing
the source of the HTTP requests it makes to GitLab when polling for jobs. The
IP address is always kept up to date so if the runner IP changes it
automatically updates in GitLab.
The IP address for shared runners and specific runners can be found in
different places.
### Determine the IP address of a shared runner
To view the IP address of a shared runner you must have admin access to
the GitLab instance. To determine this:
1. Visit **Admin Area > Overview > Runners**.
1. Look for the runner in the table and you should see a column for **IP Address**.
![shared runner IP address](img/shared_runner_ip_address.png)
### Determine the IP address of a specific runner
To can find the IP address of a runner for a specific project,
you must have the [Owner role](../../user/permissions.md#project-members-permissions) for the
project.
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section.
1. On the details page you should see a row for **IP Address**.
![specific runner IP address](img/specific_runner_ip_address.png)
## Use tags to limit the number of jobs using the runner
You must set up a runner to be able to run all the different types of jobs
that it may encounter on the projects it's shared over. This would be
problematic for large amounts of projects, if it weren't for tags.
GitLab CI tags are not the same as Git tags. GitLab CI tags are associated with runners.
Git tags are associated with commits.
By tagging a runner for the types of jobs it can handle, you can make sure
shared runners will [only run the jobs they are equipped to run](../yaml/README.md#tags).
For instance, at GitLab we have runners tagged with `rails` if they contain
the appropriate dependencies to run Rails test suites.
When you [register a runner](https://docs.gitlab.com/runner/register/), its default behavior is to **only pick**
[tagged jobs](../yaml/README.md#tags).
To change this, you must have the [Owner role](../../user/permissions.md#project-members-permissions) for the project.
To make a runner pick untagged jobs:
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section.
1. Find the runner you want to pick untagged jobs and make sure it's enabled.
1. Click the pencil button.
1. Check the **Run untagged jobs** option.
1. Click the **Save changes** button for the changes to take effect.
NOTE:
The runner tags list can not be empty when it's not allowed to pick untagged jobs.
Below are some example scenarios of different variations.
### runner runs only tagged jobs
The following examples illustrate the potential impact of the runner being set
to run only tagged jobs.
Example 1:
1. The runner is configured to run only tagged jobs and has the `docker` tag.
1. A job that has a `hello` tag is executed and stuck.
Example 2:
1. The runner is configured to run only tagged jobs and has the `docker` tag.
1. A job that has a `docker` tag is executed and run.
Example 3:
1. The runner is configured to run only tagged jobs and has the `docker` tag.
1. A job that has no tags defined is executed and stuck.
### runner is allowed to run untagged jobs
The following examples illustrate the potential impact of the runner being set
to run tagged and untagged jobs.
Example 1:
1. The runner is configured to run untagged jobs and has the `docker` tag.
1. A job that has no tags defined is executed and run.
1. A second job that has a `docker` tag defined is executed and run.
Example 2:
1. The runner is configured to run untagged jobs and has no tags defined.
1. A job that has no tags defined is executed and run.
1. A second job that has a `docker` tag defined is stuck.
## Configure runner behavior with variables
You can use [CI/CD variables](../variables/README.md) to configure runner Git behavior
globally or for individual jobs:
- [`GIT_STRATEGY`](#git-strategy)
- [`GIT_SUBMODULE_STRATEGY`](#git-submodule-strategy)
- [`GIT_CHECKOUT`](#git-checkout)
- [`GIT_CLEAN_FLAGS`](#git-clean-flags)
- [`GIT_FETCH_EXTRA_FLAGS`](#git-fetch-extra-flags)
- [`GIT_DEPTH`](#shallow-cloning) (shallow cloning)
- [`GIT_CLONE_PATH`](#custom-build-directories) (custom build directories)
You can also use variables to configure how many times a runner
[attempts certain stages of job execution](#job-stages-attempts).
### Git strategy
> - Introduced in GitLab 8.9 as an experimental feature.
> - `GIT_STRATEGY=none` requires GitLab Runner v1.7+.
You can set the `GIT_STRATEGY` used to fetch the repository content, either
globally or per-job in the [`variables`](../yaml/README.md#variables) section:
```yaml
variables:
GIT_STRATEGY: clone
```
There are three possible values: `clone`, `fetch`, and `none`. If left unspecified,
jobs use the [project's pipeline setting](../pipelines/settings.md#git-strategy).
`clone` is the slowest option. It clones the repository from scratch for every
job, ensuring that the local working copy is always pristine.
If an existing worktree is found, it is removed before cloning.
`fetch` is faster as it re-uses the local working copy (falling back to `clone`
if it does not exist). `git clean` is used to undo any changes made by the last
job, and `git fetch` is used to retrieve commits made after the last job ran.
However, `fetch` does require access to the previous worktree. This works
well when using the `shell` or `docker` executor because these
try to preserve worktrees and try to re-use them by default.
This has limitations when using the [Docker Machine executor](https://docs.gitlab.com/runner/executors/docker_machine.html).
It does not work for [the `kubernetes` executor](https://docs.gitlab.com/runner/executors/kubernetes.html),
but a [feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3847) exists.
The `kubernetes` executor always clones into an temporary directory.
A Git strategy of `none` also re-uses the local working copy, but skips all Git
operations normally done by GitLab. GitLab Runner pre-clone scripts are also skipped,
if present. This strategy could mean you need to add `fetch` and `checkout` commands
to [your `.gitlab-ci.yml` script](../yaml/README.md#script).
It can be used for jobs that operate exclusively on artifacts, like a deployment job.
Git repository data may be present, but it's likely out of date. You should only
rely on files brought into the local working copy from cache or artifacts.
### Git submodule strategy
> Requires GitLab Runner v1.10+.
The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git
submodules are included when fetching the code before a build. You can set them
globally or per-job in the [`variables`](../yaml/README.md#variables) section.
There are three possible values: `none`, `normal`, and `recursive`:
- `none` means that submodules are not included when fetching the project
code. This is the default, which matches the pre-v1.10 behavior.
- `normal` means that only the top-level submodules are included. It's
equivalent to:
```shell
git submodule sync
git submodule update --init
```
- `recursive` means that all submodules (including submodules of submodules)
are included. This feature needs Git v1.8.1 and later. When using a
GitLab Runner with an executor not based on Docker, make sure the Git version
meets that requirement. It's equivalent to:
```shell
git submodule sync --recursive
git submodule update --init --recursive
```
For this feature to work correctly, the submodules must be configured
(in `.gitmodules`) with either:
- the HTTP(S) URL of a publicly-accessible repository, or
- a relative path to another repository on the same GitLab server. See the
[Git submodules](../git_submodules.md) documentation.
### Git checkout
> Introduced in GitLab Runner 9.3.
The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either
`clone` or `fetch` to specify whether a `git checkout` should be run. If not
specified, it defaults to true. You can set them globally or per-job in the
[`variables`](../yaml/README.md#variables) section.
If set to `false`, the runner:
- when doing `fetch` - updates the repository and leaves the working copy on
the current revision,
- when doing `clone` - clones the repository and leaves the working copy on the
default branch.
If `GIT_CHECKOUT` is set to `true`, both `clone` and `fetch` work the same way.
The runner checks out the working copy of a revision related
to the CI pipeline:
```yaml
variables:
GIT_STRATEGY: clone
GIT_CHECKOUT: "false"
script:
- git checkout -B master origin/master
- git merge $CI_COMMIT_SHA
```
### Git clean flags
> Introduced in GitLab Runner 11.10
The `GIT_CLEAN_FLAGS` variable is used to control the default behavior of
`git clean` after checking out the sources. You can set it globally or per-job in the
[`variables`](../yaml/README.md#variables) section.
`GIT_CLEAN_FLAGS` accepts all possible options of the [`git clean`](https://git-scm.com/docs/git-clean)
command.
`git clean` is disabled if `GIT_CHECKOUT: "false"` is specified.
If `GIT_CLEAN_FLAGS` is:
- Not specified, `git clean` flags default to `-ffdx`.
- Given the value `none`, `git clean` is not executed.
For example:
```yaml
variables:
GIT_CLEAN_FLAGS: -ffdx -e cache/
script:
- ls -al cache/
```
### Git fetch extra flags
> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4142) in GitLab Runner 13.1.
The `GIT_FETCH_EXTRA_FLAGS` variable is used to control the behavior of
`git fetch`. You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section.
`GIT_FETCH_EXTRA_FLAGS` accepts all options of the [`git fetch`](https://git-scm.com/docs/git-fetch) command. However, `GIT_FETCH_EXTRA_FLAGS` flags are appended after the default flags that can't be modified.
The default flags are:
- [GIT_DEPTH](#shallow-cloning).
- The list of [refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec).
- A remote called `origin`.
If `GIT_FETCH_EXTRA_FLAGS` is:
- Not specified, `git fetch` flags default to `--prune --quiet` along with the default flags.
- Given the value `none`, `git fetch` is executed only with the default flags.
For example, the default flags are `--prune --quiet`, so you can make `git fetch` more verbose by overriding this with just `--prune`:
```yaml
variables:
GIT_FETCH_EXTRA_FLAGS: --prune
script:
- ls -al cache/
```
The configuration above results in `git fetch` being called this way:
```shell
git fetch origin $REFSPECS --depth 50 --prune
```
Where `$REFSPECS` is a value provided to the runner internally by GitLab.
### Shallow cloning
> Introduced in GitLab 8.9 as an experimental feature.
You can specify the depth of fetching and cloning using `GIT_DEPTH`.
`GIT_DEPTH` does a shallow clone of the repository and can significantly speed up cloning.
It can be helpful for repositories with a large number of commits or old, large binaries. The value is
passed to `git fetch` and `git clone`.
In GitLab 12.0 and later, newly-created projects automatically have a
[default `git depth` value of `50`](../pipelines/settings.md#git-shallow-clone).
If you use a depth of `1` and have a queue of jobs or retry
jobs, jobs may fail.
Git fetching and cloning is based on a ref, such as a branch name, so runners
can't clone a specific commit SHA. If multiple jobs are in the queue, or
you're retrying an old job, the commit to be tested must be within the
Git history that is cloned. Setting too small a value for `GIT_DEPTH` can make
it impossible to run these old commits and `unresolved reference` is displayed in
job logs. You should then reconsider changing `GIT_DEPTH` to a higher value.
Jobs that rely on `git describe` may not work correctly when `GIT_DEPTH` is
set since only part of the Git history is present.
To fetch or clone only the last 3 commits:
```yaml
variables:
GIT_DEPTH: "3"
```
You can set it globally or per-job in the [`variables`](../yaml/README.md#variables) section.
### Custom build directories
> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2211) in GitLab Runner 11.10.
By default, GitLab Runner clones the repository in a unique subpath of the
`$CI_BUILDS_DIR` directory. However, your project might require the code in a
specific directory (Go projects, for example). In that case, you can specify
the `GIT_CLONE_PATH` variable to tell the runner the directory to clone the
repository in:
```yaml
variables:
GIT_CLONE_PATH: $CI_BUILDS_DIR/project-name
test:
script:
- pwd
```
The `GIT_CLONE_PATH` has to always be within `$CI_BUILDS_DIR`. The directory set in `$CI_BUILDS_DIR`
is dependent on executor and configuration of [runners.builds_dir](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section)
setting.
This can only be used when `custom_build_dir` is enabled in the
[runner's configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscustom_build_dir-section).
This is the default configuration for the `docker` and `kubernetes` executors.
#### Handling concurrency
An executor that uses a concurrency greater than `1` might lead
to failures. Multiple jobs might be working on the same directory if the `builds_dir`
is shared between jobs.
The runner does not try to prevent this situation. It's up to the administrator
and developers to comply with the requirements of runner configuration.
To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because runner
exposes two additional variables that provide a unique `ID` of concurrency:
- `$CI_CONCURRENT_ID`: Unique ID for all jobs running within the given executor.
- `$CI_CONCURRENT_PROJECT_ID`: Unique ID for all jobs running within the given executor and project.
The most stable configuration that should work well in any scenario and on any executor
is to use `$CI_CONCURRENT_ID` in the `GIT_CLONE_PATH`. For example:
```yaml
variables:
GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/project-name
test:
script:
- pwd
```
The `$CI_CONCURRENT_PROJECT_ID` should be used in conjunction with `$CI_PROJECT_PATH`
as the `$CI_PROJECT_PATH` provides a path of a repository. That is, `group/subgroup/project`. For example:
```yaml
variables:
GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH
test:
script:
- pwd
```
#### Nested paths
The value of `GIT_CLONE_PATH` is expanded once and nesting variables
within is not supported.
For example, you define both the variables below in your
`.gitlab-ci.yml` file:
```yaml
variables:
GOPATH: $CI_BUILDS_DIR/go
GIT_CLONE_PATH: $GOPATH/src/namespace/project
```
The value of `GIT_CLONE_PATH` is expanded once into
`$CI_BUILDS_DIR/go/src/namespace/project`, and results in failure
because `$CI_BUILDS_DIR` is not expanded.
### Job stages attempts
> Introduced in GitLab, it requires GitLab Runner v1.9+.
You can set the number of attempts that the running job tries to execute
the following stages:
| Variable | Description |
|---------------------------------|--------------------------------------------------------|
| `ARTIFACT_DOWNLOAD_ATTEMPTS` | Number of attempts to download artifacts running a job |
| `EXECUTOR_JOB_SECTION_ATTEMPTS` | [In GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) and later, the number of attempts to run a section in a job after a [`No Such Container`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4450) error ([Docker executor](https://docs.gitlab.com/runner/executors/docker.html) only). |
| `GET_SOURCES_ATTEMPTS` | Number of attempts to fetch sources running a job |
| `RESTORE_CACHE_ATTEMPTS` | Number of attempts to restore the cache running a job |
The default is one single attempt.
Example:
```yaml
variables:
GET_SOURCES_ATTEMPTS: 3
```
You can set them globally or per-job in the [`variables`](../yaml/README.md#variables) section.
## System calls not available on GitLab.com shared runners
GitLab.com shared runners run on CoreOS. This means that you cannot use some system calls, like `getlogin`, from the C standard library.
## Artifact and cache settings
> Introduced in GitLab Runner 13.9.
Artifact and cache settings control the compression ratio of artifacts and caches.
Use these settings to specify the size of the archive produced by a job.
- On a slow network, uploads might be faster for smaller archives.
- On a fast network where bandwidth and storage are not a concern, uploads might be faster using the fastest compression ratio, despite the archive produced being larger.
For [GitLab Pages](../../user/project/pages/index.md) to serve
[HTTP Range requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests), artifacts
should use the `ARTIFACT_COMPRESSION_LEVEL: fastest` setting, as only uncompressed zip archives
support this feature.
A meter can also be enabled to provide the rate of transfer for uploads and downloads.
```yaml
variables:
# output upload and download progress every 2 seconds
TRANSFER_METER_FREQUENCY: "2s"
# Use fast compression for artifacts, resulting in larger archives
ARTIFACT_COMPRESSION_LEVEL: "fast"
# Use no compression for caches
CACHE_COMPRESSION_LEVEL: "fastest"
```
| Variable | Description |
|---------------------------------|--------------------------------------------------------|
| `TRANSFER_METER_FREQUENCY` | Specify how often to print the meter's transfer rate. It can be set to a duration (for example, `1s` or `1m30s`). A duration of `0` disables the meter (default). When a value is set, the pipeline shows a progress meter for artifact and cache uploads and downloads. |
| `ARTIFACT_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. |
| `CACHE_COMPRESSION_LEVEL` | To adjust compression ratio, set to `fastest`, `fast`, `default`, `slow`, or `slowest`. This setting works with the Fastzip archiver only, so the GitLab Runner feature flag [`FF_USE_FASTZIP`](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) must also be enabled. |
---
stage: Verify
group: Runner
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
type: reference
---
# The scope of runners
Runners are available based on who you want to have access:
- [Shared runners](#shared-runners) are available to all groups and projects in a GitLab instance.
- [Group runners](#group-runners) are available to all projects and subgroups in a group.
- [Specific runners](#specific-runners) are associated with specific projects.
Typically, specific runners are used for one project at a time.
## Shared runners
*Shared runners* are available to every project in a GitLab instance.
Use shared runners when you have multiple jobs with similar requirements. Rather than
having multiple runners idling for many projects, you can have a few runners that handle
multiple projects.
If you are using a self-managed instance of GitLab:
- Your administrator can install and register shared runners by
going to your project's **Settings > CI/CD**, expanding the **Runners** section,
and clicking **Show runner installation instructions**.
These instructions are also available [in the documentation](https://docs.gitlab.com/runner/install/index.html).
- The administrator can also configure a maximum number of shared runner [pipeline minutes for
each group](../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota).
If you are using GitLab.com:
- You can select from a list of [shared runners that GitLab maintains](../../user/gitlab_com/index.md#shared-runners).
- The shared runners consume the [pipelines minutes](../../subscriptions/gitlab_com/index.md#ci-pipeline-minutes)
included with your account.
### Enable shared runners
On GitLab.com, [shared runners](#shared-runners) are enabled in all projects by
default.
On self-managed instances of GitLab, an administrator must [install](https://docs.gitlab.com/runner/install/index.html)
and [register](https://docs.gitlab.com/runner/register/index.html) them.
You can also enable shared runners for individual projects.
To enable shared runners:
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section.
1. Select **Enable shared runners for this project**.
### Disable shared runners
You can disable shared runners for individual projects or for groups.
You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the project
or group.
To disable shared runners for a project:
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section.
1. In the **Shared runners** area, select **Enable shared runners for this project** so the toggle is grayed-out.
Shared runners are automatically disabled for a project:
- If the shared runners setting for the parent group is disabled, and
- If overriding this setting is not permitted at the project level.
To disable shared runners for a group:
1. Go to the group's **Settings > CI/CD** and expand the **Runners** section.
1. In the **Shared runners** area, turn off the **Enable shared runners for this group** toggle.
1. Optionally, to allow shared runners to be enabled for individual projects or subgroups,
click **Allow projects and subgroups to override the group setting**.
NOTE:
To re-enable the shared runners for a group, turn on the
**Enable shared runners for this group** toggle.
Then, an owner or maintainer must explicitly change this setting
for each project subgroup or project.
### How shared runners pick jobs
Shared runners process jobs by using a fair usage queue. This queue prevents
projects from creating hundreds of jobs and using all available
shared runner resources.
The fair usage queue algorithm assigns jobs based on the projects that have the
fewest number of jobs already running on shared runners.
**Example 1**
If these jobs are in the queue:
- Job 1 for Project 1
- Job 2 for Project 1
- Job 3 for Project 1
- Job 4 for Project 2
- Job 5 for Project 2
- Job 6 for Project 3
The fair usage algorithm assigns jobs in this order:
1. Job 1 is first, because it has the lowest job number from projects with no running jobs (that is, all projects).
1. Job 4 is next, because 4 is now the lowest job number from projects with no running jobs (Project 1 has a job running).
1. Job 6 is next, because 6 is now the lowest job number from projects with no running jobs (Projects 1 and 2 have jobs running).
1. Job 2 is next, because, of projects with the lowest number of jobs running (each has 1), it is the lowest job number.
1. Job 5 is next, because Project 1 now has 2 jobs running and Job 5 is the lowest remaining job number between Projects 2 and 3.
1. Finally is Job 3... because it's the only job left.
---
**Example 2**
If these jobs are in the queue:
- Job 1 for Project 1
- Job 2 for Project 1
- Job 3 for Project 1
- Job 4 for Project 2
- Job 5 for Project 2
- Job 6 for Project 3
The fair usage algorithm assigns jobs in this order:
1. Job 1 is chosen first, because it has the lowest job number from projects with no running jobs (that is, all projects).
1. We finish Job 1.
1. Job 2 is next, because, having finished Job 1, all projects have 0 jobs running again, and 2 is the lowest available job number.
1. Job 4 is next, because with Project 1 running a Job, 4 is the lowest number from projects running no jobs (Projects 2 and 3).
1. We finish Job 4.
1. Job 5 is next, because having finished Job 4, Project 2 has no jobs running again.
1. Job 6 is next, because Project 3 is the only project left with no running jobs.
1. Lastly we choose Job 3... because, again, it's the only job left.
## Group runners
Use *Group runners* when you want all projects in a group
to have access to a set of runners.
Group runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue.
### Create a group runner
You can create a group runner for your self-managed GitLab instance or for GitLab.com.
You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group.
To create a group runner:
1. [Install GitLab Runner](https://docs.gitlab.com/runner/install/).
1. Go to the group you want to make the runner work for.
1. Go to **Settings > CI/CD** and expand the **Runners** section.
1. Note the URL and token.
1. [Register the runner](https://docs.gitlab.com/runner/register/).
### View and manage group runners
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/37366/) in GitLab 13.2.
You can view and manage all runners for a group, its subgroups, and projects.
You can do this for your self-managed GitLab instance or for GitLab.com.
You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group.
1. Go to the group where you want to view the runners.
1. Go to **Settings > CI/CD** and expand the **Runners** section.
1. The following fields are displayed.
| Attribute | Description |
| ------------ | ----------- |
| Type | Displays the runner type: `group` or `specific`, together with the optional states `locked` and `paused` |
| Runner token | Token used to identify the runner, and that the runner uses to communicate with the GitLab instance |
| Description | Description given to the runner when it was created |
| Version | GitLab Runner version |
| IP address | IP address of the host on which the runner is registered |
| Projects | The count of projects to which the runner is assigned |
| Jobs | Total of jobs run by the runner |
| Tags | Tags associated with the runner |
| Last contact | Timestamp indicating when the GitLab instance last contacted the runner |
From this page, you can edit, pause, and remove runners from the group, its subgroups, and projects.
### Pause or remove a group runner
You can pause or remove a group runner for your self-managed GitLab instance or for GitLab.com.
You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the group.
1. Go to the group you want to remove or pause the runner for.
1. Go to **Settings > CI/CD** and expand the **Runners** section.
1. Click **Pause** or **Remove runner**.
- If you pause a group runner that is used by multiple projects, the runner pauses for all projects.
- From the group view, you cannot remove a runner that is assigned to more than one project.
You must remove it from each project first.
1. On the confirmation dialog, click **OK**.
## Specific runners
Use *Specific runners* when you want to use runners for specific projects. For example,
when you have:
- Jobs with specific requirements, like a deploy job that requires credentials.
- Projects with a lot of CI activity that can benefit from being separate from other runners.
You can set up a specific runner to be used by multiple projects. Specific runners
must be enabled for each project explicitly.
Specific runners process jobs by using a first in, first out ([FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics))) queue.
NOTE:
Specific runners do not get shared with forked projects automatically.
A fork *does* copy the CI/CD settings of the cloned repository.
### Create a specific runner
You can create a specific runner for your self-managed GitLab instance or for GitLab.com.
You must have the [Owner role](../../user/permissions.md#project-members-permissions) for the project.
To create a specific runner:
1. [Install runner](https://docs.gitlab.com/runner/install/).
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section.
1. Note the URL and token.
1. [Register the runner](https://docs.gitlab.com/runner/register/).
### Enable a specific runner for a specific project
A specific runner is available in the project it was created for. An administrator can
enable a specific runner to apply to additional projects.
- You must have the [Owner role](../../user/permissions.md#group-members-permissions) for the
project.
- The specific runner must not be [locked](#prevent-a-specific-runner-from-being-enabled-for-other-projects).
To enable or disable a specific runner for a project:
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section.
1. Click **Enable for this project** or **Disable for this project**.
### Prevent a specific runner from being enabled for other projects
You can configure a specific runner so it is "locked" and cannot be enabled for other projects.
This setting can be enabled when you first [register a runner](https://docs.gitlab.com/runner/register/),
but can also be changed later.
To lock or unlock a runner:
1. Go to the project's **Settings > CI/CD** and expand the **Runners** section.
1. Find the runner you want to lock or unlock. Make sure it's enabled.
1. Click the pencil button.
1. Check the **Lock to current projects** option.
1. Click **Save changes**.
...@@ -3740,7 +3740,7 @@ Possible values for `when` are: ...@@ -3740,7 +3740,7 @@ Possible values for `when` are:
- `scheduler_failure`: Retry if the scheduler failed to assign the job to a runner. - `scheduler_failure`: Retry if the scheduler failed to assign the job to a runner.
- `data_integrity_failure`: Retry if there is a structural integrity problem detected. - `data_integrity_failure`: Retry if there is a structural integrity problem detected.
You can specify the number of [retry attempts for certain stages of job execution](../runners/README.md#job-stages-attempts) using variables. You can specify the number of [retry attempts for certain stages of job execution](../runners/configure_runners.md#job-stages-attempts) using variables.
### `timeout` ### `timeout`
...@@ -4801,19 +4801,19 @@ You cannot set job-level variables to be pre-filled when you run a pipeline manu ...@@ -4801,19 +4801,19 @@ You cannot set job-level variables to be pre-filled when you run a pipeline manu
You can use [CI/CD variables](../variables/README.md) to configure how the runner processes Git requests: You can use [CI/CD variables](../variables/README.md) to configure how the runner processes Git requests:
- [`GIT_STRATEGY`](../runners/README.md#git-strategy) - [`GIT_STRATEGY`](../runners/configure_runners.md#git-strategy)
- [`GIT_SUBMODULE_STRATEGY`](../runners/README.md#git-submodule-strategy) - [`GIT_SUBMODULE_STRATEGY`](../runners/configure_runners.md#git-submodule-strategy)
- [`GIT_CHECKOUT`](../runners/README.md#git-checkout) - [`GIT_CHECKOUT`](../runners/configure_runners.md#git-checkout)
- [`GIT_CLEAN_FLAGS`](../runners/README.md#git-clean-flags) - [`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags)
- [`GIT_FETCH_EXTRA_FLAGS`](../runners/README.md#git-fetch-extra-flags) - [`GIT_FETCH_EXTRA_FLAGS`](../runners/configure_runners.md#git-fetch-extra-flags)
- [`GIT_DEPTH`](../runners/README.md#shallow-cloning) (shallow cloning) - [`GIT_DEPTH`](../runners/configure_runners.md#shallow-cloning) (shallow cloning)
- [`GIT_CLONE_PATH`](../runners/README.md#custom-build-directories) (custom build directories) - [`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) (custom build directories)
- [`TRANSFER_METER_FREQUENCY`](../runners/README.md#artifact-and-cache-settings) (artifact/cache meter update frequency) - [`TRANSFER_METER_FREQUENCY`](../runners/configure_runners.md#artifact-and-cache-settings) (artifact/cache meter update frequency)
- [`ARTIFACT_COMPRESSION_LEVEL`](../runners/README.md#artifact-and-cache-settings) (artifact archiver compression level) - [`ARTIFACT_COMPRESSION_LEVEL`](../runners/configure_runners.md#artifact-and-cache-settings) (artifact archiver compression level)
- [`CACHE_COMPRESSION_LEVEL`](../runners/README.md#artifact-and-cache-settings) (cache archiver compression level) - [`CACHE_COMPRESSION_LEVEL`](../runners/configure_runners.md#artifact-and-cache-settings) (cache archiver compression level)
You can also use variables to configure how many times a runner You can also use variables to configure how many times a runner
[attempts certain stages of job execution](../runners/README.md#job-stages-attempts). [attempts certain stages of job execution](../runners/configure_runners.md#job-stages-attempts).
## YAML-specific features ## YAML-specific features
......
...@@ -478,7 +478,7 @@ The following GitLab features are used among others: ...@@ -478,7 +478,7 @@ The following GitLab features are used among others:
- [Multi project pipelines](../../ci/multi_project_pipelines.md) - [Multi project pipelines](../../ci/multi_project_pipelines.md)
- [Review Apps](../../ci/review_apps/index.md) - [Review Apps](../../ci/review_apps/index.md)
- [Artifacts](../../ci/yaml/README.md#artifacts) - [Artifacts](../../ci/yaml/README.md#artifacts)
- [Specific runner](../../ci/runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) - [Specific runner](../../ci/runners/runners_scope.md#prevent-a-specific-runner-from-being-enabled-for-other-projects)
- [Pipelines for merge requests](../../ci/merge_request_pipelines/index.md) - [Pipelines for merge requests](../../ci/merge_request_pipelines/index.md)
## Testing ## Testing
......
...@@ -583,7 +583,7 @@ several reasons: ...@@ -583,7 +583,7 @@ several reasons:
- It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`. - It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`.
The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable
[defined by GitLab.com shared runners](../user/gitlab_com/index.md#pre-clone-script). [defined by GitLab.com shared runners](../ci/runners/README.md#pre-clone-script).
The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD variable: The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD variable:
......
...@@ -64,8 +64,8 @@ To make full use of Auto DevOps with Kubernetes, you need: ...@@ -64,8 +64,8 @@ To make full use of Auto DevOps with Kubernetes, you need:
You can configure Docker-based runners to autoscale as well, using You can configure Docker-based runners to autoscale as well, using
[Docker Machine](https://docs.gitlab.com/runner/install/autoscaling.html). [Docker Machine](https://docs.gitlab.com/runner/install/autoscaling.html).
Runners should be registered as [shared runners](../../ci/runners/README.md#shared-runners) Runners should be registered as [shared runners](../../ci/runners/runners_scope.md#shared-runners)
for the entire GitLab instance, or [specific runners](../../ci/runners/README.md#specific-runners) for the entire GitLab instance, or [specific runners](../../ci/runners/runners_scope.md#specific-runners)
that are assigned to specific projects (the default if you've installed the that are assigned to specific projects (the default if you've installed the
GitLab Runner managed application). GitLab Runner managed application).
......
...@@ -571,7 +571,7 @@ automatically generates. ...@@ -571,7 +571,7 @@ automatically generates.
To enable remediation support, the scanning tool _must_ have access to the `Dockerfile` specified by To enable remediation support, the scanning tool _must_ have access to the `Dockerfile` specified by
the [`DOCKERFILE_PATH`](#available-cicd-variables) CI/CD variable. To ensure that the scanning tool the [`DOCKERFILE_PATH`](#available-cicd-variables) CI/CD variable. To ensure that the scanning tool
has access to this has access to this
file, it's necessary to set [`GIT_STRATEGY: fetch`](../../../ci/runners/README.md#git-strategy) in file, it's necessary to set [`GIT_STRATEGY: fetch`](../../../ci/runners/configure_runners.md#git-strategy) in
your `.gitlab-ci.yml` file by following the instructions described in this document's your `.gitlab-ci.yml` file by following the instructions described in this document's
[overriding the container scanning template](#overriding-the-container-scanning-template) section. [overriding the container scanning template](#overriding-the-container-scanning-template) section.
......
...@@ -810,7 +810,7 @@ variables: ...@@ -810,7 +810,7 @@ variables:
### Cloning the project's repository ### Cloning the project's repository
The DAST job does not require the project's repository to be present when running, so by default The DAST job does not require the project's repository to be present when running, so by default
[`GIT_STRATEGY`](../../../ci/runners/README.md#git-strategy) is set to `none`. [`GIT_STRATEGY`](../../../ci/runners/configure_runners.md#git-strategy) is set to `none`.
### Debugging DAST jobs ### Debugging DAST jobs
......
...@@ -177,7 +177,7 @@ is no longer supported. When overriding the template, you must use [`rules`](../ ...@@ -177,7 +177,7 @@ is no longer supported. When overriding the template, you must use [`rules`](../
#### GIT_DEPTH #### GIT_DEPTH
The [`GIT_DEPTH` CI/CD variable](../../../ci/runners/README.md#shallow-cloning) affects Secret Detection. The [`GIT_DEPTH` CI/CD variable](../../../ci/runners/configure_runners.md#shallow-cloning) affects Secret Detection.
The Secret Detection analyzer relies on generating patches between commits to scan content for The Secret Detection analyzer relies on generating patches between commits to scan content for
secrets. If you override the default, ensure the value is greater than 1. If the number of commits secrets. If you override the default, ensure the value is greater than 1. If the number of commits
in an MR is greater than the GIT_DEPTH value, Secret Detection will [fail to detect secrets](#error-couldnt-run-the-gitleaks-command-exit-status-2). in an MR is greater than the GIT_DEPTH value, Secret Detection will [fail to detect secrets](#error-couldnt-run-the-gitleaks-command-exit-status-2).
...@@ -399,7 +399,7 @@ ERRO[2020-11-18T18:05:52Z] object not found ...@@ -399,7 +399,7 @@ ERRO[2020-11-18T18:05:52Z] object not found
[ERRO] [secrets] [2020-11-18T18:05:52Z] ▶ Gitleaks analysis failed: exit status 2 [ERRO] [secrets] [2020-11-18T18:05:52Z] ▶ Gitleaks analysis failed: exit status 2
``` ```
To resolve the issue, set the [`GIT_DEPTH` CI/CD variable](../../../ci/runners/README.md#shallow-cloning) To resolve the issue, set the [`GIT_DEPTH` CI/CD variable](../../../ci/runners/configure_runners.md#shallow-cloning)
to a higher value. To apply this only to the Secret Detection job, the following can be added to to a higher value. To apply this only to the Secret Detection job, the following can be added to
your `.gitlab-ci.yml` file: your `.gitlab-ci.yml` file:
......
...@@ -157,285 +157,13 @@ A limit of: ...@@ -157,285 +157,13 @@ A limit of:
- 100 webhooks applies to projects. - 100 webhooks applies to projects.
- 50 webhooks applies to groups. **(BRONZE ONLY)** - 50 webhooks applies to groups. **(BRONZE ONLY)**
- Payload is limited to 25MB - Payload is limited to 25MB.
## Shared runners ## Shared runners
GitLab offers Linux and Windows shared runners hosted on GitLab.com for executing your pipelines. GitLab has shared runners on GitLab.com that you can use to run your CI jobs.
NOTE: For more information, see [choosing a runner](../../ci/runners/README.md).
Shared runners provided by GitLab are **not** configurable. Consider [installing your own runner](https://docs.gitlab.com/runner/install/) if you have specific configuration needs.
### Linux shared runners
Linux shared runners on GitLab.com run in autoscale mode and are powered by Google Cloud Platform.
Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each project, thus maximizing security. These shared runners are available for users and customers on GitLab.com.
GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier.
All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, CoreOS and the latest Docker Engine
installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default
region of the VMs is US East1.
Each instance is used only for one job, this ensures any sensitive data left on the system can't be accessed by other people their CI jobs.
The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times.
Jobs handled by the shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`),
**time out after 3 hours**, regardless of the timeout configured in a
project. Check the issues [4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference.
Below are the shared runners settings.
| Setting | GitLab.com | Default |
| ----------- | ----------------- | ---------- |
| [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) | [Runner versions dashboard](https://dashboards.gitlab.com/d/000000159/ci?from=now-1h&to=now&refresh=5m&orgId=1&panelId=12&fullscreen&theme=light) | - |
| Executor | `docker+machine` | - |
| Default Docker image | `ruby:2.5` | - |
| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true` | `false` |
#### Pre-clone script
Linux shared runners on GitLab.com provide a way to run commands in a CI
job before the runner attempts to run `git init` and `git fetch` to
download a GitLab repository. The
[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section)
can be used for:
- Seeding the build directory with repository data
- Sending a request to a server
- Downloading assets from a CDN
- Any other commands that must run before the `git init`
To use this feature, define a [CI/CD variable](../../ci/variables/README.md#custom-cicd-variables) called
`CI_PRE_CLONE_SCRIPT` that contains a bash script.
[This example](../../development/pipelines.md#pre-clone-step)
demonstrates how you might use a pre-clone step to seed the build
directory.
#### `config.toml`
The full contents of our `config.toml` are:
NOTE:
Settings that are not public are shown as `X`.
**Google Cloud Platform**
```toml
concurrent = X
check_interval = 1
metrics_server = "X"
sentry_dsn = "X"
[[runners]]
name = "docker-auto-scale"
request_concurrency = X
url = "https://gitlab.com/"
token = "SHARED_RUNNER_TOKEN"
pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\""
executor = "docker+machine"
environment = [
"DOCKER_DRIVER=overlay2",
"DOCKER_TLS_CERTDIR="
]
limit = X
[runners.docker]
image = "ruby:2.5"
privileged = true
volumes = [
"/certs/client",
"/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP.
]
[runners.machine]
IdleCount = 50
IdleTime = 3600
MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused.
MachineName = "srm-%s"
MachineDriver = "google"
MachineOptions = [
"google-project=PROJECT",
"google-disk-size=25",
"google-machine-type=n1-standard-1",
"google-username=core",
"google-tags=gitlab-com,srm",
"google-use-internal-ip",
"google-zone=us-east1-d",
"engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928
"google-machine-image=PROJECT/global/images/IMAGE",
"engine-opt=ipv6", # This will create IPv6 interfaces in the containers.
"engine-opt=fixed-cidr-v6=fc00::/7",
"google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600
]
[[runners.machine.autoscaling]]
Periods = ["* * * * * sat,sun *"]
Timezone = "UTC"
IdleCount = 70
IdleTime = 3600
[[runners.machine.autoscaling]]
Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"]
Timezone = "UTC"
IdleCount = 700
IdleTime = 3600
[runners.cache]
Type = "gcs"
Shared = true
[runners.cache.gcs]
CredentialsFile = "/path/to/file"
BucketName = "bucket-name"
```
### Windows shared runners (beta)
The Windows shared runners are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta)
and shouldn't be used for production workloads.
During this beta period, the [shared runner pipeline quota](../admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota)
applies for groups and projects in the same manner as Linux runners. This may
change when the beta period ends, as discussed in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30834).
Windows shared runners on GitLab.com autoscale by launching virtual machines on
the Google Cloud Platform. This solution uses an
[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md)
developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html).
Windows shared runners execute your CI/CD jobs on `n1-standard-2` instances with
2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in
the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/master/cookbooks/preinstalled-software/README.md).
We want to keep iterating to get Windows shared runners in a stable state and
[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga).
You can follow our work towards this goal in the
[related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162).
#### Configuration
The full contents of our `config.toml` are:
NOTE:
Settings that aren't public are shown as `X`.
```toml
concurrent = X
check_interval = 3
[[runners]]
name = "windows-runner"
url = "https://gitlab.com/"
token = "TOKEN"
executor = "custom"
builds_dir = "C:\\GitLab-Runner\\builds"
cache_dir = "C:\\GitLab-Runner\\cache"
shell = "powershell"
[runners.custom]
config_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe"
config_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "config"]
prepare_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe"
prepare_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "prepare"]
run_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe"
run_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "run"]
cleanup_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe"
cleanup_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "cleanup"]
```
The full contents of our `autoscaler/config.toml` are:
```toml
Provider = "gcp"
Executor = "winrm"
OS = "windows"
LogLevel = "info"
LogFormat = "text"
LogFile = "C:\\GitLab-Runner\\autoscaler\\autoscaler.log"
VMTag = "windows"
[GCP]
ServiceAccountFile = "PATH"
Project = "some-project-df9323"
Zone = "us-east1-c"
MachineType = "n1-standard-2"
Image = "IMAGE"
DiskSize = 50
DiskType = "pd-standard"
Subnetwork = "default"
Network = "default"
Tags = ["TAGS"]
Username = "gitlab_runner"
[WinRM]
MaximumTimeout = 3600
ExecutionMaxRetries = 0
[ProviderCache]
Enabled = true
Directory = "C:\\GitLab-Runner\\autoscaler\\machines"
```
#### Example
Below is a simple `.gitlab-ci.yml` file to show how to start using the
Windows shared runners:
```yaml
.shared_windows_runners:
tags:
- shared-windows
- windows
- windows-1809
stages:
- build
- test
before_script:
- Set-Variable -Name "time" -Value (date -Format "%H:%m")
- echo ${time}
- echo "started by ${GITLAB_USER_NAME}"
build:
extends:
- .shared_windows_runners
stage: build
script:
- echo "running scripts in the build job"
test:
extends:
- .shared_windows_runners
stage: test
script:
- echo "running scripts in the test job"
```
#### Limitations and known issues
- All the limitations mentioned in our [beta
definition](https://about.gitlab.com/handbook/product/#beta).
- The average provisioning time for a new Windows VM is 5 minutes.
This means that you may notice slower build start times
on the Windows shared runner fleet during the beta. In a future
release we intend to update the autoscaler to enable
the pre-provisioning of virtual machines. This is intended to significantly reduce
the time it takes to provision a VM on the Windows fleet. You can
follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32).
- The Windows shared runner fleet may be unavailable occasionally
for maintenance or updates.
- The Windows shared runner virtual machine instances do not use the
GitLab Docker executor. This means that you can't specify
[`image`](../../ci/yaml/README.md#image) or [`services`](../../ci/yaml/README.md#services) in
your pipeline configuration.
- For the beta release, we have included a set of software packages in
the base VM image. If your CI job requires additional software that's
not included in this list, then you must add installation
commands to [`before_script`](../../ci/yaml/README.md#before_script) or [`script`](../../ci/yaml/README.md#script) to install the required
software. Note that each job runs on a new VM instance, so the
installation of additional software packages needs to be repeated for
each job in your pipeline.
- The job may stay in a pending state for longer than the
Linux shared runners.
- There is the possibility that we introduce breaking changes which will
require updates to pipelines that are using the Windows shared runner
fleet.
## Sidekiq ## Sidekiq
......
...@@ -93,7 +93,7 @@ template that is included with GitLab. ...@@ -93,7 +93,7 @@ template that is included with GitLab.
NOTE: NOTE:
For large scale k6 tests you need to ensure the GitLab Runner instance performing the actual For large scale k6 tests you need to ensure the GitLab Runner instance performing the actual
test is able to handle running the test. Refer to [k6's guidance](https://k6.io/docs/testing-guides/running-large-tests#hardware-considerations) test is able to handle running the test. Refer to [k6's guidance](https://k6.io/docs/testing-guides/running-large-tests#hardware-considerations)
for spec details. The [default shared GitLab.com runners](../../gitlab_com/#linux-shared-runners) for spec details. The [default shared GitLab.com runners](../../../ci/runners/README.md#linux-shared-runners)
likely have insufficient specs to handle most large k6 tests. likely have insufficient specs to handle most large k6 tests.
This template runs the This template runs the
......
...@@ -324,4 +324,4 @@ pages: ...@@ -324,4 +324,4 @@ pages:
- public - public
``` ```
The `FF_USE_FASTZIP` variable enables the [feature flag](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) which is needed for [`ARTIFACT_COMPRESSION_LEVEL`](../../../ci/runners/README.md#artifact-and-cache-settings). The `FF_USE_FASTZIP` variable enables the [feature flag](https://docs.gitlab.com/runner/configuration/feature-flags.html#available-feature-flags) which is needed for [`ARTIFACT_COMPRESSION_LEVEL`](../../../ci/runners/configure_runners.md#artifact-and-cache-settings).
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment