Commit 7c2ccab5 authored by Evan Read's avatar Evan Read

Merge branch 'selhorn-runner-12' into 'master'

Docs: Updated runner capitalization

See merge request gitlab-org/gitlab!42247
parents f2f8d770 07cf7bea
...@@ -173,7 +173,7 @@ Table description links: ...@@ -173,7 +173,7 @@ Table description links:
| [Gitaly](#gitaly) | Git RPC service for handling all Git calls made by GitLab | ✅ | ✅ | ✅ | ✅ | ⚙ | ✅ | CE & EE | | [Gitaly](#gitaly) | Git RPC service for handling all Git calls made by GitLab | ✅ | ✅ | ✅ | ✅ | ⚙ | ✅ | CE & EE |
| [GitLab Exporter](#gitlab-exporter) | Generates a variety of GitLab metrics | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | CE & EE | | [GitLab Exporter](#gitlab-exporter) | Generates a variety of GitLab metrics | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | CE & EE |
| [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | ⚙ | ⚙ | ❌ | ✅ | ❌ | ⚙ | EE Only | | [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | ⚙ | ⚙ | ❌ | ✅ | ❌ | ⚙ | EE Only |
| [GitLab Managed Apps](#gitlab-managed-apps) | Deploy Helm, Ingress, Cert-Manager, Prometheus, a Runner, JupyterHub, or Knative to a cluster | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | CE & EE | | [GitLab Managed Apps](#gitlab-managed-apps) | Deploy Helm, Ingress, Cert-Manager, Prometheus, GitLab Runner, JupyterHub, or Knative to a cluster | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | CE & EE |
| [GitLab Pages](#gitlab-pages) | Hosts static websites | ⚙ | ❌ | ❌ | ✅ | ⚙ | ⚙ | CE & EE | | [GitLab Pages](#gitlab-pages) | Hosts static websites | ⚙ | ❌ | ❌ | ✅ | ⚙ | ⚙ | CE & EE |
| [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | ⚙ | ✅ | ⚙ | ✅ | ❌ | ❌ | CE & EE | | [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | ⚙ | ✅ | ⚙ | ✅ | ❌ | ❌ | CE & EE |
| [GitLab self-monitoring: Grafana](#grafana) | Metrics dashboard | ✅ | ⚙ | ⤓ | ✅ | ❌ | ❌ | CE & EE | | [GitLab self-monitoring: Grafana](#grafana) | Metrics dashboard | ✅ | ⚙ | ⤓ | ✅ | ❌ | ❌ | CE & EE |
...@@ -665,7 +665,7 @@ You can install them after you create a cluster. This includes: ...@@ -665,7 +665,7 @@ You can install them after you create a cluster. This includes:
- [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) - [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/)
- [Cert-Manager](https://cert-manager.io/docs/) - [Cert-Manager](https://cert-manager.io/docs/)
- [Prometheus](https://prometheus.io/docs/introduction/overview/) - [Prometheus](https://prometheus.io/docs/introduction/overview/)
- a [Runner](https://docs.gitlab.com/runner/) - [GitLab Runner](https://docs.gitlab.com/runner/)
- [JupyterHub](https://jupyter.org) - [JupyterHub](https://jupyter.org)
- [Knative](https://cloud.google.com/knative/) - [Knative](https://cloud.google.com/knative/)
......
...@@ -53,25 +53,25 @@ The component that processes a pipeline is [`ProcessPipelineService`](https://gi ...@@ -53,25 +53,25 @@ The component that processes a pipeline is [`ProcessPipelineService`](https://gi
which is responsible for moving all the pipeline's jobs to a completed state. When a pipeline is created, all its which is responsible for moving all the pipeline's jobs to a completed state. When a pipeline is created, all its
jobs are initially in `created` state. This services looks at what jobs in `created` stage are eligible jobs are initially in `created` state. This services looks at what jobs in `created` stage are eligible
to be processed based on the pipeline structure. Then it moves them into the `pending` state, which means to be processed based on the pipeline structure. Then it moves them into the `pending` state, which means
they can now [be picked up by a Runner](#job-scheduling). After a job has been executed it can complete they can now [be picked up by a runner](#job-scheduling). After a job has been executed it can complete
successfully or fail. Each status transition for job within a pipeline triggers this service again, which successfully or fail. Each status transition for job within a pipeline triggers this service again, which
looks for the next jobs to be transitioned towards completion. While doing that, `ProcessPipelineService` looks for the next jobs to be transitioned towards completion. While doing that, `ProcessPipelineService`
updates the status of jobs, stages and the overall pipeline. updates the status of jobs, stages and the overall pipeline.
On the right side of the diagram we have a list of [Runners](../../ci/runners/README.md) On the right side of the diagram we have a list of [runners](../../ci/runners/README.md)
connected to the GitLab instance. These can be Shared Runners, Group Runners or Project-specific Runners. connected to the GitLab instance. These can be shared runners, group runners, or project-specific runners.
The communication between Runners and the Rails server occurs through a set of API endpoints, grouped as The communication between runners and the Rails server occurs through a set of API endpoints, grouped as
the `Runner API Gateway`. the `Runner API Gateway`.
We can register, delete and verify Runners, which also causes read/write queries to the database. After a Runner is connected, We can register, delete, and verify runners, which also causes read/write queries to the database. After a runner is connected,
it keeps asking for the next job to execute. This invokes the [`RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/ci/register_job_service.rb) it keeps asking for the next job to execute. This invokes the [`RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/ci/register_job_service.rb)
which will pick the next job and assign it to the Runner. At this point the job will transition to a which will pick the next job and assign it to the runner. At this point the job will transition to a
`running` state, which again triggers `ProcessPipelineService` due to the status change. `running` state, which again triggers `ProcessPipelineService` due to the status change.
For more details read [Job scheduling](#job-scheduling)). For more details read [Job scheduling](#job-scheduling)).
While a job is being executed, the Runner sends logs back to the server as well any possible artifacts While a job is being executed, the runner sends logs back to the server as well any possible artifacts
that need to be stored. Also, a job may depend on artifacts from previous jobs in order to run. In this that need to be stored. Also, a job may depend on artifacts from previous jobs in order to run. In this
case the Runner will download them using a dedicated API endpoint. case the runner will download them using a dedicated API endpoint.
Artifacts are stored in object storage, while metadata is kept in the database. An important example of artifacts Artifacts are stored in object storage, while metadata is kept in the database. An important example of artifacts
are reports (JUnit, SAST, DAST, etc.) which are parsed and rendered in the merge request. are reports (JUnit, SAST, DAST, etc.) which are parsed and rendered in the merge request.
...@@ -90,25 +90,25 @@ from the `CreatePipelineService` every time a downstream pipeline is triggered. ...@@ -90,25 +90,25 @@ from the `CreatePipelineService` every time a downstream pipeline is triggered.
When a Pipeline is created all its jobs are created at once for all stages, with an initial state of `created`. This makes it possible to visualize the full content of a pipeline. When a Pipeline is created all its jobs are created at once for all stages, with an initial state of `created`. This makes it possible to visualize the full content of a pipeline.
A job with the `created` state won't be seen by the Runner yet. To make it possible to assign a job to a Runner, the job must transition first into the `pending` state, which can happen if: A job with the `created` state won't be seen by the runner yet. To make it possible to assign a job to a runner, the job must transition first into the `pending` state, which can happen if:
1. The job is created in the very first stage of the pipeline. 1. The job is created in the very first stage of the pipeline.
1. The job required a manual start and it has been triggered. 1. The job required a manual start and it has been triggered.
1. All jobs from the previous stage have completed successfully. In this case we transition all jobs from the next stage to `pending`. 1. All jobs from the previous stage have completed successfully. In this case we transition all jobs from the next stage to `pending`.
1. The job specifies DAG dependencies using `needs:` and all the dependent jobs are completed. 1. The job specifies DAG dependencies using `needs:` and all the dependent jobs are completed.
When the Runner is connected, it requests the next `pending` job to run by polling the server continuously. When the runner is connected, it requests the next `pending` job to run by polling the server continuously.
NOTE: **Note:** NOTE: **Note:**
API endpoints used by the Runner to interact with GitLab are defined in [`lib/api/ci/runner.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/ci/runner.rb) API endpoints used by the runner to interact with GitLab are defined in [`lib/api/ci/runner.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/api/ci/runner.rb)
After the server receives the request it selects a `pending` job based on the [`Ci::RegisterJobService` algorithm](#ciregisterjobservice), then assigns and sends the job to the Runner. After the server receives the request it selects a `pending` job based on the [`Ci::RegisterJobService` algorithm](#ciregisterjobservice), then assigns and sends the job to the runner.
Once all jobs are completed for the current stage, the server "unlocks" all the jobs from the next stage by changing their state to `pending`. These can now be picked by the scheduling algorithm when the Runner requests new jobs, and continues like this until all stages are completed. Once all jobs are completed for the current stage, the server "unlocks" all the jobs from the next stage by changing their state to `pending`. These can now be picked by the scheduling algorithm when the runner requests new jobs, and continues like this until all stages are completed.
### Communication between Runner and GitLab server ### Communication between runner and GitLab server
Once the Runner is [registered](https://docs.gitlab.com/runner/register/) using the registration token, the server knows what type of jobs it can execute. This depends on: Once the runner is [registered](https://docs.gitlab.com/runner/register/) using the registration token, the server knows what type of jobs it can execute. This depends on:
- The type of runner it is registered as: - The type of runner it is registered as:
- a shared runner - a shared runner
...@@ -116,30 +116,30 @@ Once the Runner is [registered](https://docs.gitlab.com/runner/register/) using ...@@ -116,30 +116,30 @@ Once the Runner is [registered](https://docs.gitlab.com/runner/register/) using
- a project specific runner - a project specific runner
- Any associated tags. - Any associated tags.
The Runner initiates the communication by requesting jobs to execute with `POST /api/v4/jobs/request`. Although this polling generally happens every few seconds we leverage caching via HTTP headers to reduce the server-side work load if the job queue doesn't change. The runner initiates the communication by requesting jobs to execute with `POST /api/v4/jobs/request`. Although this polling generally happens every few seconds we leverage caching via HTTP headers to reduce the server-side work load if the job queue doesn't change.
This API endpoint runs [`Ci::RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/ci/register_job_service.rb), which: This API endpoint runs [`Ci::RegisterJobService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/ci/register_job_service.rb), which:
1. Picks the next job to run from the pool of `pending` jobs 1. Picks the next job to run from the pool of `pending` jobs
1. Assigns it to the Runner 1. Assigns it to the runner
1. Presents it to the Runner via the API response 1. Presents it to the runner via the API response
### `Ci::RegisterJobService` ### `Ci::RegisterJobService`
There are 3 top level queries that this service uses to gather the majority of the jobs and they are selected based on the level where the Runner is registered to: There are 3 top level queries that this service uses to gather the majority of the jobs and they are selected based on the level where the runner is registered to:
- Select jobs for shared Runner (instance level) - Select jobs for shared runner (instance level)
- Select jobs for group level Runner - Select jobs for group runner
- Select jobs for project Runner - Select jobs for project runner
This list of jobs is then filtered further by matching tags between job and Runner tags. This list of jobs is then filtered further by matching tags between job and runner tags.
NOTE: **Note:** NOTE: **Note:**
If a job contains tags, the Runner will not pick the job if it does not match **all** the tags. If a job contains tags, the runner will not pick the job if it does not match **all** the tags.
The Runner may have more tags than defined for the job, but not vice-versa. The runner may have more tags than defined for the job, but not vice-versa.
Finally if the Runner can only pick jobs that are tagged, all untagged jobs are filtered out. Finally if the runner can only pick jobs that are tagged, all untagged jobs are filtered out.
At this point we loop through remaining `pending` jobs and we try to assign the first job that the Runner "can pick" based on additional policies. For example, Runners marked as `protected` can only pick jobs that run against protected branches (such as production deployments). At this point we loop through remaining `pending` jobs and we try to assign the first job that the runner "can pick" based on additional policies. For example, runners marked as `protected` can only pick jobs that run against protected branches (such as production deployments).
As we increase the number of Runners in the pool we also increase the chances of conflicts which would arise if assigning the same job to different Runners. To prevent that we gracefully rescue conflict errors and assign the next job in the list. As we increase the number of runners in the pool we also increase the chances of conflicts which would arise if assigning the same job to different runners. To prevent that we gracefully rescue conflict errors and assign the next job in the list.
...@@ -455,7 +455,7 @@ If you want to know the in-depth details, here's what's really happening: ...@@ -455,7 +455,7 @@ If you want to know the in-depth details, here's what's really happening:
[skips the test jobs](https://gitlab.com/gitlab-org/gitlab-docs/blob/8d5d5c750c602a835614b02f9db42ead1c4b2f5e/.gitlab-ci.yml#L50-55) [skips the test jobs](https://gitlab.com/gitlab-org/gitlab-docs/blob/8d5d5c750c602a835614b02f9db42ead1c4b2f5e/.gitlab-ci.yml#L50-55)
to lower the build time. to lower the build time.
1. Once the docs site is built, the HTML files are uploaded as artifacts. 1. Once the docs site is built, the HTML files are uploaded as artifacts.
1. A specific Runner tied only to the docs project, runs the Review App job 1. A specific runner tied only to the docs project, runs the Review App job
that downloads the artifacts and uses `rsync` to transfer the files over that downloads the artifacts and uses `rsync` to transfer the files over
to a location where NGINX serves them. to a location where NGINX serves them.
...@@ -465,7 +465,7 @@ The following GitLab features are used among others: ...@@ -465,7 +465,7 @@ The following GitLab features are used among others:
- [Multi project pipelines](../../ci/multi_project_pipeline_graphs.md) - [Multi project pipelines](../../ci/multi_project_pipeline_graphs.md)
- [Review Apps](../../ci/review_apps/index.md) - [Review Apps](../../ci/review_apps/index.md)
- [Artifacts](../../ci/yaml/README.md#artifacts) - [Artifacts](../../ci/yaml/README.md#artifacts)
- [Specific Runner](../../ci/runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) - [Specific runner](../../ci/runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects)
- [Pipelines for merge requests](../../ci/merge_request_pipelines/index.md) - [Pipelines for merge requests](../../ci/merge_request_pipelines/index.md)
## Testing ## Testing
......
...@@ -1391,14 +1391,14 @@ interface: ...@@ -1391,14 +1391,14 @@ interface:
```markdown ```markdown
| Section | Description | | Section | Description |
|:-------------------------|:----------------------------------------------------------------------------------------------------------------------------| |:-------------------------|:----------------------------------------------------------------------------------------------------------------------------|
| **{overview}** Overview | View your GitLab Dashboard, and administer projects, users, groups, jobs, Runners, and Gitaly servers. | | **{overview}** Overview | View your GitLab Dashboard, and administer projects, users, groups, jobs, runners, and Gitaly servers. |
| **{monitor}** Monitoring | View GitLab system information, and information on background jobs, logs, health checks, requests profiles, and audit logs. | | **{monitor}** Monitoring | View GitLab system information, and information on background jobs, logs, health checks, requests profiles, and audit logs. |
| **{messages}** Messages | Send and manage broadcast messages for your users. | | **{messages}** Messages | Send and manage broadcast messages for your users. |
``` ```
| Section | Description | | Section | Description |
|:-------------------------|:----------------------------------------------------------------------------------------------------------------------------| |:-------------------------|:----------------------------------------------------------------------------------------------------------------------------|
| **{overview}** Overview | View your GitLab Dashboard, and administer projects, users, groups, jobs, Runners, and Gitaly servers. | | **{overview}** Overview | View your GitLab Dashboard, and administer projects, users, groups, jobs, runners, and Gitaly servers. |
| **{monitor}** Monitoring | View GitLab system information, and information on background jobs, logs, health checks, requests profiles, and audit logs. | | **{monitor}** Monitoring | View GitLab system information, and information on background jobs, logs, health checks, requests profiles, and audit logs. |
| **{messages}** Messages | Send and manage broadcast messages for your users. | | **{messages}** Messages | Send and manage broadcast messages for your users. |
...@@ -1974,8 +1974,8 @@ You can use the following fake tokens as examples: ...@@ -1974,8 +1974,8 @@ You can use the following fake tokens as examples:
| Application ID | `2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6` | | Application ID | `2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6` |
| Application secret | `04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df` | | Application secret | `04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df` |
| CI/CD variable | `Li8j-mLUVA3eZYjPfd_H` | | CI/CD variable | `Li8j-mLUVA3eZYjPfd_H` |
| Specific Runner token | `yrnZW46BrtBFqM7xDzE7dddd` | | Specific runner token | `yrnZW46BrtBFqM7xDzE7dddd` |
| Shared Runner token | `6Vk7ZsosqQyfreAxXTZr` | | Shared runner token | `6Vk7ZsosqQyfreAxXTZr` |
| Trigger token | `be20d8dcc028677c931e04f3871a9b` | | Trigger token | `be20d8dcc028677c931e04f3871a9b` |
| Webhook secret token | `6XhDroRcYPM5by_h-HLY` | | Webhook secret token | `6XhDroRcYPM5by_h-HLY` |
| Health check token | `Tu7BgjR9qeZTEyRzGG2P` | | Health check token | `Tu7BgjR9qeZTEyRzGG2P` |
......
...@@ -87,7 +87,7 @@ You should now be remoted into a Windows machine with a command prompt. ...@@ -87,7 +87,7 @@ You should now be remoted into a Windows machine with a command prompt.
- Start the runner: `gitlab-runner.exe start`. - Start the runner: `gitlab-runner.exe start`.
For more information, see [Install GitLab Runner on Windows](https://docs.gitlab.com/runner/install/windows.html) For more information, see [Install GitLab Runner on Windows](https://docs.gitlab.com/runner/install/windows.html)
and [Registering Runners](https://docs.gitlab.com/runner/register/index.html). and [Registering runners](https://docs.gitlab.com/runner/register/index.html).
## Developer tips ## Developer tips
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment