Skip to content

G
gitlab-ce
Commit 1d02d0de authored by Kati Paizee's avatar Kati Paizee
Browse files
Options

Merge branch 'docs-update-needs-keyword' into 'master' 

Update the needs keyword to new reference style

See merge request gitlab-org/gitlab!74497
parents e34fac95 25c645db
Hide whitespace changes
Inline Side-by-side
Showing with 199 additions and 172 deletions
doc/administration/cicd.md
View file @ 1d02d0de
......@@ -7,22 +7,18 @@ type: howto
# GitLab CI/CD instance configuration **(FREE SELF)**
GitLab CI/CD is enabled by default in all new projects on an instance. You can configure
the instance to have [GitLab CI/CD disabled by default](#disable-gitlab-cicd-in-new-projects)
in new projects.
You can still choose to [enable GitLab CI/CD in individual projects](../ci/enable_or_disable_ci.md#enable-cicd-in-a-project)
at any time.
GitLab administrators can manage the GitLab CI/CD configuration for their instance.
## Disable GitLab CI/CD in new projects
You can set GitLab CI/CD to be disabled by default in all new projects by modifying the settings in:
GitLab CI/CD is enabled by default in all new projects on an instance. You can set
CI/CD to be disabled by default in new projects by modifying the settings in:
- `gitlab.yml` for source installations.
- `gitlab.rb` for Omnibus GitLab installations.
Existing projects that already had CI/CD enabled are unchanged. Also, this setting only changes
the project default, so project owners can still enable CI/CD in the project settings.
the project default, so project owners [can still enable CI/CD in the project settings](../ci/enable_or_disable_ci.md#enable-cicd-in-a-project).
For installations from source:
......@@ -62,6 +58,19 @@ For Omnibus GitLab installations:
sudo gitlab-ctl reconfigure
```
## Set the `needs:` job limit **(FREE SELF)**
The maximum number of jobs that can be defined in `needs:` defaults to 50.
A GitLab administrator with [access to the GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session)
can choose a custom limit. For example, to set the limit to `100`:
```ruby
Plan.default.actual_limits.update!(ci_needs_size_limit: 100)
```
To disable directed acyclic graphs (DAG), set the limit to `0`.
<!-- ## Troubleshooting
Include any troubleshooting steps that you can foresee. If you know beforehand what issues
......
doc/ci/directed_acyclic_graph/index.md
View file @ 1d02d0de
......@@ -74,9 +74,9 @@ giving you powerful options for parallelization within your pipeline.
## Limitations
A directed acyclic graph is a complicated feature, and as of the initial MVC there
are certain use cases that you may need to work around. For more information:
are certain use cases that you may need to work around. For more information, check the:
- [`needs` requirements and limitations](../yaml/index.md#requirements-and-limitations).
- [`needs` additional details](../yaml/index.md#needs).
- Related epic [tracking planned improvements](https://gitlab.com/groups/gitlab-org/-/epics/1716).
## Needs Visualization
......
doc/ci/pipelines/multi_project_pipelines.md
View file @ 1d02d0de
......@@ -88,7 +88,7 @@ The keywords available for use in trigger jobs are:
- [`only` and `except`](../yaml/index.md#only--except)
- [`when`](../yaml/index.md#when) (only with a value of `on_success`, `on_failure`, or `always`)
- [`extends`](../yaml/index.md#extends)
- [`needs`](../yaml/index.md#needs), but not [cross project artifact downloads with `needs`](../yaml/index.md#cross-project-artifact-downloads-with-needs)
- [`needs`](../yaml/index.md#needs), but not [`needs:project`](../yaml/index.md#needsproject)
#### Specify a downstream pipeline branch
......@@ -190,7 +190,7 @@ the ones defined in the upstream project take precedence.
#### Pass CI/CD variables to a downstream pipeline by using variable inheritance
You can pass variables to a downstream pipeline with [`dotenv` variable inheritance](../variables/index.md#pass-an-environment-variable-to-another-job) and [cross project artifact downloads](../yaml/index.md#cross-project-artifact-downloads-with-needs).
You can pass variables to a downstream pipeline with [`dotenv` variable inheritance](../variables/index.md#pass-an-environment-variable-to-another-job) and [`needs:project`](../yaml/index.md#needsproject).
In the upstream pipeline:
......@@ -254,21 +254,6 @@ trigger_job:
strategy: depend
```
#### Mirror status from upstream pipeline
You can mirror the pipeline status from an upstream pipeline to a bridge job by
using the `needs:pipeline` keyword. The latest pipeline status from the default branch is
replicated to the bridge job.
For example:
```yaml
upstream_bridge:
stage: test
needs:
pipeline: other/project
```
### Create multi-project pipelines by using the API
> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) to GitLab Free in 12.4.
......
doc/ci/variables/index.md
View file @ 1d02d0de
......@@ -557,7 +557,7 @@ they can be used in job scripts.
1. Save the `.env` file as an [`artifacts:reports:dotenv`](../yaml/index.md#artifactsreportsdotenv)
artifact.
1. Set a job in a later stage to receive the artifact by using the [`dependencies`](../yaml/index.md#dependencies)
or the [`needs`](../yaml/index.md#artifact-downloads-with-needs) keywords.
or the [`needs`](../yaml/index.md#needs) keywords.
1. The later job can then [use the variable in scripts](#use-cicd-variables-in-job-scripts).
For example, with the [`dependencies`](../yaml/index.md#dependencies) keyword:
......@@ -579,7 +579,7 @@ deploy:
- build
```
For example, with the [`needs`](../yaml/index.md#artifact-downloads-with-needs) keyword:
For example, with the [`needs:artifacts`](../yaml/index.md#needsartifacts) keyword:
```yaml
build:
......
doc/ci/yaml/index.md
View file @ 1d02d0de
......@@ -1364,16 +1364,14 @@ that use `needs` can be visualized as a [directed acyclic graph](../directed_acy
You can ignore stage ordering and run some jobs without waiting for others to complete.
Jobs in multiple stages can run concurrently.
The following example creates four paths of execution:
**Keyword type**: Job keyword. You can use it only as part of a job.
- Linter: the `lint` job runs immediately without waiting for the `build` stage
to complete because it has no needs (`needs: []`).
- Linux path: the `linux:rspec` and `linux:rubocop` jobs runs as soon as the `linux:build`
job finishes without waiting for `mac:build` to finish.
- macOS path: the `mac:rspec` and `mac:rubocop` jobs runs as soon as the `mac:build`
job finishes, without waiting for `linux:build` to finish.
- The `production` job runs as soon as all previous jobs finish; in this case:
`linux:build`, `linux:rspec`, `linux:rubocop`, `mac:build`, `mac:rspec`, `mac:rubocop`.
**Possible inputs**:
- An array of jobs.
- An empty array (`[]`), to set the job to start as soon as the pipeline is created.
**Example of `needs`**:
```yaml
linux:build:
......@@ -1394,32 +1392,33 @@ linux:rspec:
needs: ["linux:build"]
script: echo "Running rspec on linux..."
linux:rubocop:
stage: test
needs: ["linux:build"]
script: echo "Running rubocop on linux..."
mac:rspec:
stage: test
needs: ["mac:build"]
script: echo "Running rspec on mac..."
mac:rubocop:
stage: test
needs: ["mac:build"]
script: echo "Running rubocop on mac..."
production:
stage: deploy
script: echo "Running production..."
```
#### Requirements and limitations
This example creates four paths of execution:
- Linter: The `lint` job runs immediately without waiting for the `build` stage
to complete because it has no needs (`needs: []`).
- Linux path: The `linux:rspec` job runs as soon as the `linux:build`
job finishes, without waiting for `mac:build` to finish.
- macOS path: The `mac:rspec` jobs runs as soon as the `mac:build`
job finishes, without waiting for `linux:build` to finish.
- The `production` job runs as soon as all previous jobs finish:
`linux:build`, `linux:rspec`, `mac:build`, `mac:rspec`.
**Additional details**:
- The maximum number of jobs that a single job can need in the `needs:` array is limited:
- The maximum number of jobs that a single job can have in the `needs:` array is limited:
- For GitLab.com, the limit is 50. For more information, see our
[infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7541).
- For self-managed instances, the default limit is 50. This limit [can be changed](#changing-the-needs-job-limit).
- For self-managed instances, the default limit is 50. This limit [can be changed](../../administration/cicd.md#set-the-needs-job-limit).
- If `needs:` refers to a job that uses the [`parallel`](#parallel) keyword,
it depends on all jobs created in parallel, not just one job. It also downloads
artifacts from all the parallel jobs by default. If the artifacts have the same
......@@ -1434,20 +1433,7 @@ production:
- In GitLab 13.9 and older, if `needs:` refers to a job that might not be added to
a pipeline because of `only`, `except`, or `rules`, the pipeline might fail to create.
##### Changing the `needs:` job limit **(FREE SELF)**
The maximum number of jobs that can be defined in `needs:` defaults to 50.
A GitLab administrator with [access to the GitLab Rails console](../../administration/feature_flags.md)
can choose a custom limit. For example, to set the limit to 100:
```ruby
Plan.default.actual_limits.update!(ci_needs_size_limit: 100)
```
To disable directed acyclic graphs (DAG), set the limit to `0`.
#### Artifact downloads with `needs`
#### `needs:artifacts`
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.6.
......@@ -1458,84 +1444,72 @@ by default, because jobs with `needs` can start before earlier stages complete.
Use `artifacts: true` (default) or `artifacts: false` to control when artifacts are
downloaded in jobs that use `needs`.
In the following example, the `rspec` job downloads the `build_job` artifacts, but the
`rubocop` job does not:
**Keyword type**: Job keyword. You can use it only as part of a job. Must be used with `needs:job`.
```yaml
build_job:
stage: build
artifacts:
paths:
- binaries/
**Possible inputs**:
rspec:
- `true` (default) or `false`.
**Example of `needs:artifacts`**:
```yaml
test-job1:
stage: test
needs:
- job: build_job
- job: build_job1
artifacts: true
rubocop:
test-job2:
stage: test
needs:
- job: build_job
- job: build_job2
artifacts: false
```
In the following example, the `rspec` job downloads the artifacts from all three `build_jobs`.
`artifacts` is:
- Set to true for `build_job_1`.
- Defaults to true for both `build_job_2` and `build_job_3`.
```yaml
rspec:
test-job3:
needs:
- job: build_job_1
- job: build_job1
artifacts: true
- job: build_job_2
- build_job_3
- job: build_job2
- build_job3
```
In GitLab 12.6 and later, you can't combine the [`dependencies`](#dependencies) keyword
with `needs`.
In this example:
#### Cross project artifact downloads with `needs` **(PREMIUM)**
- The `test-job1` job downloads the `build_job1` artifacts
- The `test-job2` job does not download the `build_job2` artifacts.
- The `test-job3` job downloads the artifacts from all three `build_jobs`, because
`artifacts:` is `true`, or defaults to `true`, for all three needed jobs.
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.7.
**Additional details**:
Use `needs` to download artifacts from up to five jobs in pipelines:
- In GitLab 12.6 and later, you can't combine the [`dependencies`](#dependencies) keyword
with `needs`.
- [On other refs in the same project](#artifact-downloads-between-pipelines-in-the-same-project).
- In different projects, groups and namespaces.
#### `needs:project` **(PREMIUM)**
```yaml
build_job:
stage: build
script:
- ls -lhR
needs:
- project: namespace/group/project-name
job: build-1
ref: main
artifacts: true
```
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.7.
Use `needs:project` to download artifacts from up to five jobs in other pipelines.
The artifacts are downloaded from the latest successful pipeline for the specified ref.
`build_job` downloads the artifacts from the latest successful `build-1` job
on the `main` branch in the `group/project-name` project. If the project is in the
same group or namespace, you can omit them from the `project:` keyword. For example,
`project: group/project-name` or `project: project-name`.
If there is a pipeline running for the specified ref, a job with `needs:project`
does not wait for the pipeline to complete. Instead, the job downloads the artifact
from the latest pipeline that completed successfully.
The user running the pipeline must have at least `reporter` access to the group or project, or the group/project must have public visibility.
`needs:project` must be used with `job:`, `ref:`, and `artifacts:`.
You cannot use cross project artifact downloads in the same job as [`trigger`](#trigger).
**Keyword type**: Job keyword. You can use it only as part of a job.
##### Artifact downloads between pipelines in the same project
**Possible inputs**:
Use `needs` to download artifacts from different pipelines in the current project.
Set the `project` keyword as the current project's name, and specify a ref.
- `needs:project`: A full project path, including namespace and group. If the
project is in the same group or namespace, you can omit them from the `project:`
keyword. For example: `project: group/project-name` or `project: project-name`.
- `job`: The job to download artifacts from.
- `ref`: The ref to download artifacts from.
- `artifacts`: Must be `true` to download artifacts.
In the following example, `build_job` downloads the artifacts for the latest successful
`build-1` job with the `other-ref` ref:
**Examples of `needs:project`**:
```yaml
build_job:
......@@ -1543,16 +1517,17 @@ build_job:
script:
- ls -lhR
needs:
- project: group/same-project-name
- project: namespace/group/project-name
job: build-1
ref: other-ref
ref: main
artifacts: true
```
CI/CD variable support for `project:`, `job:`, and `ref` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202093)
in GitLab 13.3. [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235761) in GitLab 13.4.
In this example, `build_job` downloads the artifacts from the latest successful `build-1` job
on the `main` branch in the `group/project-name` project.
For example:
In GitLab 13.3 and later, you can use [CI/CD variables](../variables/index.md) in `needs:project`,
for example:
```yaml
build_job:
......@@ -1566,62 +1541,83 @@ build_job:
artifacts: true
```
You can't download artifacts from jobs that run in [`parallel:`](#parallel).
**Additional details**:
When using `needs` to download artifacts from another pipeline, the job does not wait for
the needed job to complete. [Directed acyclic graph](../directed_acyclic_graph/index.md)
behavior is limited to jobs in the same pipeline. Make sure that the needed job in the other
pipeline completes before the job that needs it tries to download the artifacts.
- To download artifacts from a different pipeline in the current project, set `project:`
to be the same as the current project, but use a different ref than the current pipeline.
Concurrent pipelines running on the same ref could override the artifacts.
- The user running the pipeline must have at least the Reporter role for the group or project,
or the group/project must have public visibility.
- You can't use `needs:project` in the same job as [`trigger`](#trigger).
- When using `needs:project` to download artifacts from another pipeline, the job does not wait for
the needed job to complete. [Directed acyclic graph](../directed_acyclic_graph/index.md)
behavior is limited to jobs in the same pipeline. Make sure that the needed job in the other
pipeline completes before the job that needs it tries to download the artifacts.
- You can't download artifacts from jobs that run in [`parallel:`](#parallel).
- Support for [CI/CD variables](../variables/index.md) in `project`, `job`, and `ref` was
[introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202093) in GitLab 13.3.
[Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235761) in GitLab 13.4.
To download artifacts between [parent-child pipelines](../pipelines/parent_child_pipelines.md),
use [`needs:pipeline`](#artifact-downloads-to-child-pipelines).
**Related topics**:
You should not download artifacts from the same ref as a running pipeline. Concurrent
pipelines running on the same ref could override the artifacts.
- To download artifacts between [parent-child pipelines](../pipelines/parent_child_pipelines.md),
use [`needs:pipeline:job`](#needspipelinejob).
#### Artifact downloads to child pipelines
#### `needs:pipeline:job`
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/255983) in GitLab 13.7.
A [child pipeline](../pipelines/parent_child_pipelines.md) can download artifacts from a job in
its parent pipeline or another child pipeline in the same parent-child pipeline hierarchy.
For example, with the following parent pipeline that has a job that creates some artifacts:
**Keyword type**: Job keyword. You can use it only as part of a job.
```yaml
create-artifact:
stage: build
script: echo 'sample artifact' > artifact.txt
artifacts:
paths: [artifact.txt]
**Possible inputs**:
child-pipeline:
stage: test
trigger:
include: child.yml
strategy: depend
variables:
PARENT_PIPELINE_ID: $CI_PIPELINE_ID
```
- `needs:pipeline`: A pipeline ID. Must be a pipeline present in the same parent-child pipeline hierarchy.
- `job:`: The job to download artifacts from.
A job in the child pipeline can download artifacts from the `create-artifact` job in
the parent pipeline:
**Example of `needs:pipeline:job`**:
```yaml
use-artifact:
script: cat artifact.txt
needs:
- pipeline: $PARENT_PIPELINE_ID
job: create-artifact
```
- Parent pipeline (`.gitlab-ci.yml`):
The `pipeline` attribute accepts a pipeline ID and it must be a pipeline present
in the same parent-child pipeline hierarchy of the given pipeline.
```yaml
create-artifact:
stage: build
script: echo 'sample artifact' > artifact.txt
artifacts:
paths: [artifact.txt]
child-pipeline:
stage: test
trigger:
include: child.yml
strategy: depend
variables:
PARENT_PIPELINE_ID: $CI_PIPELINE_ID
```
- Child pipeline (`child.yml`):
The `pipeline` attribute does not accept the current pipeline ID (`$CI_PIPELINE_ID`).
To download artifacts from a job in the current pipeline, use the basic form of [`needs`](#artifact-downloads-with-needs).
```yaml
use-artifact:
script: cat artifact.txt
needs:
- pipeline: $PARENT_PIPELINE_ID
job: create-artifact
```
#### Optional `needs`
In this example, the `create-artifact` job in the parent pipeline creates some artifacts.
The `child-pipeline` job triggers a child pipeline, and passes the `CI_PIPELINE_ID`
variable to the child pipeline as a new `PARENT_PIPELINE_ID` variable. The child pipeline
can use that variable in `needs:pipeline` to download artifacts from the parent pipeline.
**Additional details**:
- The `pipeline` attribute does not accept the current pipeline ID (`$CI_PIPELINE_ID`).
To download artifacts from a job in the current pipeline, use [`needs`](#needsartifacts).
#### `needs:optional`
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30680) in GitLab 13.10.
> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323891) in GitLab 14.0.
......@@ -1630,20 +1626,21 @@ To need a job that sometimes does not exist in the pipeline, add `optional: true
to the `needs` configuration. If not defined, `optional: false` is the default.
Jobs that use [`rules`](#rules), [`only`, or `except`](#only--except), might
not always exist in a pipeline. When the pipeline starts, it checks the `needs`
relationships before running. Without `optional: true`, needs relationships that
not always exist in a pipeline. When the pipeline is created, GitLab checks the `needs`
relationships before starting it. Without `optional: true`, needs relationships that
point to a job that does not exist stops the pipeline from starting and causes a pipeline
error similar to:
- `'job1' job needs 'job2' job, but it was not added to the pipeline`
In this example:
**Keyword type**: Job keyword. You can use it only as part of a job.
- When the branch is the default branch, the `build` job exists in the pipeline, and the `rspec`
job waits for it to complete before starting.
- When the branch is not the default branch, the `build` job does not exist in the pipeline.
The `rspec` job runs immediately (similar to `needs: []`) because its `needs`
relationship to the `build` job is optional.
**Possible inputs**:
- `job:`: The job to make optional.
- `true` or `false` (default).
**Example of `needs:optional`**:
```yaml
build:
......@@ -1658,6 +1655,42 @@ rspec:
optional: true
```
In this example:
- When the branch is the default branch, the `build` job exists in the pipeline, and the `rspec`
job waits for it to complete before starting.
- When the branch is not the default branch, the `build` job does not exist in the pipeline.
The `rspec` job runs immediately (similar to `needs: []`) because its `needs`
relationship to the `build` job is optional.
#### `needs:pipeline`
You can mirror the pipeline status from an upstream pipeline to a bridge job by
using the `needs:pipeline` keyword. The latest pipeline status from the default branch is
replicated to the bridge job.
**Keyword type**: Job keyword. You can use it only as part of a job.
**Possible inputs**:
- A full project path, including namespace and group. If the
project is in the same group or namespace, you can omit them from the `project:`
keyword. For example: `project: group/project-name` or `project: project-name`.
**Example of `needs:pipeline`**:
```yaml
upstream_bridge:
stage: test
needs:
pipeline: other/project
```
**Additional details**:
- If you add the `job` keyword to `needs:pipeline`, the job no longer mirrors the
pipeline status. The behavior changes to [`needs:pipeline:job`](#needspipelinejob).
### `tags`
> - A limit of 50 tags per job [enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/338929) in GitLab 14.3.
......@@ -2512,7 +2545,7 @@ By default, jobs in later stages automatically download all the artifacts create
by jobs in earlier stages. You can control artifact download behavior in jobs with
[`dependencies`](#dependencies).
When using the [`needs`](#artifact-downloads-with-needs) keyword, jobs can only download
When using the [`needs`](#needs) keyword, jobs can only download
artifacts from the jobs defined in the `needs` configuration.
Job artifacts are only collected for successful jobs by default, and
......@@ -2840,7 +2873,7 @@ pipeline features from each job.
To be able to browse the report output files, include the [`artifacts:paths`](#artifactspaths) keyword.
NOTE:
Combined reports in parent pipelines using [artifacts from child pipelines](#artifact-downloads-to-child-pipelines) is
Combined reports in parent pipelines using [artifacts from child pipelines](#needspipelinejob) is
not supported. Track progress on adding support in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215725).
##### `artifacts:reports:accessibility` **(FREE)**
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7