multi_project_pipelines.md 9.81 KB
Newer Older
Marcia Ramos's avatar
Marcia Ramos committed
1 2 3 4
---
type: reference
---

5
# Multi-project pipelines
6

7 8
> - [Introduced](https://about.gitlab.com/releases/2015/08/22/gitlab-7-14-released/#build-triggers-api-gitlab-ci) in GitLab 7.14, as Build Triggers.
> - [Made available](https://gitlab.com/gitlab-org/gitlab/issues/199224) in all tiers in GitLab 12.8.
9

10 11
You can set up [GitLab CI/CD](README.md) across multiple projects, so that a pipeline
in one project can trigger a pipeline in another project.
12 13 14

## Overview

15 16
GitLab CI/CD is a powerful continuous integration tool that works not only per project,
but also across projects with multi-project pipelines.
17 18

Multi-project pipelines are useful for larger products that require cross-project inter-dependencies, such as those
19
adopting a [microservices architecture](https://about.gitlab.com/blog/2016/08/16/trends-in-version-control-land-microservices/).
20

21 22 23 24
For a demonstration of how cross-functional development teams can use cross-pipeline
triggering to trigger multiple pipelines for different microservices projects, see
[Cross-project Pipeline Triggering and Visualization](https://about.gitlab.com/handbook/marketing/product-marketing/demo/#cross-project-pipeline-triggering-and-visualization-may-2019---1110).

25 26 27
Additionally, it's possible to visualize the entire pipeline, including all cross-project
inter-dependencies. **(PREMIUM)**

28 29 30 31 32 33 34 35
## Use cases

Let's assume you deploy your web app from different projects in GitLab:

- One for the free version, which has its own pipeline that builds and tests your app
- One for the paid version add-ons, which also pass through builds and tests
- One for the documentation, which also builds, tests, and deploys with an SSG

36
With Multi-Project Pipelines you can visualize the entire pipeline, including all build and test stages for the three projects.
37

38 39 40 41 42 43 44 45 46 47 48 49 50 51
## Multi-project pipeline visualization **(PREMIUM)**

> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/2121) in [GitLab Premium 9.3](https://about.gitlab.com/releases/2017/06/22/gitlab-9-3-released/#multi-project-pipeline-graphs).

When you configure GitLab CI for your project, you can visualize the stages of your
[jobs](pipelines.md#configuring-pipelines) on a [pipeline graph](pipelines.md#visualizing-pipelines).

![Multi-project pipeline graph](img/multi_project_pipeline_graph.png)

In the Merge Request Widget, multi-project pipeline mini-graphs are displayed,
and when hovering or tapping (on touchscreen devices) they will expand and be shown adjacent to each other.

![Multi-project mini graph](img/multi_pipeline_mini_graph.gif)

52 53
## Triggering multi-project pipelines through API

54 55 56
> - Use of `CI_JOB_TOKEN` for multi-project pipelines was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2017) in [GitLab Premium][ee] 9.3.
> - Use of `CI_JOB_TOKEN` for multi-project pipelines was [made available](https://gitlab.com/gitlab-org/gitlab/issues/31573) in all tiers in GitLab 12.4.

57 58 59 60 61 62 63 64 65
When you use the [`CI_JOB_TOKEN` to trigger pipelines](triggers/README.md#ci-job-token), GitLab
recognizes the source of the job token, and thus internally ties these pipelines
together, allowing you to visualize their relationships on pipeline graphs.

These relationships are displayed in the pipeline graph by showing inbound and
outbound connections for upstream and downstream pipeline dependencies.

## Creating multi-project pipelines from `.gitlab-ci.yml`

66
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8.
67
> - [Made available](https://gitlab.com/gitlab-org/gitlab/issues/199224) in all tiers in 12.8.
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102

### Triggering a downstream pipeline using a bridge job

Before GitLab 11.8, it was necessary to implement a pipeline job that was
responsible for making the API request [to trigger a pipeline](#triggering-multi-project-pipelines-through-api)
in a different project.

In GitLab 11.8, GitLab provides a new CI/CD configuration syntax to make this
task easier, and avoid needing GitLab Runner for triggering cross-project
pipelines. The following illustrates configuring a bridge job:

```yaml
rspec:
  stage: test
  script: bundle exec rspec

staging:
  variables:
    ENVIRONMENT: staging
  stage: deploy
  trigger: my/deployment
```

In the example above, as soon as `rspec` job succeeds in the `test` stage,
the `staging` bridge job is going to be started. The initial status of this
job will be `pending`. GitLab will create a downstream pipeline in the
`my/deployment` project and, as soon as the pipeline gets created, the
`staging` job will succeed. `my/deployment` is a full path to that project.

The user that created the upstream pipeline needs to have access rights to the
downstream project (`my/deployment` in this case). If a downstream project can
not be found, or a user does not have access rights to create pipeline there,
the `staging` job is going to be marked as _failed_.

CAUTION: **Caution:**
103 104 105
In the example, `staging` will be marked as succeeded as soon as a downstream pipeline
gets created. If you want to display the downstream pipeline's status instead, see
[Mirroring status from triggered pipeline](#mirroring-status-from-triggered-pipeline).
106 107 108

NOTE: **Note:**
Bridge jobs do not support every configuration entry that a user can use
109 110
in the case of regular jobs. Bridge jobs will not be picked by a Runner,
so there is no point in adding support for `script`, for example. If a user
111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129
tries to use unsupported configuration syntax, YAML validation will fail upon
pipeline creation.

### Specifying a downstream pipeline branch

It is possible to specify a branch name that a downstream pipeline will use:

```yaml
rspec:
  stage: test
  script: bundle exec rspec

staging:
  stage: deploy
  trigger:
    project: my/deployment
    branch: stable-11-2
```

130 131 132 133
Use:

- The `project` keyword to specify the full path to a downstream project.
- The `branch` keyword to specify the name of a branch in the project specified by `project`.
134 135
  [From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/issues/10126), variable expansion is
  supported.
136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160

GitLab will use a commit that is currently on the HEAD of the branch when
creating a downstream pipeline.

### Passing variables to a downstream pipeline

Sometimes you might want to pass variables to a downstream pipeline.
You can do that using the `variables` keyword, just like you would when
defining a regular job.

```yaml
rspec:
  stage: test
  script: bundle exec rspec

staging:
  variables:
    ENVIRONMENT: staging
  stage: deploy
  trigger: my/deployment
```

The `ENVIRONMENT` variable will be passed to every job defined in a downstream
pipeline. It will be available as an environment variable when GitLab Runner picks a job.

161 162 163
In the following configuration, the `MY_VARIABLE` variable will be passed to the downstream pipeline
that is created when the `trigger-downstream` job is queued. This is because `trigger-downstream`
job inherits variables declared in global variables blocks, and then we pass these variables to a downstream pipeline.
164 165 166 167 168

```yaml
variables:
  MY_VARIABLE: my-value

169
trigger-downstream:
170 171 172 173 174 175 176 177 178 179
  variables:
    ENVIRONMENT: something
  trigger: my/project
```

You might want to pass some information about the upstream pipeline using, for
example, predefined variables. In order to do that, you can use interpolation
to pass any variable. For example:

```yaml
180
downstream-job:
181 182 183 184 185 186
  variables:
    UPSTREAM_BRANCH: $CI_COMMIT_REF_NAME
  trigger: my/project
```

In this scenario, the `UPSTREAM_BRANCH` variable with a value related to the
187
upstream pipeline will be passed to the `downstream-job` job, and will be available
188 189
within the context of all downstream builds.

190 191 192 193 194
NOTE: **Tip:**
Upstream pipelines take precedence over downstream ones. If there are two
variables with the same name defined in both upstream and downstream projects,
the ones defined in the upstream project will take precedence.

195 196
### Mirroring status from triggered pipeline

197 198
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/11238) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3.
> - [Moved](https://gitlab.com/gitlab-org/gitlab/issues/199224) to GitLab Core in 12.8.
199

200 201 202 203 204 205 206 207 208 209
You can mirror the pipeline status from the triggered pipeline to the source
bridge job by using `strategy: depend`. For example:

```yaml
trigger_job:
  trigger:
    project: my/project
    strategy: depend
```

Matija Čupić's avatar
Matija Čupić committed
210 211 212 213 214 215 216 217 218 219 220 221 222 223 224
### Mirroring status from upstream pipeline

You can mirror the pipeline status from an upstream pipeline to a bridge job by
using the `needs:pipeline` keyword. The latest pipeline status from master is
replicated to the bridge job.

Example:

```yaml
upstream_bridge:
  stage: test
  needs:
    pipeline: other/project
```

225 226 227 228 229 230 231 232 233 234 235 236 237
### Limitations

Because bridge jobs are a little different to regular jobs, it is not
possible to use exactly the same configuration syntax here, as one would
normally do when defining a regular job that will be picked by a runner.

Some features are not implemented yet. For example, support for environments.

[Configuration keywords](yaml/README.md) available for bridge jobs are:

- `trigger` (to define a downstream pipeline trigger)
- `stage`
- `allow_failure`
238
- [`rules`](yaml/README.md#rules)
239
- `only` and `except`
240
- `when` (only with `on_success`, `on_failure`, and `always` values)
241
- `extends`
242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257

## Trigger a pipeline when an upstream project is rebuilt

> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/9045) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.8.

You can trigger a pipeline in your project whenever a pipeline finishes for a new
tag in a different project:

1. Go to the project's **Settings > CI / CD** page, and expand the **Pipeline subscriptions** section.
1. Enter the path to the project you want to subscribe to.
1. Click subscribe.

Any pipelines that complete successfully for new tags in the subscribed project
will now trigger a pipeline on the current project's default branch. The maximum
number of upstream pipeline subscriptions is 2, for both the upstream and
downstream projects.