Commit 9b06dffc authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Merge branch 'docs-repo-merge-24-ci' into 'master'

Docs: Merge EE doc/ci to CE

See merge request gitlab-org/gitlab-ce!27956
parents 4eac38d4 339d0a5b
# Using GitLab CI/CD with a Bitbucket Cloud repository **[PREMIUM]**
GitLab CI/CD can be used with Bitbucket Cloud by creating a
[CI/CD project](https://docs.gitlab.com/ee/user/project/ci_cd_for_external_repo.html) and connecting
your Git repository via URL.
1. In GitLab create a **CI/CD for external repo**, select **Repo by URL** and
create the project.
![Create project](img/external_repository.png)
GitLab will import the repository and enable [Pull Mirroring][pull-mirroring].
1. In GitLab create a
[Personal Access Token](../../user/profile/personal_access_tokens.md)
with `api` scope. This will be used to authenticate requests from the web
hook that will be created in Bitbucket to notify GitLab of new commits.
1. In Bitbucket from **Settings > Webhooks** create a new web hook to notify
GitLab of new commits.
The web hook URL should be set to the GitLab API to trigger pull mirroring,
using the Personal Access Token we just generated for authentication.
```
https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN>
```
The web hook Trigger should be set to 'Repository Push'.
![Bitbucket Cloud webhook](img/bitbucket_webhook.png)
After saving, test the web hook by pushing a change to your Bitbucket
repository.
1. In Bitbucket create an **App Password** from **Bitbucket Settings > App
Passwords** to authenticate the build status script setting commit build
statuses in Bitbucket. Repository write permissions are required.
![Bitbucket Cloud webhook](img/bitbucket_app_password.png)
1. In GitLab from **Settings > CI/CD > Environment variables** add variables to allow
communication with Bitbucket via the Bitbucket API.
`BITBUCKET_ACCESS_TOKEN`: the Bitbucket app password created above
`BITBUCKET_USERNAME`: the username of the Bitbucket account
`BITBUCKET_NAMESPACE`: set this if your GitLab and Bitbucket namespaces differ
`BITBUCKET_REPOSITORY`: set this if your GitLab and Bitbucket project names differ
1. In Bitbucket add a script to push the pipeline status to Bitbucket.
> Note: changes made in GitLab will be overwritten by any changes made
upstream in Bitbucket.
Create a file `build_status` and insert the script below and run
`chmod +x build_status` in your terminal to make the script executable.
```bash
#!/usr/bin/env bash
# Push GitLab CI/CD build status to Bitbucket Cloud
if [ -z "$BITBUCKET_ACCESS_TOKEN" ]; then
echo "ERROR: BITBUCKET_ACCESS_TOKEN is not set"
exit 1
fi
if [ -z "$BITBUCKET_USERNAME" ]; then
echo "ERROR: BITBUCKET_USERNAME is not set"
exit 1
fi
if [ -z "$BITBUCKET_NAMESPACE" ]; then
echo "Setting BITBUCKET_NAMESPACE to $CI_PROJECT_NAMESPACE"
BITBUCKET_NAMESPACE=$CI_PROJECT_NAMESPACE
fi
if [ -z "$BITBUCKET_REPOSITORY" ]; then
echo "Setting BITBUCKET_REPOSITORY to $CI_PROJECT_NAME"
BITBUCKET_REPOSITORY=$CI_PROJECT_NAME
fi
BITBUCKET_API_ROOT="https://api.bitbucket.org/2.0"
BITBUCKET_STATUS_API="$BITBUCKET_API_ROOT/repositories/$BITBUCKET_NAMESPACE/$BITBUCKET_REPOSITORY/commit/$CI_COMMIT_SHA/statuses/build"
BITBUCKET_KEY="ci/gitlab-ci/$CI_JOB_NAME"
case "$BUILD_STATUS" in
running)
BITBUCKET_STATE="INPROGRESS"
BITBUCKET_DESCRIPTION="The build is running!"
;;
passed)
BITBUCKET_STATE="SUCCESSFUL"
BITBUCKET_DESCRIPTION="The build passed!"
;;
failed)
BITBUCKET_STATE="FAILED"
BITBUCKET_DESCRIPTION="The build failed."
;;
esac
echo "Pushing status to $BITBUCKET_STATUS_API..."
curl --request POST $BITBUCKET_STATUS_API \
--user $BITBUCKET_USERNAME:$BITBUCKET_ACCESS_TOKEN \
--header "Content-Type:application/json" \
--silent \
--data "{ \"state\": \"$BITBUCKET_STATE\", \"key\": \"$BITBUCKET_KEY\", \"description\":
\"$BITBUCKET_DESCRIPTION\",\"url\": \"$CI_PROJECT_URL/-/jobs/$CI_JOB_ID\" }"
```
1. Still in Bitbucket, create a `.gitlab-ci.yml` file to use the script to push
pipeline success and failures to Bitbucket.
```
stages:
- test
- ci_status
unit-tests:
script:
- echo "Success. Add your tests!"
success:
stage: ci_status
before_script:
- ""
after_script:
- ""
script:
- BUILD_STATUS=passed BUILD_KEY=push ./build_status
when: on_success
failure:
stage: ci_status
before_script:
- ""
after_script:
- ""
script:
- BUILD_STATUS=failed BUILD_KEY=push ./build_status
when: on_failure
```
GitLab is now configured to mirror changes from Bitbucket, run CI/CD pipelines
configured in `.gitlab-ci.yml` and push the status to Bitbucket.
[pull-mirroring]: ../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter
# Using GitLab CI/CD with a GitHub repository **[PREMIUM]**
GitLab CI/CD can be used with **GitHub.com** and **GitHub Enterprise** by
creating a [CI/CD project](https://docs.gitlab.com/ee/user/project/ci_cd_for_external_repo.html) to connect your GitHub repository to
GitLab.
NOTE: **Note:**
To use **GitHub Enterprise** with **GitLab.com** you should use the
manual method.
## Connect with GitHub integration
If the [GitHub integration](../../integration/github.md) has been enabled by your GitLab
administrator:
NOTE: **Note:**
Due to a 10-token limitation on the [GitHub OAuth Implementation](https://developer.github.com/apps/building-oauth-apps/authorizing-oauth-apps/#creating-multiple-tokens-for-oauth-apps),
if you import more than 10 times, your oldest imported project's token will be
revoked. See issue [#9147](https://gitlab.com/gitlab-org/gitlab-ee/issues/9147)
for more information.
1. In GitLab create a **CI/CD for external repo** project and select
**GitHub**.
![Create project](img/github_omniauth.png)
1. Once authenticated, you will be redirected to a list of your repositories to
connect. Click **Connect** to select the repository.
![Create project](img/github_repo_list.png)
1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md).
GitLab will import the project, enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), enable
[GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html), and create a web hook
on GitHub to notify GitLab of new commits.
## Connect with Personal Access Token
NOTE: **Note:** Personal access tokens can only be used to connect GitHub.com
repositories to GitLab.
If you are not using the [GitHub integration](../../integration/github.md), you can
still perform a one-off authorization with GitHub to grant GitLab access your
repositories:
1. Open https://github.com/settings/tokens/new to create a **Personal Access
Token**. This token with be used to access your repository and push commit
statuses to GitHub.
The `repo` and `admin:repo_hook` should be enable to allow GitLab access to
your project, update commit statuses, and create a web hook to notify
GitLab of new commits.
1. In GitLab create a **CI/CD for external repo** project and select
**GitHub**.
![Create project](img/github_omniauth.png)
1. Paste the token into the **Personal access token** field and click **List
Repositories**. Click **Connect** to select the repository.
1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md).
GitLab will import the project, enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), enable
[GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html), and create a web hook
on GitHub to notify GitLab of new commits.
## Connect manually
If the [GitHub integration](../../integration/github.md) is not enabled, or is enabled
for a different GitHub instance, you GitLab CI/CD can be manually enabled for
your repository.
1. In GitHub open https://github.com/settings/tokens/new create a **Personal
Access Token.** GitLab will use this token to access your repository and
push commit statuses.
Enter a **Token description** and update the scope to allow:
`repo` so that GitLab can access your project and update commit statuses
1. In GitLab create a **CI/CD project** using the Git URL option and the HTTPS
URL for your GitHub repository. If your project is private, use the personal
access token you just created for authentication.
GitLab will automatically configure polling-based pull mirroring.
1. Still in GitLab, enable the [GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html)
from **Settings > Integrations.**
Check the **Active** checkbox to enable the integration, paste your
personal access token and HTTPS repository URL into the form, and **Save.**
1. Still in GitLab create a **Personal Access Token** with `API` scope to
authenticate the GitHub web hook notifying GitLab of new commits.
1. In GitHub from **Settings > Webhooks** create a web hook to notify GitLab of
new commits.
The web hook URL should be set to the GitLab API to
[trigger pull mirroring](https://docs.gitlab.com/ee/api/projects.html#start-the-pull-mirroring-process-for-a-project-starter),
using the GitLab personal access token we just created.
```
https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN>
```
![Create web hook](img/github_push_webhook.png)
1. In GitHub add a `.gitlab-ci.yml` to configure GitLab CI/CD.
# GitLab CI/CD for external repositories **[PREMIUM]**
NOTE: **Note:**
This feature [is available for free](https://about.gitlab.com/2019/03/21/six-more-months-ci-cd-github/) to
GitLab.com users until September 22nd, 2019.
>[Introduced][ee-4642] in [GitLab Premium][eep] 10.6.
GitLab CI/CD can be used with GitHub or any other Git server.
Instead of moving your entire project to GitLab, you can connect your
external repository to get the benefits of GitLab CI/CD.
- [GitHub](github_integration.md)
- [Bitbucket Cloud](bitbucket_integration.md)
Connecting an external repository will set up [repository mirroring][mirroring]
and create a lightweight project where issues, merge requests, wiki, and
snippets disabled. These features
[can be re-enabled later][settings].
1. From your GitLab dashboard click **New project**
1. Switch to the **CI/CD for external repo** tab
1. Choose **GitHub** or **Repo by URL**
1. The next steps are similar to the [import flow](../../user/project/import/index.md)
![CI/CD for external repository project creation](img/ci_cd_for_external_repo.png)
[ee-4642]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4642
[eep]: https://about.gitlab.com/pricing/
[mirroring]: ../../workflow/repository_mirroring.md
[settings]: ../../user/project/settings/index.md#sharing-and-permissions
......@@ -669,7 +669,7 @@ fetch = +refs/environments/*:refs/remotes/origin/environments/*
Some GitLab [Enterprise Edition](https://about.gitlab.com/pricing/) features can
behave differently for each environment. For example, you can
[create a secret variable to be injected only into a production environment](https://docs.gitlab.com/ee/ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium).
[create a secret variable to be injected only into a production environment](variables/README.md#limiting-environment-scopes-of-environment-variables-premium).
In most cases, these features use the _environment specs_ mechanism, which offers
an efficient way to implement scoping within each environment group.
......
......@@ -18,30 +18,30 @@ Examples are available in several forms. As a collection of:
The following table lists examples for different use cases:
| Use case | Resource |
|:-----------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------|
| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](browser_performance.md). |
| Clojure | [Test a Clojure application with GitLab CI/CD](test-clojure-application.md). |
| Code quality analysis | [Analyze your project's Code Quality](code_quality.md). **[STARTER]** |
| Container scanning | [Container Scanning with GitLab CI/CD](container_scanning.md). |
| Dependency scanning | [Dependency Scanning with GitLab CI/CD](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html). **[ULTIMATE]** |
| Deployment with `dpl` | [Using `dpl` as deployment tool](deployment/README.md). |
| Dynamic application<br>security testing (DAST) | [Dynamic Application Security Testing with GitLab CI/CD](dast.md) **[ULTIMATE]** |
| Elixir | [Testing a Phoenix application with GitLab CI/CD](test_phoenix_app_with_gitlab_ci_cd/index.md). |
| Game development | [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md). |
| GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example. |
| Java | [Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD](deploy_spring_boot_to_cloud_foundry/index.md). |
| JUnit | [JUnit test reports](../junit_test_reports.md). |
| License management | [Dependencies license management with GitLab CI/CD](https://docs.gitlab.com/ee/ci/examples/license_management.html) **[ULTIMATE]** |
| Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md). |
| PHP | [Testing PHP projects](php.md). |
| PHP | [Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). |
| PHP | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). |
| Python | [Test and deploy a Python application with GitLab CI/CD](test-and-deploy-python-application-to-heroku.md). |
| Ruby | [Test and deploy a Ruby application with GitLab CI/CD](test-and-deploy-ruby-application-to-heroku.md). |
| Scala | [Test and deploy a Scala application to Heroku](test-scala-application.md). |
| Static application<br>security testing (SAST) | [Static Application Security Testing with GitLab CI/CD](https://docs.gitlab.com/ee/ci/examples/sast.html) **[ULTIMATE]** |
| Testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). |
| Use case | Resource |
|:-----------------------------------------------|:---------------------------------------------------------------------------------------------------------------------|
| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](browser_performance.md). |
| Clojure | [Test a Clojure application with GitLab CI/CD](test-clojure-application.md). |
| Code quality analysis | [Analyze your project's Code Quality](code_quality.md). **[STARTER]** |
| Container scanning | [Container Scanning with GitLab CI/CD](container_scanning.md). |
| Dependency scanning | [Dependency Scanning with GitLab CI/CD](dependency_scanning.md). **[ULTIMATE]** |
| Deployment with `dpl` | [Using `dpl` as deployment tool](deployment/README.md). |
| Dynamic application<br>security testing (DAST) | [Dynamic Application Security Testing with GitLab CI/CD](dast.md) **[ULTIMATE]** |
| Elixir | [Testing a Phoenix application with GitLab CI/CD](test_phoenix_app_with_gitlab_ci_cd/index.md). |
| Game development | [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md). |
| GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example. |
| Java | [Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD](deploy_spring_boot_to_cloud_foundry/index.md). |
| JUnit | [JUnit test reports](../junit_test_reports.md). |
| License management | [Dependencies license management with GitLab CI/CD](license_management.md) **[ULTIMATE]** |
| Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md). |
| PHP | [Testing PHP projects](php.md). |
| PHP | [Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). |
| PHP | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). |
| Python | [Test and deploy a Python application with GitLab CI/CD](test-and-deploy-python-application-to-heroku.md). |
| Ruby | [Test and deploy a Ruby application with GitLab CI/CD](test-and-deploy-ruby-application-to-heroku.md). |
| Scala | [Test and deploy a Scala application to Heroku](test-scala-application.md). |
| Static application<br>security testing (SAST) | [Static Application Security Testing with GitLab CI/CD](sast.md) **[ULTIMATE]** |
| Testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). |
### Contributing examples
......
......@@ -56,7 +56,7 @@ provide a list of URLs to test, please consult
TIP: **Tip:**
For [GitLab Premium](https://about.gitlab.com/pricing/) users, key metrics are automatically
extracted and shown right in the merge request widget.
[Learn more on Browser Performance Testing in merge requests](https://docs.gitlab.com/ee//user/project/merge_requests/browser_performance_testing.html).
[Learn more on Browser Performance Testing in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html).
## Performance testing on Review Apps
......
# Container Scanning with GitLab CI/CD
---
redirect_to: 'https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html'
---
CAUTION: **Caution:**
The job definition shown below is supported on GitLab 11.5 and later versions.
It also requires the GitLab Runner 11.5 or later.
For earlier versions, use the [previous job definitions](#previous-job-definitions).
You can check your Docker images (or more precisely the containers) for known
vulnerabilities by using [Clair](https://github.com/coreos/clair) and
[clair-scanner](https://github.com/arminc/clair-scanner), two open source tools
for Vulnerability Static Analysis for containers.
First, you need GitLab Runner with
[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor).
Once you set up the Runner, add a new job to `.gitlab-ci.yml` that
generates the expected report:
```yaml
container_scanning:
image: docker:stable
variables:
DOCKER_DRIVER: overlay2
## Define two new variables based on GitLab's CI/CD predefined variables
## https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables
CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG
CI_APPLICATION_TAG: $CI_COMMIT_SHA
allow_failure: true
services:
- docker:stable-dind
script:
- docker run -d --name db arminc/clair-db:latest
- docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:v2.0.6
- apk add -U wget ca-certificates
- docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG}
- wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64
- mv clair-scanner_linux_amd64 clair-scanner
- chmod +x clair-scanner
- touch clair-whitelist.yml
- while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done
- retries=0
- echo "Waiting for clair daemon to start"
- while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done
- ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true
artifacts:
reports:
container_scanning: gl-container-scanning-report.json
```
The above example will create a `container_scanning` job in your CI/CD pipeline, pull
the image from the [Container Registry](../../user/project/container_registry.md)
(whose name is defined from the two `CI_APPLICATION_` variables) and scan it
for possible vulnerabilities. The report will be saved as a
[Container Scanning report artifact](../yaml/README.md#artifactsreportscontainer_scanning-ultimate)
that you can later download and analyze.
Due to implementation limitations we always take the latest Container Scanning artifact available.
If you want to whitelist some specific vulnerabilities, you can do so by defining
them in a [YAML file](https://github.com/arminc/clair-scanner/blob/master/README.md#example-whitelist-yaml-file),
in our case its named `clair-whitelist.yml`.
TIP: **Tip:**
For [GitLab Ultimate][ee] users, this information will
be automatically extracted and shown right in the merge request widget.
[Learn more on Container Scanning in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html).
CAUTION: **Caution:**
Starting with GitLab 11.5, Container Scanning feature is licensed under the name `container_scanning`.
While the old name `sast_container` is still maintained, it has been deprecated with GitLab 11.5 and
may be removed in next major release, GitLab 12.0. You are advised to update your current `.gitlab-ci.yml`
configuration to reflect that change if you are using the `$GITLAB_FEATURES` environment variable.
## Previous job definitions
CAUTION: **Caution:**
Before GitLab 11.5, Container Scanning job and artifact had to be named specifically
to automatically extract report data and show it in the merge request widget.
While these old job definitions are still maintained they have been deprecated
and may be removed in next major release, GitLab 12.0.
You are advised to update your current `.gitlab-ci.yml` configuration to reflect that change.
For GitLab 11.4 and earlier, the job should look like:
```yaml
container_scanning:
image: docker:stable
variables:
DOCKER_DRIVER: overlay2
## Define two new variables based on GitLab's CI/CD predefined variables
## https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables
CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG
CI_APPLICATION_TAG: $CI_COMMIT_SHA
allow_failure: true
services:
- docker:stable-dind
script:
- docker run -d --name db arminc/clair-db:latest
- docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:v2.0.6
- apk add -U wget ca-certificates
- docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG}
- wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64
- mv clair-scanner_linux_amd64 clair-scanner
- chmod +x clair-scanner
- touch clair-whitelist.yml
- while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done
- retries=0
- echo "Waiting for clair daemon to start"
- while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done
- ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true
artifacts:
paths: [gl-container-scanning-report.json]
```
Alternatively the job name could be `sast:container`
and the artifact name could be `gl-sast-container-report.json`.
These names have been deprecated with GitLab 11.0
and may be removed in next major release, GitLab 12.0.
[ee]: https://about.gitlab.com/pricing/
This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html).
# Dynamic Application Security Testing with GitLab CI/CD
---
redirect_to: 'https://docs.gitlab.com/ee/user/application_security/dast/index.html'
---
CAUTION: **Caution:**
The job definition shown below is supported on GitLab 11.5 and later versions.
It also requires the GitLab Runner 11.5 or later.
For earlier versions, use the [previous job definitions](#previous-job-definitions).
[Dynamic Application Security Testing (DAST)](https://en.wikipedia.org/wiki/Dynamic_program_analysis)
is using the popular open source tool [OWASP ZAProxy](https://github.com/zaproxy/zaproxy)
to perform an analysis on your running web application.
Since it is based on [ZAP Baseline](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan)
DAST will perform passive scanning only;
it will not actively attack your application.
It can be very useful combined with [Review Apps](../review_apps/index.md).
## Example
First, you need GitLab Runner with
[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor).
Once you set up the Runner, add a new job to `.gitlab-ci.yml` that
generates the expected report:
```yaml
dast:
image: registry.gitlab.com/gitlab-org/security-products/zaproxy
variables:
website: "https://example.com"
allow_failure: true
script:
- mkdir /zap/wrk/
- /zap/zap-baseline.py -J gl-dast-report.json -t $website || true
- cp /zap/wrk/gl-dast-report.json .
artifacts:
reports:
dast: gl-dast-report.json
```
The above example will create a `dast` job in your CI/CD pipeline which will run
the tests on the URL defined in the `website` variable (change it to use your
own) and scan it for possible vulnerabilities. The report will be saved as a
[DAST report artifact](../yaml/README.md#artifactsreportsdast-ultimate)
that you can later download and analyze.
Due to implementation limitations we always take the latest DAST artifact available.
It's also possible to authenticate the user before performing DAST checks:
```yaml
dast:
image: registry.gitlab.com/gitlab-org/security-products/zaproxy
variables:
website: "https://example.com"
login_url: "https://example.com/sign-in"
username: "john.doe@example.com"
password: "john-doe-password"
allow_failure: true
script:
- mkdir /zap/wrk/
- /zap/zap-baseline.py -J gl-dast-report.json -t $website
--auth-url $login_url
--auth-username $username
--auth-password $password || true
- cp /zap/wrk/gl-dast-report.json .
artifacts:
reports:
dast: gl-dast-report.json
```
See [zaproxy documentation](https://gitlab.com/gitlab-org/security-products/zaproxy)
to learn more about authentication settings.
TIP: **Tip:**
For [GitLab Ultimate][ee] users, this information will
be automatically extracted and shown right in the merge request widget.
[Learn more on DAST in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/dast.html).
## Previous job definitions
CAUTION: **Caution:**
Before GitLab 11.5, DAST job and artifact had to be named specifically
to automatically extract report data and show it in the merge request widget.
While these old job definitions are still maintained they have been deprecated
and may be removed in next major release, GitLab 12.0.
You are advised to update your current `.gitlab-ci.yml` configuration to reflect that change.
For GitLab 11.4 and earlier, the job should look like:
```yaml
dast:
image: registry.gitlab.com/gitlab-org/security-products/zaproxy
variables:
website: "https://example.com"
allow_failure: true
script:
- mkdir /zap/wrk/
- /zap/zap-baseline.py -J gl-dast-report.json -t $website || true
- cp /zap/wrk/gl-dast-report.json .
artifacts:
paths: [gl-dast-report.json]
```
[ee]: https://about.gitlab.com/pricing/
This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/dast/index.html).
---
redirect_to: 'https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html'
---
This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html).
---
redirect_to: 'https://docs.gitlab.com/ee/user/application_security/license_management/index.html'
---
This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/license_management/index.html).
---
redirect_to: 'https://docs.gitlab.com/ee/user/application_security/sast/index.html'
---
This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/sast/index.html).
---
redirect_to: 'container_scanning.md'
redirect_to: 'https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html'
---
This document was moved to [another location](container_scanning.md).
This document was moved to [another location](../../user/application_security/container_scanning/index.html).
......@@ -54,3 +54,8 @@ terminal will block the job from finishing for the duration configured in
close the terminal window.
![finished job with terminal open](img/finished_job_with_terminal_open.png)
## Interactive Web Terminals for the Web IDE **[ULTIMATE ONLY]**
Read the Web IDE docs to learn how to run [Interactive Terminals through the Web IDE](../../user/project/web_ide/index.md).
# Metrics Reports **[PREMIUM]**
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9788) in [GitLab Premium](https://about.gitlab.com/pricing) 11.10.
Requires GitLab Runner 11.10 and above.
## Overview
GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [JUnit reports](./junit_test_reports.md), [codequality](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change.
You can configure your job to use custom Metrics Reports, and GitLab will display a report on the merge request so that it's easier and faster to identify changes without having to check the entire log.
![Metrics Reports](./img/metrics_reports.png)
## Use cases
Consider the following examples of data that can utilize Metrics Reports:
1. Memory usage
1. Load testing results
1. Code complexity
1. Code coverage stats
## How it works
Metrics are read from the metrics report (default: `metrics.txt`). They are parsed and displayed in the MR widget.
## How to set it up
Add a job that creates a [metrics report](yaml/README.md#artifactsreportsmetrics-premium) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format.
For example:
```yaml
metrics:
script:
- echo 'metric_name metric_value' > metrics.txt
reports:
metrics: metrics.txt
```
---
redirect_to: 'multi_project_pipelines.md'
---
This document was moved to [another location](multi_project_pipelines.md).
# Multi-project pipelines **[PREMIUM]**
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/2121) in
[GitLab Premium 9.3](https://about.gitlab.com/2017/06/22/gitlab-9-3-released/#multi-project-pipeline-graphs).
When you set up [GitLab CI/CD](README.md) across multiple projects, you can visualize
the entire pipeline, including all cross-project inter-dependencies.
## Overview
GitLab CI/CD is a powerful continuous integration tool that works not only per project, but also across projects. When you
configure GitLab CI for your project, you can visualize the stages
of your [jobs](pipelines.md#configuring-pipelines) on a [pipeline graph](pipelines.md#visualizing-pipelines).
![Multi-project pipeline graph](img/multi_project_pipeline_graph.png)
In the Merge Request Widget, multi-project pipeline mini-graphs are displayed,
and when hovering or tapping (on touchscreen devices) they will expand and be shown adjacent to each other.
![Multi-project mini graph](img/multi_pipeline_mini_graph.gif)
Multi-project pipelines are useful for larger products that require cross-project inter-dependencies, such as those
adopting a [microservices architecture](https://about.gitlab.com/2016/08/16/trends-in-version-control-land-microservices/).
## Use cases
Let's assume you deploy your web app from different projects in GitLab:
- One for the free version, which has its own pipeline that builds and tests your app
- One for the paid version add-ons, which also pass through builds and tests
- One for the documentation, which also builds, tests, and deploys with an SSG
With Multi-Project Pipelines, you can visualize the entire pipeline, including all stages of builds and tests for the three projects.
## Triggering multi-project pipelines through API
When you use the [`CI_JOB_TOKEN` to trigger pipelines](triggers/README.md#ci-job-token), GitLab
recognizes the source of the job token, and thus internally ties these pipelines
together, allowing you to visualize their relationships on pipeline graphs.
These relationships are displayed in the pipeline graph by showing inbound and
outbound connections for upstream and downstream pipeline dependencies.
## Creating multi-project pipelines from `.gitlab-ci.yml`
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8.
### Triggering a downstream pipeline using a bridge job
Before GitLab 11.8, it was necessary to implement a pipeline job that was
responsible for making the API request [to trigger a pipeline](#triggering-multi-project-pipelines-through-api)
in a different project.
In GitLab 11.8, GitLab provides a new CI/CD configuration syntax to make this
task easier, and avoid needing GitLab Runner for triggering cross-project
pipelines. The following illustrates configuring a bridge job:
```yaml
rspec:
stage: test
script: bundle exec rspec
staging:
variables:
ENVIRONMENT: staging
stage: deploy
trigger: my/deployment
```
In the example above, as soon as `rspec` job succeeds in the `test` stage,
the `staging` bridge job is going to be started. The initial status of this
job will be `pending`. GitLab will create a downstream pipeline in the
`my/deployment` project and, as soon as the pipeline gets created, the
`staging` job will succeed. `my/deployment` is a full path to that project.
The user that created the upstream pipeline needs to have access rights to the
downstream project (`my/deployment` in this case). If a downstream project can
not be found, or a user does not have access rights to create pipeline there,
the `staging` job is going to be marked as _failed_.
CAUTION: **Caution:**
`staging` will succeed as soon as a downstream pipeline gets created.
GitLab does not support status attribution yet, however adding first-class
`trigger` configuration syntax is ground work for implementing
[status attribution](https://gitlab.com/gitlab-org/gitlab-ce/issues/39640).
NOTE: **Note:**
Bridge jobs do not support every configuration entry that a user can use
in the case of regular jobs. Bridge jobs will not to be picked by a Runner,
thus there is no point in adding support for `script`, for example. If a user
tries to use unsupported configuration syntax, YAML validation will fail upon
pipeline creation.
### Specifying a downstream pipeline branch
It is possible to specify a branch name that a downstream pipeline will use:
```yaml
rspec:
stage: test
script: bundle exec rspec
staging:
stage: deploy
trigger:
project: my/deployment
branch: stable-11-2
```
Use a `project` keyword to specify full path to a downstream project. Use
a `branch` keyword to specify a branch name.
GitLab will use a commit that is currently on the HEAD of the branch when
creating a downstream pipeline.
### Passing variables to a downstream pipeline
Sometimes you might want to pass variables to a downstream pipeline.
You can do that using the `variables` keyword, just like you would when
defining a regular job.
```yaml
rspec:
stage: test
script: bundle exec rspec
staging:
variables:
ENVIRONMENT: staging
stage: deploy
trigger: my/deployment
```
The `ENVIRONMENT` variable will be passed to every job defined in a downstream
pipeline. It will be available as an environment variable when GitLab Runner picks a job.
### Limitations
Because bridge jobs are a little different to regular jobs, it is not
possible to use exactly the same configuration syntax here, as one would
normally do when defining a regular job that will be picked by a runner.
Some features are not implemented yet. For example, support for environments.
[Configuration keywords](yaml/README.md) available for bridge jobs are:
- `trigger` (to define a downstream pipeline trigger)
- `stage`
- `allow_failure`
- `only` and `except`
- `when`
- `extends`
......@@ -19,7 +19,7 @@ If all the jobs in a stage:
- Fail, the next stage is not (usually) executed and the pipeline ends early.
NOTE: **Note:**
If you have a [mirrored repository that GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter),
If you have a [mirrored repository that GitLab pulls from](../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter),
you may need to enable pipeline triggering in your project's
**Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**.
......@@ -220,7 +220,7 @@ For information on adding pipeline badges to projects, see [Pipeline badges](../
Pipelines for different projects can be combined and visualized together.
For more information, see [Multi-project pipelines](https://docs.gitlab.com/ee/ci/multi_project_pipelines.html).
For more information, see [Multi-project pipelines](multi_project_pipelines.md).
## Working with pipelines
......
......@@ -127,7 +127,7 @@ Now if you go to the **Pipelines** page you will see that the pipeline is
pending.
NOTE: **Note:**
If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter),
If you have a [mirrored repository where GitLab pulls from](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter),
you may need to enable pipeline triggering in your project's
**Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**.
......
......@@ -23,6 +23,63 @@ attackers can impersonate the user that exposed their trigger token publicly in
their `.gitlab-ci.yml` file. Use [variables](../variables/README.md#gitlab-cicd-environment-variables)
to protect trigger tokens.
### CI job token
You can use the `CI_JOB_TOKEN` [variable][predef] (used to authenticate
with the [GitLab Container Registry][registry]) in the following cases.
#### When used with multi-project pipelines **[PREMIUM]**
> **Note**:
The use of `CI_JOB_TOKEN` for multi-project pipelines was [introduced][ee-2017]
in [GitLab Premium][ee] 9.3.
This way of triggering can only be used when invoked inside `.gitlab-ci.yml`,
and it creates a dependent pipeline relation visible on the
[pipeline graph](../multi_project_pipelines.md#overview). For example:
```yaml
build_docs:
stage: deploy
script:
- curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=master https://gitlab.example.com/api/v4/projects/9/trigger/pipeline
only:
- tags
```
Pipelines triggered that way also expose a special variable:
`CI_PIPELINE_SOURCE=pipeline`.
Read more about the [pipelines trigger API][trigapi].
#### When a pipeline depends on the artifacts of another pipeline **[PREMIUM]**
> The use of `CI_JOB_TOKEN` in the artifacts download API was [introduced][ee-2346]
in [GitLab Premium][ee] 9.5.
With the introduction of dependencies between different projects, one of
them may need to access artifacts created by a previous one. This process
must be granted for authorized accesses, and it can be done using the
`CI_JOB_TOKEN` variable that identifies a specific job. For example:
```yaml
build_submodule:
image: debian
stage: test
script:
- apt update && apt install -y unzip
- curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test&job_token=$CI_JOB_TOKEN"
- unzip artifacts.zip
only:
- tags
```
This allows you to use that for multi-project pipelines and download artifacts
from any project to which you have access as this follows the same principles
with the [permission model][permissions].
Read more about the [jobs API](../../api/jobs.md#download-the-artifacts-archive).
## Adding a new trigger
You can add a new trigger by going to your project's
......@@ -225,7 +282,10 @@ removed with one of the future versions of GitLab. You are advised to
[take ownership](#taking-ownership-of-a-trigger) of any legacy triggers.
[ee-2017]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2017
[ee-2346]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2346
[ee]: https://about.gitlab.com/pricing/
[variables]: ../variables/README.md
[predef]: ../variables/README.md#predefined-environment-variables
[registry]: ../../user/project/container_registry.md
[permissions]: ../../user/permissions.md#job-permissions
[trigapi]: ../../api/pipeline_triggers.md
......@@ -366,6 +366,25 @@ Protected variables can be added by going to your project's
Once you set them, they will be available for all subsequent pipelines.
### Limiting environment scopes of environment variables **[PREMIUM]**
> [Introduced][ee-2112] in [GitLab Premium](https://about.gitlab.com/pricing/) 9.4.
You can limit the environment scope of a variable by
[defining which environments][envs] it can be available for.
Wildcards can be used, and the default environment scope is `*` which means
any jobs will have this variable, not matter if an environment is defined or
not.
For example, if the environment scope is `production`, then only the jobs
having the environment `production` defined would have this specific variable.
Wildcards (`*`) can be used along with the environment name, therefore if the
environment scope is `review/*` then any jobs with environment names starting
with `review/` would have that particular variable.
To learn more about about scoping environments, see [Scoping environments with specs](../environments.md#scoping-environments-with-specs-premium).
### Deployment environment variables
> Introduced in GitLab 8.15.
......@@ -677,13 +696,15 @@ MIIFQzCCBCugAwIBAgIRAL/ElDjuf15xwja1ZnCocWAwDQYJKoZIhvcNAQELBQAw'
...
```
[ee-2112]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2112
[ce-13784]: https://gitlab.com/gitlab-org/gitlab-ce/issues/13784 "Simple protection of CI variables"
[eep]: https://about.gitlab.com/pricing/ "Available only in GitLab Premium"
[envs]: ../environments.md
[protected branches]: ../../user/project/protected_branches.md
[protected tags]: ../../user/project/protected_tags.md
[shellexecutors]: https://docs.gitlab.com/runner/executors/
[triggered]: ../triggers/README.md
[trigger-job-token]: ../triggers/README.md#ci-job-token
[gitlab-deploy-token]: ../../user/project/deploy_tokens/index.md#gitlab-deploy-token
[registry]: ../../user/project/container_registry.md
[dependent-repositories]: ../../user/project/new_ci_build_permissions_model.md#dependent-repositories
[get-job-artifacts]: ../../api/jobs.html#get-job-artifacts
......@@ -112,3 +112,22 @@ They are:
- Not supported:
- For definitions where the ["Expansion place"](#gitlab-ciyml-file) is GitLab.
- In the `only` and `except` [variables expressions](README.md#environment-variables-expressions).
## Variables with an environment scope
Variables defined with an environment scope are supported. Given that
there is a variable `$STAGING_SECRET` defined in a scope of
`review/staging/*`, the following job that is using dynamic environments
is going to be created, based on the matching variable expression:
```yaml
my-job:
stage: staging
environment:
name: review/$CI_JOB_STAGE/deploy
script:
- 'deploy staging'
only:
variables:
- $STAGING_SECRET == 'something'
```
......@@ -19,7 +19,7 @@ We have complete examples of configuring pipelines:
- To see a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab-ce`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml).
NOTE: **Note:**
If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter),
If you have a [mirrored repository where GitLab pulls from](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter),
you may need to enable pipeline triggering in your project's
**Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**.
......@@ -56,7 +56,7 @@ independently from each other.
Each instance of GitLab CI has an embedded debug tool called Lint, which validates the
content of your `.gitlab-ci.yml` files. You can find the Lint under the page `ci/lint` of your
project namespace. For example, `http://gitlab.example.com/gitlab-org/project-123/-/ci/lint`.
project namespace. For example, `https://gitlab.example.com/gitlab-org/project-123/-/ci/lint`.
### Unavailable names for jobs
......@@ -101,7 +101,7 @@ The following table lists available parameters for jobs:
| [`when`](#when) | When to run job. Also available: `when:manual` and `when:delayed`. |
| [`environment`](#environment) | Name of an environment to which the job deploys. Also available: `environment:name`, `environment:url`, `environment:on_stop`, and `environment:action`. |
| [`cache`](#cache) | List of files that should be cached between subsequent runs. Also available: `cache:paths`, `cache:key`, `cache:untracked`, and `cache:policy`. |
| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, `artifacts:reports`, and `artifacts:reports:junit`.<br><br>In GitLab [Enterprise Edition](https://about.gitlab.com/pricing/), these are available: `artifacts:reports:codequality`, `artifacts:reports:sast`, `artifacts:reports:dependency_scanning`, `artifacts:reports:container_scanning`, `artifacts:reports:dast`, `artifacts:reports:license_management`, and `artifacts:reports:performance`. |
| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, `artifacts:reports`, and `artifacts:reports:junit`.<br><br>In GitLab [Enterprise Edition](https://about.gitlab.com/pricing/), these are available: `artifacts:reports:codequality`, `artifacts:reports:sast`, `artifacts:reports:dependency_scanning`, `artifacts:reports:container_scanning`, `artifacts:reports:dast`, `artifacts:reports:license_management`, `artifacts:reports:performance` and `artifacts:reports:metrics`. |
| [`dependencies`](#dependencies) | Other jobs that a job depends on so that you can pass artifacts between them. |
| [`coverage`](#coverage) | Code coverage settings for a given job. |
| [`retry`](#retry) | When and how many times a job can be auto-retried in case of a failure. |
......@@ -1457,7 +1457,7 @@ dashboards.
> Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above.
The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.html)
The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html)
as artifacts.
The collected Dependency Scanning report will be uploaded to GitLab as an artifact and will
......@@ -1468,7 +1468,7 @@ dashboards.
> Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above.
The `container_scanning` report collects [Container Scanning vulnerabilities](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html)
The `container_scanning` report collects [Container Scanning vulnerabilities](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html)
as artifacts.
The collected Container Scanning report will be uploaded to GitLab as an artifact and will
......@@ -1479,7 +1479,7 @@ dashboards.
> Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above.
The `dast` report collects [DAST vulnerabilities](https://docs.gitlab.com/ee/user/project/merge_requests/dast.html)
The `dast` report collects [DAST vulnerabilities](https://docs.gitlab.com/ee/user/application_security/dast/index.html)
as artifacts.
The collected DAST report will be uploaded to GitLab as an artifact and will
......@@ -1507,6 +1507,14 @@ as artifacts.
The collected Performance report will be uploaded to GitLab as an artifact and will
be automatically shown in merge requests.
##### `artifacts:reports:metrics` **[PREMIUM]**
The `metrics` report collects [Metrics](../../ci/metrics_reports.md)
as artifacts.
The collected Metrics report will be uploaded to GitLab as an artifact and will
be automatically shown in merge requests.
### `dependencies`
> Introduced in GitLab 8.6 and GitLab Runner v1.1.1.
......@@ -1702,7 +1710,7 @@ test:
from `trigger` definition is started by GitLab, a downstream pipeline gets
created.
Learn more about [multi-project pipelines](https://docs.gitlab.com/ee/ci/multi_project_pipelines.html#creating-cross-project-pipelines-from-gitlab-ci-yml).
Learn more about [multi-project pipelines](../multi_project_pipelines.md#creating-multi-project-pipelines-from-gitlab-ciyml).
#### Simple `trigger` syntax
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment