Commit b7cff43f authored by Marcel Amirault's avatar Marcel Amirault Committed by Suzanne Selhorn

Refactor CI script documentation

There are some ways to format script sections
to change how they display in job logs. These
should be documented separately, to allow for
more detailed examples (covering different OSes,
for example). We also need to split the before_script
and after_script details, as they work differently.
parent 57d87f8b
...@@ -25,6 +25,7 @@ exceptions: ...@@ -25,6 +25,7 @@ exceptions:
- CPU - CPU
- CSS - CSS
- CSV - CSV
- DAG
- DAST - DAST
- DNS - DNS
- EKS - EKS
......
...@@ -452,9 +452,9 @@ CI jobs: ...@@ -452,9 +452,9 @@ CI jobs:
from `Dockerfile` that may be overridden in `.gitlab-ci.yml`) from `Dockerfile` that may be overridden in `.gitlab-ci.yml`)
1. The runner attaches itself to a running container. 1. The runner attaches itself to a running container.
1. The runner prepares a script (the combination of 1. The runner prepares a script (the combination of
[`before_script`](../yaml/README.md#before_script-and-after_script), [`before_script`](../yaml/README.md#before_script),
[`script`](../yaml/README.md#script), [`script`](../yaml/README.md#script),
and [`after_script`](../yaml/README.md#before_script-and-after_script)). and [`after_script`](../yaml/README.md#after_script)).
1. The runner sends the script to the container's shell STDIN and receives the 1. The runner sends the script to the container's shell STDIN and receives the
output. output.
......
...@@ -258,7 +258,7 @@ stages: ...@@ -258,7 +258,7 @@ stages:
``` ```
Setting a step to be performed before and after any job can be done via the Setting a step to be performed before and after any job can be done via the
[`before_script` and `after_script` keywords](../yaml/README.md#before_script-and-after_script): [`before_script`](../yaml/README.md#before_script) and [`after_script`](../yaml/README.md#after_script) keywords:
```yaml ```yaml
default: default:
......
...@@ -93,7 +93,7 @@ to access it. This is where an SSH key pair comes in handy. ...@@ -93,7 +93,7 @@ to access it. This is where an SSH key pair comes in handy.
# - git config --global user.name "User name" # - git config --global user.name "User name"
``` ```
The [`before_script`](../yaml/README.md#before_script-and-after_script) can be set globally The [`before_script`](../yaml/README.md#before_script) can be set globally
or per-job. or per-job.
1. Make sure the private server's [SSH host keys are verified](#verifying-the-ssh-host-keys). 1. Make sure the private server's [SSH host keys are verified](#verifying-the-ssh-host-keys).
......
...@@ -69,7 +69,7 @@ Kubernetes-specific environment variables are detailed in the ...@@ -69,7 +69,7 @@ Kubernetes-specific environment variables are detailed in the
| `CI_JOB_MANUAL` | 8.12 | all | The flag to indicate that job was manually started | | `CI_JOB_MANUAL` | 8.12 | all | The flag to indicate that job was manually started |
| `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` | | `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` |
| `CI_JOB_STAGE` | 9.0 | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` | | `CI_JOB_STAGE` | 9.0 | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` |
| `CI_JOB_STATUS` | all | 13.5 | The state of the job as each runner stage is executed. Use with [`after_script`](../yaml/README.md#before_script-and-after_script) where `CI_JOB_STATUS` can be either: `success`, `failed` or `canceled`. | | `CI_JOB_STATUS` | all | 13.5 | The state of the job as each runner stage is executed. Use with [`after_script`](../yaml/README.md#after_script) where `CI_JOB_STATUS` can be either: `success`, `failed` or `canceled`. |
| `CI_JOB_TOKEN` | 9.0 | 1.2 | Token used for authenticating with [a few API endpoints](../../api/README.md#gitlab-ci-job-token) and downloading [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories). The token is valid as long as the job is running. | | `CI_JOB_TOKEN` | 9.0 | 1.2 | Token used for authenticating with [a few API endpoints](../../api/README.md#gitlab-ci-job-token) and downloading [dependent repositories](../../user/project/new_ci_build_permissions_model.md#dependent-repositories). The token is valid as long as the job is running. |
| `CI_JOB_JWT` | 12.10 | all | RS256 JSON web token that can be used for authenticating with third party systems that support JWT authentication, for example [HashiCorp's Vault](../secrets/index.md). | | `CI_JOB_JWT` | 12.10 | all | RS256 JSON web token that can be used for authenticating with third party systems that support JWT authentication, for example [HashiCorp's Vault](../secrets/index.md). |
| `CI_JOB_URL` | 11.1 | 0.5 | Job details URL | | `CI_JOB_URL` | 11.1 | 0.5 | Job details URL |
......
...@@ -99,7 +99,7 @@ In the case of `after_script` scripts, they can: ...@@ -99,7 +99,7 @@ In the case of `after_script` scripts, they can:
- Not use variables defined in `before_script` and `script`. - Not use variables defined in `before_script` and `script`.
These restrictions are because `after_script` scripts are executed in a These restrictions are because `after_script` scripts are executed in a
[separated shell context](../yaml/README.md#before_script-and-after_script). [separated shell context](../yaml/README.md#after_script).
## Persisted variables ## Persisted variables
......
...@@ -26,10 +26,10 @@ The following table lists available keywords for jobs: ...@@ -26,10 +26,10 @@ The following table lists available keywords for jobs:
| Keyword | Description | | Keyword | Description |
|:---------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |:---------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`script`](#script) | Shell script that is executed by a runner. | | [`script`](#script) | Shell script that is executed by a runner. |
| [`after_script`](#before_script-and-after_script) | Override a set of commands that are executed after job. | | [`after_script`](#after_script) | Override a set of commands that are executed after job. |
| [`allow_failure`](#allow_failure) | Allow job to fail. Failed job does not contribute to commit status. | | [`allow_failure`](#allow_failure) | Allow job to fail. Failed job does not contribute to commit status. |
| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:exclude`, `artifacts:expose_as`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, and `artifacts:reports`. | | [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:exclude`, `artifacts:expose_as`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, and `artifacts:reports`. |
| [`before_script`](#before_script-and-after_script) | Override a set of commands that are executed before job. | | [`before_script`](#before_script) | Override a set of commands that are executed before job. |
| [`cache`](#cache) | List of files that should be cached between subsequent runs. Also available: `cache:paths`, `cache:key`, `cache:untracked`, `cache:when`, and `cache:policy`. | | [`cache`](#cache) | List of files that should be cached between subsequent runs. Also available: `cache:paths`, `cache:key`, `cache:untracked`, `cache:when`, and `cache:policy`. |
| [`coverage`](#coverage) | Code coverage settings for a given job. | | [`coverage`](#coverage) | Code coverage settings for a given job. |
| [`dependencies`](#dependencies) | Restrict which artifacts are passed to a specific job by providing a list of jobs to fetch artifacts from. | | [`dependencies`](#dependencies) | Restrict which artifacts are passed to a specific job by providing a list of jobs to fetch artifacts from. |
...@@ -90,8 +90,8 @@ The following job keywords can be defined inside a `default:` block: ...@@ -90,8 +90,8 @@ The following job keywords can be defined inside a `default:` block:
- [`image`](#image) - [`image`](#image)
- [`services`](#services) - [`services`](#services)
- [`before_script`](#before_script-and-after_script) - [`before_script`](#before_script)
- [`after_script`](#before_script-and-after_script) - [`after_script`](#after_script)
- [`tags`](#tags) - [`tags`](#tags)
- [`cache`](#cache) - [`cache`](#cache)
- [`artifacts`](#artifacts) - [`artifacts`](#artifacts)
...@@ -421,7 +421,7 @@ include: ...@@ -421,7 +421,7 @@ include:
file: '/templates/.gitlab-ci-template.yml' file: '/templates/.gitlab-ci-template.yml'
``` ```
You can also specify `ref`, with the default being the `HEAD` of the project: You can also specify a `ref`. If not specified, it defaults to the `HEAD` of the project:
```yaml ```yaml
include: include:
...@@ -481,8 +481,8 @@ Feature.disable(:ci_include_multiple_files_from_project) ...@@ -481,8 +481,8 @@ Feature.disable(:ci_include_multiple_files_from_project)
#### `include:remote` #### `include:remote`
`include:remote` can be used to include a file from a different location, `include:remote` can be used to include a file from a different location,
using HTTP/HTTPS, referenced by using the full URL. The remote file must be using HTTP/HTTPS, referenced by the full URL. The remote file must be
publicly accessible through a simple GET request as authentication schemas publicly accessible by a GET request, because authentication schemas
in the remote URL are not supported. For example: in the remote URL are not supported. For example:
```yaml ```yaml
...@@ -528,7 +528,7 @@ Nested includes allow you to compose a set of includes. ...@@ -528,7 +528,7 @@ Nested includes allow you to compose a set of includes.
A total of 100 includes is allowed, but duplicate includes are considered a configuration error. A total of 100 includes is allowed, but duplicate includes are considered a configuration error.
In [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/28212) and later, the time limit In [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/28212) and later, the time limit
for resolving all files is 30 seconds. to resolve all files is 30 seconds.
#### Additional `includes` examples #### Additional `includes` examples
...@@ -544,7 +544,7 @@ Used to specify [a Docker image](../docker/using_docker_images.md#what-is-an-ima ...@@ -544,7 +544,7 @@ Used to specify [a Docker image](../docker/using_docker_images.md#what-is-an-ima
For: For:
- Simple definition examples, see [Define `image` and `services` from `.gitlab-ci.yml`](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml). - Usage examples, see [Define `image` and `services` from `.gitlab-ci.yml`](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml).
- Detailed usage information, refer to [Docker integration](../docker/README.md) documentation. - Detailed usage information, refer to [Docker integration](../docker/README.md) documentation.
#### `image:name` #### `image:name`
...@@ -565,7 +565,7 @@ Used to specify a [service Docker image](../docker/using_docker_images.md#what-i ...@@ -565,7 +565,7 @@ Used to specify a [service Docker image](../docker/using_docker_images.md#what-i
For: For:
- Simple definition examples, see [Define `image` and `services` from `.gitlab-ci.yml`](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml). - Usage examples, see [Define `image` and `services` from `.gitlab-ci.yml`](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml).
- Detailed usage information, refer to [Docker integration](../docker/README.md) documentation. - Detailed usage information, refer to [Docker integration](../docker/README.md) documentation.
- For example services, see [GitLab CI/CD Services](../services/README.md). - For example services, see [GitLab CI/CD Services](../services/README.md).
...@@ -603,9 +603,9 @@ job: ...@@ -603,9 +603,9 @@ job:
script: "bundle exec rspec" script: "bundle exec rspec"
``` ```
[YAML anchors for scripts](#yaml-anchors-for-script) are available. You can use [YAML anchors with `script`](#yaml-anchors-for-scripts).
This keyword can also contain several commands using an array: This keyword can also contain several commands in an array:
```yaml ```yaml
job: job:
...@@ -621,8 +621,8 @@ a "key: value" pair. Be careful when using special characters: ...@@ -621,8 +621,8 @@ a "key: value" pair. Be careful when using special characters:
`:`, `{`, `}`, `[`, `]`, `,`, `&`, `*`, `#`, `?`, `|`, `-`, `<`, `>`, `=`, `!`, `%`, `@`, `` ` ``. `:`, `{`, `}`, `[`, `]`, `,`, `&`, `*`, `#`, `?`, `|`, `-`, `<`, `>`, `=`, `!`, `%`, `@`, `` ` ``.
If any of the script commands return an exit code other than zero, the job If any of the script commands return an exit code other than zero, the job
fails and further commands are not executed. You can avoid this behavior by fails and further commands are not executed. Store the exit code in a variable to
storing the exit code in a variable: avoid this behavior:
```yaml ```yaml
job: job:
...@@ -631,191 +631,86 @@ job: ...@@ -631,191 +631,86 @@ job:
- if [ $exit_code -ne 0 ]; then echo "Previous command failed"; fi; - if [ $exit_code -ne 0 ]; then echo "Previous command failed"; fi;
``` ```
#### `before_script` and `after_script` #### `before_script`
> Introduced in GitLab 8.7 and requires GitLab Runner v1.2. > Introduced in GitLab 8.7 and requires GitLab Runner v1.2.
`before_script` is used to define commands that should be run before each `before_script` is used to define commands that should run before each job, including
job, including deploy jobs, but after the restoration of any [artifacts](#artifacts). deploy jobs, but after the restoration of any [artifacts](#artifacts). This must be an array.
This must be an array.
Scripts specified in `before_script` are concatenated with any scripts specified Scripts specified in `before_script` are concatenated with any scripts specified
in the main [`script`](#script), and executed together in a single shell. in the main [`script`](#script), and executed together in a single shell.
`after_script` is used to define commands that run after each It's possible to overwrite a globally defined `before_script` if you define it in a job:
job, including failed jobs. This must be an array. If a job times out or is cancelled,
the `after_script` commands are not executed. Support for executing `after_script`
commands for timed-out or cancelled jobs
[is planned](https://gitlab.com/gitlab-org/gitlab/-/issues/15603).
Scripts specified in `after_script` are executed in a new shell, separate from any
`before_script` or `script` scripts. As a result, they:
- Have a current working directory set back to the default.
- Have no access to changes done by scripts defined in `before_script` or `script`, including:
- Command aliases and variables exported in `script` scripts.
- Changes outside of the working tree (depending on the runner executor), like
software installed by a `before_script` or `script` script.
- Have a separate timeout, which is hard coded to 5 minutes. See
[related issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2716) for details.
- Don't affect the job's exit code. If the `script` section succeeds and the
`after_script` times out or fails, the job exits with code `0` (`Job Succeeded`).
It's possible to overwrite a globally defined `before_script` or `after_script`
if you set it per-job:
```yaml ```yaml
default: default:
before_script: before_script:
- global before script - echo "Execute this in all jobs that don't already have a before_script section."
job:
before_script:
- execute this instead of global before script
script:
- my command
after_script:
- execute this after my script
```
[YAML anchors for `before_script` and `after_script`](#yaml-anchors-for-before_script-and-after_script) are available.
#### Coloring script output
Script output can be colored using [ANSI escape codes](https://en.wikipedia.org/wiki/ANSI_escape_code#Colors),
or by running commands or programs that output ANSI escape codes.
For example, using [Bash with color codes](https://misc.flogisoft.com/bash/tip_colors_and_formatting):
```yaml job1:
job:
script:
- echo -e "\e[31mThis text is red,\e[0m but this text isn't\e[31m however this text is red again."
```
You can define the color codes in Shell variables, or even [custom environment variables](../variables/README.md#custom-environment-variables),
which makes the commands easier to read and reusable.
For example, using the same example as above and variables defined in a `before_script`:
```yaml
job:
before_script:
- TXT_RED="\e[31m" && TXT_CLEAR="\e[0m"
script: script:
- echo -e "${TXT_RED}This text is red,${TXT_CLEAR} but this part isn't${TXT_RED} however this part is again." - echo "This executes after the global before_script."
- echo "This text is not colored"
```
Or with [PowerShell color codes](https://superuser.com/a/1259916):
```yaml
job: job:
before_script: before_script:
- $esc="$([char]27)"; $TXT_RED="$esc[31m"; $TXT_CLEAR="$esc[0m" - echo "Execute this instead of the global before_script."
script: script:
- Write-Host $TXT_RED"This text is red,"$TXT_CLEAR" but this text isn't"$TXT_RED" however this text is red again." - echo "This executes after the job's `before_script`"
- Write-Host "This text is not colored"
``` ```
#### Multi-line commands You can use [YAML anchors with `before_script`](#yaml-anchors-for-scripts).
You can split long commands into multi-line commands to improve readability
using [`|` (literal) and `>` (folded) YAML multi-line block scalar indicators](https://yaml-multiline.info/).
CAUTION: **Warning:** #### `after_script`
If multiple commands are combined into one command string, only the last command's
failure or success is reported.
[Failures from earlier commands are ignored due to a bug](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25394).
To work around this,
run each command as a separate `script:` item, or add an `exit 1` command
to each command string.
You can use the `|` (literal) YAML multiline block scalar indicator to write Introduced in GitLab 8.7 and requires GitLab Runner v1.2.
commands over multiple lines in the `script` section of a job description.
Each line is treated as a separate command.
Only the first command is repeated in the job log, but additional
commands are still executed:
```yaml `after_script` is used to define commands that run after each job, including failed
job: jobs. This must be an array.
script:
- |
echo "First command line."
echo "Second command line."
echo "Third command line."
```
The example above renders in the job log as: If a job times out or is cancelled, the `after_script` commands are not executed.
Support for executing `after_script` commands for timed-out or cancelled jobs
[is planned](https://gitlab.com/gitlab-org/gitlab/-/issues/15603).
```shell Scripts specified in `after_script` are executed in a new shell, separate from any
$ echo First command line # collapsed multi-line command `before_script` or `script` scripts. As a result, they:
First command line
Second command line.
Third command line.
```
The `>` (folded) YAML multiline block scalar indicator treats empty lines between - Have a current working directory set back to the default.
sections as the start of a new command: - Have no access to changes done by scripts defined in `before_script` or `script`, including:
- Command aliases and variables exported in `script` scripts.
- Changes outside of the working tree (depending on the runner executor), like
software installed by a `before_script` or `script` script.
- Have a separate timeout, which is hard coded to 5 minutes. See the
[related issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2716) for details.
- Don't affect the job's exit code. If the `script` section succeeds and the
`after_script` times out or fails, the job exits with code `0` (`Job Succeeded`).
```yaml ```yaml
job: default:
script: after_script:
- > - echo "Execute this in all jobs that don't already have an after_script section."
echo "First command line
is split over two lines."
echo "Second command line."
```
This behaves similarly to writing multiline commands without the `>` or `|` block
scalar indicators:
```yaml job1:
job:
script: script:
- echo "First command line - echo "This executes first. When it completes, the global after_script executes."
is split over two lines."
echo "Second command line."
```
Both examples above render in the job log as:
```shell
$ echo First command line is split over two lines. # collapsed multi-line command
First command line is split over two lines.
Second command line.
```
When you omit the `>` or `|` block scalar indicators, GitLab forms the command
by concatenating non-empty lines. Make sure the lines can run when concatenated.
Shell [here documents](https://en.wikipedia.org/wiki/Here_document) work with the
`|` and `>` operators as well. The example below transliterates the lower case letters
to upper case:
```yaml
job: job:
script: script:
- | - echo "This executes first. When it completes, the job's `after_script` executes."
tr a-z A-Z << END_TEXT after_script:
one two three - echo "Execute this instead of the global after_script."
four five six
END_TEXT
``` ```
Results in: You can use [YAML anchors with `after_script`](#yaml-anchors-for-scripts).
```shell #### Script syntax
$ tr a-z A-Z << END_TEXT # collapsed multi-line command
ONE TWO THREE
FOUR FIVE SIX
```
#### Custom collapsible sections You can use special syntax in [`script`](README.md#script) sections to:
See [custom collapsible sections](../pipelines/index.md#custom-collapsible-sections). - [Split long commands](script.md#split-long-commands) into multiline commands.
- [Use color codes](script.md#add-color-codes-to-script-output) to make job logs easier to review.
- [Create custom collapsible sections](../pipelines/index.md#custom-collapsible-sections)
to simplify job log output.
### `stage` ### `stage`
...@@ -954,7 +849,7 @@ rspec: ...@@ -954,7 +849,7 @@ rspec:
- $RSPEC - $RSPEC
``` ```
If you do want to include the `rake test`, see [`before_script` and `after_script`](#before_script-and-after_script). If you do want to include the `rake test`, see [`before_script`](#before_script) or [`after_script`](#after_script).
`.tests` in this example is a [hidden job](#hide-jobs), but it's `.tests` in this example is a [hidden job](#hide-jobs), but it's
possible to inherit from regular jobs as well. possible to inherit from regular jobs as well.
...@@ -3536,7 +3431,7 @@ exceed the runner-specific timeout. ...@@ -3536,7 +3431,7 @@ exceed the runner-specific timeout.
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21480) in GitLab 11.5. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21480) in GitLab 11.5.
Use `parallel` to configure how many instances of a job to run in Use `parallel` to configure how many instances of a job to run in
parallel. This value has to be greater than or equal to two (2) and less than or equal to 50. parallel. This value can be from 2 to 50.
This creates N instances of the same job that run in parallel. They are named This creates N instances of the same job that run in parallel. They are named
sequentially from `job_name 1/N` to `job_name N/N`. sequentially from `job_name 1/N` to `job_name N/N`.
...@@ -3635,8 +3530,8 @@ Use `trigger` to define a downstream pipeline trigger. When GitLab starts a job ...@@ -3635,8 +3530,8 @@ Use `trigger` to define a downstream pipeline trigger. When GitLab starts a job
with a `trigger` definition, a downstream pipeline is created. with a `trigger` definition, a downstream pipeline is created.
Jobs with `trigger` can only use a [limited set of keywords](../multi_project_pipelines.md#limitations). Jobs with `trigger` can only use a [limited set of keywords](../multi_project_pipelines.md#limitations).
For example, you can't run commands with [`script`](#script), [`before_script`](#before_script-and-after_script), For example, you can't run commands with [`script`](#script), [`before_script`](#before_script),
or [`after_script`](#before_script-and-after_script). or [`after_script`](#after_script).
You can use this keyword to create two different types of downstream pipelines: You can use this keyword to create two different types of downstream pipelines:
...@@ -4384,50 +4279,30 @@ test:mysql: ...@@ -4384,50 +4279,30 @@ test:mysql:
You can see that the hidden jobs are conveniently used as templates, and You can see that the hidden jobs are conveniently used as templates, and
`tags: [dev]` has been overwritten by `tags: [postgres]`. `tags: [dev]` has been overwritten by `tags: [postgres]`.
#### YAML anchors for `before_script` and `after_script` #### YAML anchors for scripts
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23005) in GitLab 12.5. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23005) in GitLab 12.5.
You can use [YAML anchors](#anchors) with `before_script` and `after_script`, You can use [YAML anchors](#anchors) with [script](#script), [`before_script`](#before_script),
which makes it possible to include a predefined list of commands in multiple and [`after_script`](#after_script) to use predefined commands in multiple jobs:
jobs.
Example:
```yaml ```yaml
.something_before: &something_before .some-script: &some-script
- echo 'something before' - echo "Execute this in `before_script` sections"
.something_after: &something_after
- echo 'something after'
- echo 'another thing after'
job_name:
before_script:
- *something_before
script:
- echo 'this is the script'
after_script:
- *something_after
```
#### YAML anchors for `script`
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23005) in GitLab 12.5.
You can use [YAML anchors](#anchors) with scripts, which makes it possible to
include a predefined list of commands in multiple jobs.
For example: .some-script-before: &some-script-before
- echo "Execute this in `script` sections"
```yaml .some-script-after: &some-script-after
.something: &something - echo "Execute this in `after_script` sections"
- echo 'something'
job_name: job_name:
before_script:
- *some-script-before
script: script:
- *something - *some-script
- echo 'this is the script' before_script:
- *some-script-after
``` ```
#### YAML anchors for variables #### YAML anchors for variables
...@@ -4503,13 +4378,13 @@ The following keywords are deprecated. ...@@ -4503,13 +4378,13 @@ The following keywords are deprecated.
### Globally-defined `types` ### Globally-defined `types`
CAUTION: **Deprecated:** DANGER: **Deprecated:**
`types` is deprecated, and could be removed in a future release. `types` is deprecated, and could be removed in a future release.
Use [`stages`](#stages) instead. Use [`stages`](#stages) instead.
### Job-defined `type` ### Job-defined `type`
CAUTION: **Deprecated:** DANGER: **Deprecated:**
`type` is deprecated, and could be removed in one of the future releases. `type` is deprecated, and could be removed in one of the future releases.
Use [`stage`](#stage) instead. Use [`stage`](#stage) instead.
......
---
stage: Verify
group: Continuous Integration
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
type: reference
---
# GitLab CI/CD script syntax
You can use special syntax in [`script`](README.md#script) sections to:
- [Split long commands](#split-long-commands) into multiline commands.
- [Use color codes](#add-color-codes-to-script-output) to make job logs easier to review.
- [Create custom collapsible sections](../pipelines/index.md#custom-collapsible-sections)
to simplify job log output.
## Split long commands
You can split long commands into multiline commands to improve readability with
`|` (literal) and `>` (folded) [YAML multiline block scalar indicators](https://yaml-multiline.info/).
CAUTION: **Warning:**
If multiple commands are combined into one command string, only the last command's
failure or success is reported.
[Failures from earlier commands are ignored due to a bug](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/25394).
To work around this, run each command as a separate `script:` item, or add an `exit 1`
command to each command string.
You can use the `|` (literal) YAML multiline block scalar indicator to write
commands over multiple lines in the `script` section of a job description.
Each line is treated as a separate command.
Only the first command is repeated in the job log, but additional
commands are still executed:
```yaml
job:
script:
- |
echo "First command line."
echo "Second command line."
echo "Third command line."
```
The example above renders in the job log as:
```shell
$ echo First command line # collapsed multiline command
First command line
Second command line.
Third command line.
```
The `>` (folded) YAML multiline block scalar indicator treats empty lines between
sections as the start of a new command:
```yaml
job:
script:
- >
echo "First command line
is split over two lines."
echo "Second command line."
```
This behaves similarly to multiline commands without the `>` or `|` block
scalar indicators:
```yaml
job:
script:
- echo "First command line
is split over two lines."
echo "Second command line."
```
Both examples above render in the job log as:
```shell
$ echo First command line is split over two lines. # collapsed multiline command
First command line is split over two lines.
Second command line.
```
When you omit the `>` or `|` block scalar indicators, GitLab concatenates non-empty
lines to form the command. Make sure the lines can run when concatenated.
[Shell here documents](https://en.wikipedia.org/wiki/Here_document) work with the
`|` and `>` operators as well. The example below transliterates lower case letters
to upper case:
```yaml
job:
script:
- |
tr a-z A-Z << END_TEXT
one two three
four five six
END_TEXT
```
Results in:
```shell
$ tr a-z A-Z << END_TEXT # collapsed multiline command
ONE TWO THREE
FOUR FIVE SIX
```
## Add color codes to script output
Script output can be colored using [ANSI escape codes](https://en.wikipedia.org/wiki/ANSI_escape_code#Colors),
or by running commands or programs that output ANSI escape codes.
For example, using [Bash with color codes](https://misc.flogisoft.com/bash/tip_colors_and_formatting):
```yaml
job:
script:
- echo -e "\e[31mThis text is red,\e[0m but this text isn't\e[31m however this text is red again."
```
You can define the color codes in Shell variables, or even [custom environment variables](../variables/README.md#custom-environment-variables),
which makes the commands easier to read and reusable.
For example, using the same example as above and variables defined in a `before_script`:
```yaml
job:
before_script:
- TXT_RED="\e[31m" && TXT_CLEAR="\e[0m"
script:
- echo -e "${TXT_RED}This text is red,${TXT_CLEAR} but this part isn't${TXT_RED} however this part is again."
- echo "This text is not colored"
```
Or with [PowerShell color codes](https://superuser.com/a/1259916):
```yaml
job:
before_script:
- $esc="$([char]27)"; $TXT_RED="$esc[31m"; $TXT_CLEAR="$esc[0m"
script:
- Write-Host $TXT_RED"This text is red,"$TXT_CLEAR" but this text isn't"$TXT_RED" however this text is red again."
- Write-Host "This text is not colored"
```
...@@ -46,12 +46,12 @@ Because the `script` entry can't be left empty, it must be set to the command th ...@@ -46,12 +46,12 @@ Because the `script` entry can't be left empty, it must be set to the command th
It is not possible to rely on the predefined `ENTRYPOINT` and `CMD` of the Docker image It is not possible to rely on the predefined `ENTRYPOINT` and `CMD` of the Docker image
to perform the scan automatically, without passing any command. to perform the scan automatically, without passing any command.
The [`before_script`](../../ci/yaml/README.md#before_script-and-after_script) The [`before_script`](../../ci/yaml/README.md#before_script)
should not be used in the job definition because users may rely on this to prepare their projects before performing the scan. should not be used in the job definition because users may rely on this to prepare their projects before performing the scan.
For instance, it is common practice to use `before_script` to install system libraries For instance, it is common practice to use `before_script` to install system libraries
a particular project needs before performing SAST or Dependency Scanning. a particular project needs before performing SAST or Dependency Scanning.
Similarly, [`after_script`](../../ci/yaml/README.md#before_script-and-after_script) Similarly, [`after_script`](../../ci/yaml/README.md#after_script)
should not be used in the job definition, because it may be overridden by users. should not be used in the job definition, because it may be overridden by users.
### Stage ### Stage
......
...@@ -148,8 +148,8 @@ variables: ...@@ -148,8 +148,8 @@ variables:
Version `3` of the `container_scanning` Docker image uses [`centos:centos8`](https://hub.docker.com/_/centos) Version `3` of the `container_scanning` Docker image uses [`centos:centos8`](https://hub.docker.com/_/centos)
as the new base. It also removes the use of the [start.sh](https://gitlab.com/gitlab-org/security-products/analyzers/klar/-/merge_requests/77) as the new base. It also removes the use of the [start.sh](https://gitlab.com/gitlab-org/security-products/analyzers/klar/-/merge_requests/77)
script and instead executes the analyzer by default. Any customizations made to the script and instead executes the analyzer by default. Any customizations made to the
`container_scanning` job's [`before_script`](../../../ci/yaml/README.md#before_script-and-after_script) `container_scanning` job's [`before_script`](../../../ci/yaml/README.md#before_script)
and [`after_script`](../../../ci/yaml/README.md#before_script-and-after_script) and [`after_script`](../../../ci/yaml/README.md#after_script)
blocks may not work with the new version. To roll back to the previous [`alpine:3.11.3`](https://hub.docker.com/_/alpine)-based blocks may not work with the new version. To roll back to the previous [`alpine:3.11.3`](https://hub.docker.com/_/alpine)-based
Docker image, you can specify the major version through the [`CS_MAJOR_VERSION`](#available-variables) Docker image, you can specify the major version through the [`CS_MAJOR_VERSION`](#available-variables)
variable. variable.
......
...@@ -444,7 +444,7 @@ documentation for a list of settings that you can apply. ...@@ -444,7 +444,7 @@ documentation for a list of settings that you can apply.
The `license_scanning` job runs in a [Debian 10](https://www.debian.org/releases/buster/) Docker The `license_scanning` job runs in a [Debian 10](https://www.debian.org/releases/buster/) Docker
image. The supplied image ships with some build tools such as [CMake](https://cmake.org/) and [GCC](https://gcc.gnu.org/). image. The supplied image ships with some build tools such as [CMake](https://cmake.org/) and [GCC](https://gcc.gnu.org/).
However, not all project types are supported by default. To install additional tools needed to However, not all project types are supported by default. To install additional tools needed to
compile dependencies, use a [`before_script`](../../../ci/yaml/README.md#before_script-and-after_script) compile dependencies, use a [`before_script`](../../../ci/yaml/README.md#before_script)
to install the necessary build tools using the [`apt`](https://wiki.debian.org/PackageManagementTools) to install the necessary build tools using the [`apt`](https://wiki.debian.org/PackageManagementTools)
package manager. For a comprehensive list, consult [the Conan documentation](https://docs.conan.io/en/latest/introduction.html#all-platforms-all-build-systems-and-compilers). package manager. For a comprehensive list, consult [the Conan documentation](https://docs.conan.io/en/latest/introduction.html#all-platforms-all-build-systems-and-compilers).
......
...@@ -409,7 +409,7 @@ test: ...@@ -409,7 +409,7 @@ test:
- For the beta release, we have included a set of software packages in - For the beta release, we have included a set of software packages in
the base VM image. If your CI job requires additional software that's the base VM image. If your CI job requires additional software that's
not included in this list, then you will need to add installation not included in this list, then you will need to add installation
commands to [`before_script`](../../ci/yaml/README.md#before_script-and-after_script) or [`script`](../../ci/yaml/README.md#script) to install the required commands to [`before_script`](../../ci/yaml/README.md#before_script) or [`script`](../../ci/yaml/README.md#script) to install the required
software. Note that each job runs on a new VM instance, so the software. Note that each job runs on a new VM instance, so the
installation of additional software packages needs to be repeated for installation of additional software packages needs to be repeated for
each job in your pipeline. each job in your pipeline.
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment