Skip to content

G
gitlab-ce
Commit 59087136 authored by Marcel Amirault's avatar Marcel Amirault
Browse files
Options

Copy edit CI troubleshooting doc 

Reorder sections, group related things together,
and copy edit for style and clarity
parent 5e150c42
Hide whitespace changes
Inline Side-by-side
Showing with 181 additions and 81 deletions
doc/.vale/gitlab/spelling-exceptions.txt
View file @ 59087136
......@@ -403,6 +403,7 @@ sanitization
sbt
scatterplot
scatterplots
Schemastore
Sendmail
Sentry
serializer
......
doc/ci/merge_request_pipelines/pipelines_for_merged_results/index.md
View file @ 59087136
......@@ -127,5 +127,5 @@ unexpected timing. For example, when a source or target branch is advanced.
In this case, the pipeline fails because of `fatal: reference is not a tree:` error,
which indicates that the checkout-SHA is not found in the merge ref.
This behavior was improved at GitLab 12.4 by introducing [Persistent pipeline refs](../../pipelines/index.md#troubleshooting-fatal-reference-is-not-a-tree).
This behavior was improved at GitLab 12.4 by introducing [Persistent pipeline refs](../../troubleshooting.md#fatal-reference-is-not-a-tree-error).
You should be able to create pipelines at any timings without concerning the error.
doc/ci/pipelines/index.md
View file @ 59087136
......@@ -535,32 +535,3 @@ GitLab provides API endpoints to:
- Trigger pipeline runs. For more information, see:
- [Triggering pipelines through the API](../triggers/README.md).
- [Pipeline triggers API](../../api/pipeline_triggers.md).
## Troubleshooting `fatal: reference is not a tree:`
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17043) in GitLab 12.4.
Previously, you'd have encountered unexpected pipeline failures when you force-pushed
a branch to its remote repository. To illustrate the problem, suppose you've had the current workflow:
1. A user creates a feature branch named `example` and pushes it to a remote repository.
1. A new pipeline starts running on the `example` branch.
1. A user rebases the `example` branch on the latest `master` branch and force-pushes it to its remote repository.
1. A new pipeline starts running on the `example` branch again, however,
the previous pipeline (2) fails because of `fatal: reference is not a tree:` error.
This is because the previous pipeline cannot find a checkout-SHA (which associated with the pipeline record)
from the `example` branch that the commit history has already been overwritten by the force-push.
Similarly, [Pipelines for merged results](../merge_request_pipelines/pipelines_for_merged_results/index.md)
might have failed intermittently due to [the same reason](../merge_request_pipelines/pipelines_for_merged_results/index.md#intermittently-pipelines-fail-by-fatal-reference-is-not-a-tree-error).
As of GitLab 12.4, we've improved this behavior by persisting pipeline refs exclusively.
To illustrate its life cycle:
1. A pipeline is created on a feature branch named `example`.
1. A persistent pipeline ref is created at `refs/pipelines/<pipeline-id>`,
which retains the checkout-SHA of the associated pipeline record.
This persistent ref stays intact during the pipeline execution,
even if the commit history of the `example` branch has been overwritten by force-push.
1. The runner fetches the persistent pipeline ref and gets source code from the checkout-SHA.
1. When the pipeline finished, its persistent ref is cleaned up in a background process.
doc/ci/troubleshooting.md
View file @ 59087136
......@@ -7,30 +7,76 @@ type: reference
# Troubleshooting CI/CD
Troubleshooting your pipelines is an important part of the process, and GitLab provides several tools to help make testing your pipelines easier. See also below for common issues and their solutions.
GitLab provides several tools to help make troubleshooting your pipelines easier.
## Editing
This guide also lists common issues and possible solutions.
The [GitLab Web IDE](../user/project/web_ide/index.md) offers advanced authoring tools, including syntax highlighting for the `.gitlab-ci.yml`, and is the recommended editing experience (rather than the single file editor). If you prefer to use another editor, you can use a schema like [this externally maintained one](https://json.schemastore.org/gitlab-ci) with your editor of choice.
## Verify syntax
## CI Reference Documentation
An early source of problems can be incorrect syntax. The pipeline shows a `yaml invalid`
badge and does not start running if any syntax or formatting problems are found.
The [`gitlab-ci.yml` complete reference](yaml/README.md) contains everything you may need to know on how pipelines are defined. There are also some more complex authoring features that have their own detailed usage guides that we recommend looking at if you are considering using these features:
### Edit `gitlab-ci.yml` with the Web IDE
- The [`rules` keyword](yaml/README.md#rules) behaves quite different if you are coming from `only/except`, so be sure to check out the guide if that's your situation. The section on [common `if` clauses](yaml/README.md#common-if-clauses-for-rules) can be very helpful for examples.
- [Multi-project pipelines](multi_project_pipelines.md).
- [Child/parent pipelines](parent_child_pipelines.md) (running a separate `.gitlab-ci.yml` in the same repository as a connected but separate pipeline) and [Dynamic child/parent pipelines](parent_child_pipelines.md#dynamic-child-pipelines) which allows you to dynamically generate the child pipeline's YAML at runtime).
- [Pipelines for Merge Requests](merge_request_pipelines/index.md) (running a pipeline in the context of a merge request), [Pipelines for Merge Results](merge_request_pipelines/pipelines_for_merged_results/index.md) (the same, but on the combined ref of the source and target branch), and [Merge Trains](merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md) (the previous two combined, and automatically queued and merged in sequence).
The [GitLab Web IDE](../user/project/web_ide/index.md) offers advanced authoring tools,
including syntax highlighting for the `.gitlab-ci.yml`, and is the recommended editing
experience (rather than the single file editor). It offers code completion suggestions
that ensure you are only using accepted keywords.
Apart from these, looking at [examples](examples/README.md) in the documentation can be helpful and we have several.
If you prefer to use another editor, you can use a schema like [the Schemastore `gitlab-ci` schema](https://json.schemastore.org/gitlab-ci)
with your editor of choice.
## Linter
### Verify syntax with CI Lint tool
The [CI Linter](yaml/README.md#validate-the-gitlab-ciyml) can be helpful for making sure your syntax is correct.
The [CI Lint tool](lint.md) is a simple way to ensure the syntax of a CI/CD configuration
file is correct. Paste in full `gitlab-ci.yml` files or individual jobs configuration,
to verify the basic syntax.
## Guides
When a `.gitlab-ci.yml` file is present in a project, you can also use the CI Lint
tool to [simulate the creation of a full pipeline](lint.md#pipeline-simulation).
It does deeper verification of the configuration syntax.
There are various troubleshooting guides available for different topic areas/features:
## Verify variables
A key part of troubleshooting CI/CD is to verify which variables are present in a
pipeline, and what their values are. A lot of pipeline configuration is dependent
on variables, and verifying them is one of the fastest ways to find the source of
a problem.
[Export the full list of variables](variables/README.md#list-all-environment-variables)
available in each problematic job. Check if the variables you expect are present,
and check if their values are what you expect.
## GitLab CI/CD documentation
The [complete `gitlab-ci.yml` reference](yaml/README.md) contains a full list of
every keyword you may need to use to configure your pipelines.
You can also look at a large number of pipeline configuration [examples](examples/README.md)
and [templates](examples/README.md#cicd-templates).
### Documentation for pipeline types
Some pipeline types have their own detailed usage guides that you should read
if you are using that type:
- [Multi-project pipelines](multi_project_pipelines.md): Have your pipeline trigger
a pipeline in a different project.
- [Parent/child pipelines](parent_child_pipelines.md): Have your main pipeline trigger
and run separate pipelines in the same project. You can also
[dynamically generate the child pipeline's configuration](parent_child_pipelines.md#dynamic-child-pipelines)
at runtime.
- [Pipelines for Merge Requests](merge_request_pipelines/index.md): Run a pipeline
in the context of a merge request.
- [Pipelines for Merge Results](merge_request_pipelines/pipelines_for_merged_results/index.md):
Pipelines for merge requests that run on the combined source and target branch
- [Merge Trains](merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md):
Multiple pipelines for merged results that queue and run automatically before
changes are merged.
### Troubleshooting Guides for CI/CD features
There are troubleshooting guides available for some CI/CD features and related topics:
- [Container Registry](../user/packages/container_registry/index.md#troubleshooting-the-gitlab-container-registry)
- [GitLab Runner](https://docs.gitlab.com/runner/faq/)
......@@ -38,62 +84,142 @@ There are various troubleshooting guides available for different topic areas/fea
- [Docker Build](docker/using_docker_build.md#troubleshooting)
- [Environments](environments/deployment_safety.md#ensure-only-one-deployment-job-runs-at-a-time)
## Common issues and their resolution
## Common CI/CD issues
A lot of common pipeline issues can be fixed by analyzing the behavior of the `rules`
or `only/except` configuration. You shouldn't use these two configurations in the same
pipeline, as they behave differently. It's hard to predict how a pipeline runs with
this mixed behavior.
If your `rules` or `only/except` configuration makes use of [predefined variables](variables/predefined_variables.md)
like `CI_PIPELINE_SOURCE`, `CI_MERGE_REQUEST_ID`, you should [verify them](#verify-variables)
as the first troubleshooting step.
### Jobs or pipelines don't run when expected
### `fatal: reference is not a tree`
The `rules` or `only/except` keywords are what determine whether or not a job is
added to a pipeline. If a pipeline runs, but a job is not added to the pipeline,
it's usually due to `rules` or `only/except` configuration issues.
See the details about how to fix this error on the [pipelines reference page](pipelines/index.md#troubleshooting-fatal-reference-is-not-a-tree).
If a pipeline does not seem to run at all, with no error message, it may also be
due to `rules` or `only/except` configuration, or the `workflow: rules` keyword.
### Duplicate pipelines with Pipelines for MRs
If you are converting from `only/except` to the `rules` keyword, you should check
the [`rules` configuration details](yaml/README.md#rules) carefully. The behavior
of `only/except` and `rules` is different and can cause unexpected behavior when migrating
between the two.
This is typically caused by the different behavior of `rules`. Take a look at the [`workflow:rules` templates](yaml/README.md#workflowrules-templates) for ready to add solutions for this issue.
The [common `if` clauses for `rules`](yaml/README.md#common-if-clauses-for-rules)
can be very helpful for examples of how to write rules that behave the way you expect.
### My pipeline didn't create the job I expected
#### Two pipelines run at the same time
GitLab determines if a job is run based on the [`only/except`](yaml/README.md#onlyexcept-basic) or [`rules`](yaml/README.md#rules) defined on your job. If it didn't run, perhaps it is not evaluating as you expect. Confusion around what [different predefined variables mean/values they provide](variables/predefined_variables.md) (for example `CI_PIPELINE_SOURCE`), can be a source of problems here, so it's worth double checking the definitions if you rely on any and they are not behaving as you expect.
Two pipelines can run when pushing a commit to a branch that has an open merge request
associated with it. Usually one pipeline is a merge request pipeline, and the other
is a branch pipeline.
### My pipeline didn't run at all
This is usually caused by the `rules` configuration, and there are several ways to
[prevent duplicate pipelines](yaml/README.md#prevent-duplicate-pipelines).
This can happen when the [`rules`](yaml/README.md#rules) or `only/except` definitions for your pipeline didn't result in _any_ jobs being created for this instance. Check the troubleshooting steps above for jobs, and also double check your [`workflow: rules`](yaml/README.md#workflowrules) section if you are using one; that defines the rules for the entire pipeline.
#### A job is not in the pipeline
### My pipeline runs despite the rules I set up
GitLab determines if a job is added to a pipeline based on the [`only/except`](yaml/README.md#onlyexcept-basic)
or [`rules`](yaml/README.md#rules) defined for the job. If it didn't run, it's probably
not evaluating as you expect.
A common case where this happens is when a tag is pushed or a new branch is created, in which case `only/except` and `rules` for `changes` matches every file in the repository (for example, every file is "new" to that branch). If you think of a new tag or branch as having _no_ changes, this can be surprising. Using a rule like the following (mentioned in the [docs for `rules`](yaml/README.md#common-if-clauses-for-rules)) can be helpful here:
#### No pipeline or the wrong type of pipeline runs
```yaml
rules:
- if: $CI_COMMIT_BEFORE_SHA == '0000000000000000000000000000000000000000'
when: never
```
Before a pipeline can run, GitLab evaluates all the jobs in the configuration and tries
to add them to all available pipeline types. A pipeline does not run if no jobs are added
to it at the end of the evaluation.
### Merge request pipeline widget
If a pipeline did not run, it's likely that all the jobs had `rules` or `only/except` that
blocked them from being added to the pipeline.
The merge request pipeline widget shows information about the pipeline status in a Merge Request. It's displayed above the [merge request ability to merge widget](#merge-request-ability-to-merge-widget). There are a few messages there that you might run into:
If the wrong pipeline type ran, then the `rules` or `only/except` configuration should
be checked to make sure the jobs are added to the correct pipeline type. For
example, if a merge request pipeline did not run, the jobs may have been added to
a branch pipeline instead.
There are several messages that can be displayed depending on the status of the pipeline.
It's also possible that your [`workflow: rules`](yaml/README.md#workflowrules) configuration
blocked the pipeline, or allowed the wrong pipeline type.
#### "Checking pipeline status"
### A job runs unexpectedly
This message is shown when the merge request has no pipeline associated with the latest commit yet. This might be because:
A common reason a job is added to a pipeline unexpectedly is because the `changes`
keyword always evaluates to true in certain cases. For example, `changes` is always
true in certain pipeline types, including scheduled pipelines and pipelines for tags.
The `changes` keyword is used in combination with [`only/except`](yaml/README.md#onlychangesexceptchanges)
or [`rules`](yaml/README.md#ruleschanges)). It's recommended to use `changes` with
`rules` or `only/except` configuration that ensures the job is only added to branch
pipelines or merge request pipelines.
### "fatal: reference is not a tree" error
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17043) in GitLab 12.4.
Previously, you'd have encountered unexpected pipeline failures when you force-pushed
a branch to its remote repository. To illustrate the problem, suppose you've had the current workflow:
1. A user creates a feature branch named `example` and pushes it to a remote repository.
1. A new pipeline starts running on the `example` branch.
1. A user rebases the `example` branch on the latest `master` branch and force-pushes it to its remote repository.
1. A new pipeline starts running on the `example` branch again, however,
the previous pipeline (2) fails because of `fatal: reference is not a tree:` error.
This is because the previous pipeline cannot find a checkout-SHA (which is associated with the pipeline record)
from the `example` branch that the commit history has already been overwritten by the force-push.
Similarly, [Pipelines for merged results](merge_request_pipelines/pipelines_for_merged_results/index.md)
might have failed intermittently due to [the same reason](merge_request_pipelines/pipelines_for_merged_results/index.md#intermittently-pipelines-fail-by-fatal-reference-is-not-a-tree-error).
As of GitLab 12.4, we've improved this behavior by persisting pipeline refs exclusively.
To illustrate its life cycle:
1. A pipeline is created on a feature branch named `example`.
1. A persistent pipeline ref is created at `refs/pipelines/<pipeline-id>`,
which retains the checkout-SHA of the associated pipeline record.
This persistent ref stays intact during the pipeline execution,
even if the commit history of the `example` branch has been overwritten by force-push.
1. The runner fetches the persistent pipeline ref and gets source code from the checkout-SHA.
1. When the pipeline finishes, its persistent ref is cleaned up in a background process.
### Merge request pipeline messages
The merge request pipeline widget shows information about the pipeline status in
a merge request. It's displayed above the [ability to merge status widget](#merge-request-status-messages).
#### "Checking pipeline status" message
This message is shown when the merge request has no pipeline associated with the
latest commit yet. This might be because:
- GitLab hasn't finished creating the pipeline yet.
- You are using an external CI service and GitLab hasn't heard back from the service yet.
- You are not using CI/CD pipelines in your project.
- The latest pipeline was deleted (this is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/214323)).
After the pipeline is created, the message will update with the pipeline status.
After the pipeline is created, the message updates with the pipeline status.
#### Merge request ability to merge widget
### Merge request status messages
The merge request status widget shows the **Merge** button and whether or not a merge request is ready to merge. If the merge request can't be merged, the reason for this is displayed.
The merge request status widget shows the **Merge** button and whether or not a merge
request is ready to merge. If the merge request can't be merged, the reason for this
is displayed.
If the pipeline is still running, the **Merge** button is replaced with the **Merge when pipeline succeeds** button.
If the pipeline is still running, the **Merge** button is replaced with the
**Merge when pipeline succeeds** button.
If [**Merge Trains**](merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md) are enabled, the button is either **Add to merge train** or **Add to merge train when pipeline succeeds**. **(PREMIUM)**
If [**Merge Trains**](merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md)
are enabled, the button is either **Add to merge train** or **Add to merge train when pipeline succeeds**. **(PREMIUM)**
#### "A CI/CD pipeline must run and be successful before merge"
#### "A CI/CD pipeline must run and be successful before merge" message
This message is shown if the [Pipelines must succeed](../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds) setting is enabled in the project and a pipeline has not yet run successfully. This also applies if the pipeline has not been created yet, or if you are waiting for an external CI service. If you don't use pipelines for your project, then you should disable **Pipelines must succeed** so you can accept merge requests.
This message is shown if the [Pipelines must succeed](../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds)
setting is enabled in the project and a pipeline has not yet run successfully.
This also applies if the pipeline has not been created yet, or if you are waiting
for an external CI service. If you don't use pipelines for your project, then you
should disable **Pipelines must succeed** so you can accept merge requests.
## Pipeline warnings
......@@ -102,17 +228,19 @@ Pipeline configuration warnings are shown when you:
- [Validate configuration with the CI Lint tool](yaml/README.md#validate-the-gitlab-ciyml).
- [Manually run a pipeline](pipelines/index.md#run-a-pipeline-manually).
### "Job may allow multiple pipelines to run for a single action"
### "Job may allow multiple pipelines to run for a single action" warning
When you use [`rules`](yaml/README.md#rules) with a `when:` clause without
an `if:` clause, multiple pipelines may run. Usually
this occurs when you push a commit to a branch that has an open merge request associated with it.
When you use [`rules`](yaml/README.md#rules) with a `when:` clause without an `if:`
clause, multiple pipelines may run. Usually this occurs when you push a commit to
a branch that has an open merge request associated with it.
To [prevent duplicate pipelines](yaml/README.md#prevent-duplicate-pipelines), use
[`workflow: rules`](yaml/README.md#workflowrules) or rewrite your rules
to control which pipelines can run.
[`workflow: rules`](yaml/README.md#workflowrules) or rewrite your rules to control
which pipelines can run.
## How to get help
- [GitLab Community Forum](https://forum.gitlab.com/)
- [Support](https://about.gitlab.com/support/)
If you are unable to resolve pipeline issues, you can get help from:
- The [GitLab community forum](https://forum.gitlab.com/)
- GitLab [Support](https://about.gitlab.com/support/)
doc/user/project/repository/reducing_the_repo_size_using_git.md
View file @ 59087136
......@@ -37,7 +37,7 @@ other internal references (refs) that are automatically created by GitLab. These
- `refs/merge-requests/*` for merge requests.
- `refs/pipelines/*` for
[pipelines](../../../ci/pipelines/index.md#troubleshooting-fatal-reference-is-not-a-tree).
[pipelines](../../../ci/troubleshooting.md#fatal-reference-is-not-a-tree-error).
- `refs/environments/*` for environments.
Git doesn't usually download these refs to make cloning and fetch faster, but we can use the `--mirror` option to
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7