Skip to content

G
gitlab-ce
Commit 7c6ca329 authored by Amy Qualls's avatar Amy Qualls
Browse files
Options

Merge branch 'docs-auto-devops-trouble' into 'master' 

Move Auto DevOps troubleshooting to new file

See merge request gitlab-org/gitlab!54694
parents a6f7acd3 4cfd8b9a
Hide whitespace changes
Inline Side-by-side
Showing with 248 additions and 244 deletions
doc/topics/autodevops/index.md
View file @ 7c6ca329
......@@ -335,246 +335,3 @@ spec:
- name: https_proxy
value: "PUT_YOUR_HTTPS_PROXY_HERE"
```
## Troubleshooting
### Unable to select a buildpack
Auto Build and Auto Test may fail to detect your language or framework with the
following error:
```plaintext
Step 5/11 : RUN /bin/herokuish buildpack build
---> Running in eb468cd46085
-----> Unable to select a buildpack
The command '/bin/sh -c /bin/herokuish buildpack build' returned a non-zero code: 1
```
The following are possible reasons:
- Your application may be missing the key files the buildpack is looking for.
Ruby applications require a `Gemfile` to be properly detected,
even though it's possible to write a Ruby app without a `Gemfile`.
- No buildpack may exist for your application. Try specifying a
[custom buildpack](customize.md#custom-buildpacks).
### Pipeline that extends Auto DevOps with only / except fails
If your pipeline fails with the following message:
```plaintext
Found errors in your .gitlab-ci.yml:
jobs:test config key may not be used with `rules`: only
```
This error appears when the included job’s rules configuration has been overridden with the `only` or `except` syntax.
To fix this issue, you must either:
- Transition your `only/except` syntax to rules.
- (Temporarily) Pin your templates to the [GitLab 12.10 based templates](https://gitlab.com/gitlab-org/auto-devops-v12-10).
### Failure to create a Kubernetes namespace
Auto Deploy fails if GitLab can't create a Kubernetes namespace and
service account for your project. For help debugging this issue, see
[Troubleshooting failed deployment jobs](../../user/project/clusters/index.md#troubleshooting).
### Detected an existing PostgreSQL database
After upgrading to GitLab 13.0, you may encounter this message when deploying
with Auto DevOps:
```plaintext
Detected an existing PostgreSQL database installed on the
deprecated channel 1, but the current channel is set to 2. The default
channel changed to 2 in of GitLab 13.0.
[...]
```
Auto DevOps, by default, installs an in-cluster PostgreSQL database alongside
your application. The default installation method changed in GitLab 13.0, and
upgrading existing databases requires user involvement. The two installation
methods are:
- **channel 1 (deprecated):** Pulls in the database as a dependency of the associated
Helm chart. Only supports Kubernetes versions up to version 1.15.
- **channel 2 (current):** Installs the database as an independent Helm chart. Required
for using the in-cluster database feature with Kubernetes versions 1.16 and greater.
If you receive this error, you can do one of the following actions:
- You can *safely* ignore the warning and continue using the channel 1 PostgreSQL
database by setting `AUTO_DEVOPS_POSTGRES_CHANNEL` to `1` and redeploying.
- You can delete the channel 1 PostgreSQL database and install a fresh channel 2
database by setting `AUTO_DEVOPS_POSTGRES_DELETE_V1` to a non-empty value and
redeploying.
WARNING:
Deleting the channel 1 PostgreSQL database permanently deletes the existing
channel 1 database and all its data. See
[Upgrading PostgreSQL](upgrading_postgresql.md)
for more information on backing up and upgrading your database.
- If you are not using the in-cluster database, you can set
`POSTGRES_ENABLED` to `false` and re-deploy. This option is especially relevant to
users of *custom charts without the in-chart PostgreSQL dependency*.
Database auto-detection is based on the `postgresql.enabled` Helm value for
your release. This value is set based on the `POSTGRES_ENABLED` CI variable
and persisted by Helm, regardless of whether or not your chart uses the
variable.
WARNING:
Setting `POSTGRES_ENABLED` to `false` permanently deletes any existing
channel 1 database for your environment.
### Error: unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1"
After upgrading your Kubernetes cluster to [v1.16+](stages.md#kubernetes-116),
you may encounter this message when deploying with Auto DevOps:
```plaintext
UPGRADE FAILED
Error: failed decoding reader into objects: unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1"
```
This can occur if your current deployments on the environment namespace were deployed with a
deprecated/removed API that doesn't exist in Kubernetes v1.16+. For example,
if [your in-cluster PostgreSQL was installed in a legacy way](#detected-an-existing-postgresql-database),
the resource was created via the `extensions/v1beta1` API. However, the deployment resource
was moved to the `app/v1` API in v1.16.
To recover such outdated resources, you must convert the current deployments by mapping legacy APIs
to newer APIs. There is a helper tool called [`mapkubeapis`](https://github.com/hickeyma/helm-mapkubeapis)
that works for this problem. Follow these steps to use the tool in Auto DevOps:
1. Modify your `.gitlab-ci.yml` with:
```yaml
include:
- template: Auto-DevOps.gitlab-ci.yml
- remote: https://gitlab.com/shinya.maeda/ci-templates/-/raw/master/map-deprecated-api.gitlab-ci.yml
variables:
HELM_VERSION_FOR_MAPKUBEAPIS: "v2" # If you're using auto-depoy-image v2 or above, please specify "v3".
```
1. Run the job `<environment-name>:map-deprecated-api`. Ensure that this job succeeds before moving
to the next step. You should see something like the following output:
```shell
2020/10/06 07:20:49 Found deprecated or removed Kubernetes API:
"apiVersion: extensions/v1beta1
kind: Deployment"
Supported API equivalent:
"apiVersion: apps/v1
kind: Deployment"
```
1. Revert your `.gitlab-ci.yml` to the previous version. You no longer need to include the
supplemental template `map-deprecated-api`.
1. Continue the deployments as usual.
### Error: error initializing: Looks like "https://kubernetes-charts.storage.googleapis.com" is not a valid chart repository or cannot be reached
As [announced in the official CNCF blog post](https://www.cncf.io/blog/2020/10/07/important-reminder-for-all-helm-users-stable-incubator-repos-are-deprecated-and-all-images-are-changing-location/),
the stable Helm chart repository was deprecated and removed on November 13th, 2020.
You may encounter this error after that date.
Some GitLab features had dependencies on the stable chart. To mitigate the impact, we changed them
to use new official repositories or the [Helm Stable Archive repository maintained by GitLab](https://gitlab.com/gitlab-org/cluster-integration/helm-stable-archive).
Auto Deploy contains [an example fix](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/merge_requests/127).
In Auto Deploy, `v1.0.6+` of `auto-deploy-image` no longer adds the deprecated stable repository to
the `helm` command. If you use a custom chart and it relies on the deprecated stable repository,
specify an older `auto-deploy-image` like this example:
```yaml
include:
- template: Auto-DevOps.gitlab-ci.yml
.auto-deploy:
image: "registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v1.0.5"
```
Keep in mind that this approach stops working when the stable repository is removed,
so you must eventually fix your custom chart.
To fix your custom chart:
1. In your chart directory, update the `repository` value in your `requirements.yaml` file from :
```yaml
repository: "https://kubernetes-charts.storage.googleapis.com/"
```
to:
```yaml
repository: "https://charts.helm.sh/stable"
```
1. In your chart directory, run `helm dep update .` using the same Helm major version as Auto DevOps.
1. Commit the changes for the `requirements.yaml` file.
1. If you previously had a `requirements.lock` file, commit the changes to the file.
If you did not previously have a `requirements.lock` file in your chart,
you do not need to commit the new one. This file is optional, but when present,
it's used to verify the integrity of the downloaded dependencies.
You can find more information in
[issue #263778, "Migrate PostgreSQL from stable Helm repository"](https://gitlab.com/gitlab-org/gitlab/-/issues/263778).
### Error: release .... failed: timed out waiting for the condition
When getting started with Auto DevOps, you may encounter this error when first
deploying your application:
```plaintext
INSTALL FAILED
PURGING CHART
Error: release staging failed: timed out waiting for the condition
```
This is most likely caused by a failed liveness (or readiness) probe attempted
during the deployment process. By default, these probes are run against the root
page of the deployed application on port 5000. If your application isn't configured
to serve anything at the root page, or is configured to run on a specific port
*other* than 5000, this check fails.
If it fails, you should see these failures in the events for the relevant
Kubernetes namespace. These events look like the following example:
```plaintext
LAST SEEN TYPE REASON OBJECT MESSAGE
3m20s Warning Unhealthy pod/staging-85db88dcb6-rxd6g Readiness probe failed: Get http://10.192.0.6:5000/: dial tcp 10.192.0.6:5000: connect: connection refused
3m32s Warning Unhealthy pod/staging-85db88dcb6-rxd6g Liveness probe failed: Get http://10.192.0.6:5000/: dial tcp 10.192.0.6:5000: connect: connection refused
```
To change the port used for the liveness checks, pass
[custom values to the Helm chart](customize.md#customize-values-for-helm-chart)
used by Auto DevOps:
1. Create a directory and file at the root of your repository named `.gitlab/auto-deploy-values.yaml`.
1. Populate the file with the following content, replacing the port values with
the actual port number your application is configured to use:
```yaml
service:
internalPort: <port_value>
externalPort: <port_value>
```
1. Commit your changes.
After committing your changes, subsequent probes should use the newly-defined ports.
The page that's probed can also be changed by overriding the `livenessProbe.path`
and `readinessProbe.path` values (shown in the
[default `values.yaml`](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/blob/master/assets/auto-deploy-app/values.yaml)
file) in the same fashion.
## Development guides
[Development guide for Auto DevOps](../../development/auto_devops.md)
doc/topics/autodevops/troubleshooting.md 0 → 100644
View file @ 7c6ca329
---
stage: Configure
group: Configure
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Troubleshooting Auto DevOps **(FREE)**
The information in this documentation page describes common errors when using
Auto DevOps, and any available workarounds.
## Unable to select a buildpack
Auto Build and Auto Test may fail to detect your language or framework with the
following error:
```plaintext
Step 5/11 : RUN /bin/herokuish buildpack build
---> Running in eb468cd46085
-----> Unable to select a buildpack
The command '/bin/sh -c /bin/herokuish buildpack build' returned a non-zero code: 1
```
The following are possible reasons:
- Your application may be missing the key files the buildpack is looking for.
Ruby applications require a `Gemfile` to be properly detected,
even though it's possible to write a Ruby app without a `Gemfile`.
- No buildpack may exist for your application. Try specifying a
[custom buildpack](customize.md#custom-buildpacks).
## Pipeline that extends Auto DevOps with only / except fails
If your pipeline fails with the following message:
```plaintext
Found errors in your .gitlab-ci.yml:
jobs:test config key may not be used with `rules`: only
```
This error appears when the included job’s rules configuration has been overridden with the `only` or `except` syntax.
To fix this issue, you must either:
- Transition your `only/except` syntax to rules.
- (Temporarily) Pin your templates to the [GitLab 12.10 based templates](https://gitlab.com/gitlab-org/auto-devops-v12-10).
## Failure to create a Kubernetes namespace
Auto Deploy fails if GitLab can't create a Kubernetes namespace and
service account for your project. For help debugging this issue, see
[Troubleshooting failed deployment jobs](../../user/project/clusters/index.md#troubleshooting).
## Detected an existing PostgreSQL database
After upgrading to GitLab 13.0, you may encounter this message when deploying
with Auto DevOps:
```plaintext
Detected an existing PostgreSQL database installed on the
deprecated channel 1, but the current channel is set to 2. The default
channel changed to 2 in of GitLab 13.0.
[...]
```
Auto DevOps, by default, installs an in-cluster PostgreSQL database alongside
your application. The default installation method changed in GitLab 13.0, and
upgrading existing databases requires user involvement. The two installation
methods are:
- **channel 1 (deprecated):** Pulls in the database as a dependency of the associated
Helm chart. Only supports Kubernetes versions up to version 1.15.
- **channel 2 (current):** Installs the database as an independent Helm chart. Required
for using the in-cluster database feature with Kubernetes versions 1.16 and greater.
If you receive this error, you can do one of the following actions:
- You can *safely* ignore the warning and continue using the channel 1 PostgreSQL
database by setting `AUTO_DEVOPS_POSTGRES_CHANNEL` to `1` and redeploying.
- You can delete the channel 1 PostgreSQL database and install a fresh channel 2
database by setting `AUTO_DEVOPS_POSTGRES_DELETE_V1` to a non-empty value and
redeploying.
WARNING:
Deleting the channel 1 PostgreSQL database permanently deletes the existing
channel 1 database and all its data. See
[Upgrading PostgreSQL](upgrading_postgresql.md)
for more information on backing up and upgrading your database.
- If you are not using the in-cluster database, you can set
`POSTGRES_ENABLED` to `false` and re-deploy. This option is especially relevant to
users of *custom charts without the in-chart PostgreSQL dependency*.
Database auto-detection is based on the `postgresql.enabled` Helm value for
your release. This value is set based on the `POSTGRES_ENABLED` CI variable
and persisted by Helm, regardless of whether or not your chart uses the
variable.
WARNING:
Setting `POSTGRES_ENABLED` to `false` permanently deletes any existing
channel 1 database for your environment.
## Error: unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1"
After upgrading your Kubernetes cluster to [v1.16+](stages.md#kubernetes-116),
you may encounter this message when deploying with Auto DevOps:
```plaintext
UPGRADE FAILED
Error: failed decoding reader into objects: unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1"
```
This can occur if your current deployments on the environment namespace were deployed with a
deprecated/removed API that doesn't exist in Kubernetes v1.16+. For example,
if [your in-cluster PostgreSQL was installed in a legacy way](#detected-an-existing-postgresql-database),
the resource was created via the `extensions/v1beta1` API. However, the deployment resource
was moved to the `app/v1` API in v1.16.
To recover such outdated resources, you must convert the current deployments by mapping legacy APIs
to newer APIs. There is a helper tool called [`mapkubeapis`](https://github.com/hickeyma/helm-mapkubeapis)
that works for this problem. Follow these steps to use the tool in Auto DevOps:
1. Modify your `.gitlab-ci.yml` with:
```yaml
include:
- template: Auto-DevOps.gitlab-ci.yml
- remote: https://gitlab.com/shinya.maeda/ci-templates/-/raw/master/map-deprecated-api.gitlab-ci.yml
variables:
HELM_VERSION_FOR_MAPKUBEAPIS: "v2" # If you're using auto-depoy-image v2 or above, please specify "v3".
```
1. Run the job `<environment-name>:map-deprecated-api`. Ensure that this job succeeds before moving
to the next step. You should see something like the following output:
```shell
2020/10/06 07:20:49 Found deprecated or removed Kubernetes API:
"apiVersion: extensions/v1beta1
kind: Deployment"
Supported API equivalent:
"apiVersion: apps/v1
kind: Deployment"
```
1. Revert your `.gitlab-ci.yml` to the previous version. You no longer need to include the
supplemental template `map-deprecated-api`.
1. Continue the deployments as usual.
## Error: error initializing: Looks like "https://kubernetes-charts.storage.googleapis.com" is not a valid chart repository or cannot be reached
As [announced in the official CNCF blog post](https://www.cncf.io/blog/2020/10/07/important-reminder-for-all-helm-users-stable-incubator-repos-are-deprecated-and-all-images-are-changing-location/),
the stable Helm chart repository was deprecated and removed on November 13th, 2020.
You may encounter this error after that date.
Some GitLab features had dependencies on the stable chart. To mitigate the impact, we changed them
to use new official repositories or the [Helm Stable Archive repository maintained by GitLab](https://gitlab.com/gitlab-org/cluster-integration/helm-stable-archive).
Auto Deploy contains [an example fix](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/merge_requests/127).
In Auto Deploy, `v1.0.6+` of `auto-deploy-image` no longer adds the deprecated stable repository to
the `helm` command. If you use a custom chart and it relies on the deprecated stable repository,
specify an older `auto-deploy-image` like this example:
```yaml
include:
- template: Auto-DevOps.gitlab-ci.yml
.auto-deploy:
image: "registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v1.0.5"
```
Keep in mind that this approach stops working when the stable repository is removed,
so you must eventually fix your custom chart.
To fix your custom chart:
1. In your chart directory, update the `repository` value in your `requirements.yaml` file from :
```yaml
repository: "https://kubernetes-charts.storage.googleapis.com/"
```
to:
```yaml
repository: "https://charts.helm.sh/stable"
```
1. In your chart directory, run `helm dep update .` using the same Helm major version as Auto DevOps.
1. Commit the changes for the `requirements.yaml` file.
1. If you previously had a `requirements.lock` file, commit the changes to the file.
If you did not previously have a `requirements.lock` file in your chart,
you do not need to commit the new one. This file is optional, but when present,
it's used to verify the integrity of the downloaded dependencies.
You can find more information in
[issue #263778, "Migrate PostgreSQL from stable Helm repository"](https://gitlab.com/gitlab-org/gitlab/-/issues/263778).
## Error: release .... failed: timed out waiting for the condition
When getting started with Auto DevOps, you may encounter this error when first
deploying your application:
```plaintext
INSTALL FAILED
PURGING CHART
Error: release staging failed: timed out waiting for the condition
```
This is most likely caused by a failed liveness (or readiness) probe attempted
during the deployment process. By default, these probes are run against the root
page of the deployed application on port 5000. If your application isn't configured
to serve anything at the root page, or is configured to run on a specific port
*other* than 5000, this check fails.
If it fails, you should see these failures in the events for the relevant
Kubernetes namespace. These events look like the following example:
```plaintext
LAST SEEN TYPE REASON OBJECT MESSAGE
3m20s Warning Unhealthy pod/staging-85db88dcb6-rxd6g Readiness probe failed: Get http://10.192.0.6:5000/: dial tcp 10.192.0.6:5000: connect: connection refused
3m32s Warning Unhealthy pod/staging-85db88dcb6-rxd6g Liveness probe failed: Get http://10.192.0.6:5000/: dial tcp 10.192.0.6:5000: connect: connection refused
```
To change the port used for the liveness checks, pass
[custom values to the Helm chart](customize.md#customize-values-for-helm-chart)
used by Auto DevOps:
1. Create a directory and file at the root of your repository named `.gitlab/auto-deploy-values.yaml`.
1. Populate the file with the following content, replacing the port values with
the actual port number your application is configured to use:
```yaml
service:
internalPort: <port_value>
externalPort: <port_value>
```
1. Commit your changes.
After committing your changes, subsequent probes should use the newly-defined ports.
The page that's probed can also be changed by overriding the `livenessProbe.path`
and `readinessProbe.path` values (shown in the
[default `values.yaml`](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/blob/master/assets/auto-deploy-app/values.yaml)
file) in the same fashion.
doc/topics/autodevops/upgrading_postgresql.md
View file @ 7c6ca329
......@@ -34,7 +34,7 @@ involves:
so that the in-cluster database does not get modified after the database dump is created.
1. Ensure you have not set `POSTGRES_ENABLED` to `false`, as this setting deletes
any existing channel 1 database. For more information, see
[Detected an existing PostgreSQL database](index.md#detected-an-existing-postgresql-database).
[Detected an existing PostgreSQL database](troubleshooting.md#detected-an-existing-postgresql-database).
NOTE:
If you have configured Auto DevOps to have staging,
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7