Merge branch 'canary-ingress-rails' into 'master'

Update the latest Auto Deploy template for V2.0.0 auto-deploy-image

Closes #218522

See merge request gitlab-org/gitlab!39438

doc/topics/autodevops/customize.md

@@ -332,7 +332,7 @@ applications.
 | `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME` | From GitLab 11.11, used to set a username to connect to the Helm repository. Defaults to no credentials. Also set `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD`. |
 | `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD` | From GitLab 11.11, used to set a password to connect to the Helm repository. Defaults to no credentials. Also set `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME`. |
 | `AUTO_DEVOPS_DEPLOY_DEBUG`              | From GitLab 13.1, if this variable is present, Helm will output debug logs. |
-| `AUTO_DEVOPS_ALLOW_TO_FORCE_DEPLOY_V<N>` | From [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) v1.0.0, if this variable is present, a new major version of chart is forcibly deployed. [More details](upgrading_chart.md#ignore-warning-and-continue-deploying) |
+| `AUTO_DEVOPS_ALLOW_TO_FORCE_DEPLOY_V<N>` | From [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) v1.0.0, if this variable is present, a new major version of chart is forcibly deployed. For more information, see [Ignore warnings and continue deploying](upgrading_auto_deploy_dependencies.md#ignore-warnings-and-continue-deploying). |
 | `AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE` | From GitLab 12.5, used in combination with [ModSecurity feature flag](../../user/clusters/applications.md#web-application-firewall-modsecurity) to toggle [ModSecurity's `SecRuleEngine`](https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#SecRuleEngine) behavior. Defaults to `DetectionOnly`. |
 | `BUILDPACK_URL`                         | Buildpack's full URL. Can point to either [a Git repository URL or a tarball URL](#custom-buildpacks). |
 | `CANARY_ENABLED`                        | From GitLab 11.0, used to define a [deploy policy for canary environments](#deploy-policy-for-canary-environments). |

doc/topics/autodevops/stages.md

@@ -466,7 +466,7 @@ application runs.
 
 ### Upgrade auto-deploy-app Chart
 
-You can upgrade auto-deploy-app chart by following the [upgrade guide](upgrading_chart.md).
+You can upgrade the auto-deploy-app chart by following the [upgrade guide](upgrading_auto_deploy_dependencies.md).
 
 ### Workers
 

doc/topics/autodevops/upgrading_auto_deploy_dependencies.md

+---
+stage: Release
+group: Progressive Delivery
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+type: reference
+---
+
+# Upgrading deployments for newer Auto Deploy dependencies (Auto Deploy template, auto-deploy-image and auto-deploy-app chart)
+
+[Auto Deploy](stages.md#auto-deploy) is a feature that deploys your application to a Kubernetes cluster.
+It consists of several dependencies:
+
+- [Auto Deploy template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) is a set of pipeline jobs and scripts that makes use of `auto-deploy-image`.
+- [`auto-deploy-image`](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) is the executable image that communicates with the Kubernetes cluster.
+- [`auto-deploy-app chart`](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/tree/master/assets/auto-deploy-app) is the Helm chart for deploying your application.
+
+The `auto-deploy-image` and `auto-deploy-app` charts use [Semantic Versioning](https://semver.org/).
+By default, your Auto DevOps project keeps using the stable and non-breaking version.
+However, these dependencies could be upgraded in a major version release of GitLab
+with breaking changes requiring you to upgrade your deployments.
+
+This guide explains how to upgrade your deployments with newer or different major versions of Auto Deploy dependencies.
+
+## Verify dependency versions
+
+The process to check the current versions differs depending on which template you
+are using. First verify which template is in use:
+
+- For self-managed instances, the [stable Auto Deploy template bundled with the GitLab package](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml)
+  is being used.
+- [The GitLab.com stable Auto Deploy template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml)
+  is being used if **one** of the following is true:
+  - Your Auto DevOps project doesn't have a `.gitlab-ci.yml` file.
+  - Your Auto DevOps project has a `.gitlab-ci.yml` and [includes](../../ci/yaml/README.md#includetemplate)
+    the `Auto-DevOps.gitlab-ci.yml` template.
+- [The latest Auto Deploy template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.latest.gitlab-ci.yml)
+  is being used if **both** of the following is true:
+  - Your Auto DevOps project has a `.gitlab-ci.yml` file and [includes](../../ci/yaml/README.md#includetemplate)
+    the `Auto-DevOps.gitlab-ci.yml` template.
+  - It also includes [the latest Auto Deploy template](#early-adopters)
+
+If you know what template is being used:
+
+- The `auto-deploy-image` version is in the template (for example `auto-deploy-image:v1.0.3`).
+- The `auto-deploy-app` chart version is [in the auto-deploy-image repository](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/blob/v1.0.3/assets/auto-deploy-app/Chart.yaml)
+  (for example `version: 1.0.3`).
+
+## Compatibility
+
+The following table explains the version compatibility between GitLab and Auto Deploy dependencies:
+
+| GitLab version   | `auto-deploy-image` version | Notes |
+|------------------|-----------------------------|-------|
+| v10.0 to v14.0   | v0.1.0 to v2.0.0            | v0 and v1 auto-deploy-image are backwards compatible. |
+| v13.4 and higher | v2.0.0 and higher           | v2 auto-deploy-image contains breaking changes, as explained in the [upgrade guide](#upgrade-deployments-to-the-v2-auto-deploy-image). |
+
+You can find the current stable version of auto-deploy-image in the [Auto Deploy stable template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml).
+
+## Upgrade Guide
+
+Projects using Auto DevOps must use the unmodified chart managed by GitLab.
+[Customized charts](customize.md#custom-helm-chart) are unsupported.
+
+### Upgrade deployments to the v1 `auto-deploy-image`
+
+The v1 chart is backward compatible with the v0 chart, so no configuration changes are needed.
+
+### Upgrade deployments to the v2 `auto-deploy-image`
+
+The v2 auto-deploy-image contains multiple dependency and architectural changes.
+If your Auto DevOps project has an active environment deployed with the v1 `auto-deploy-image`,
+please proceed with the following upgrade guide. Otherwise, you can skip this process.
+
+#### Helm 3
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/228609) in GitLab 13.4.
+
+The `auto-deploy-image` uses the Helm binary to manipulate the releases.
+Previously, `auto-deploy-image` used Helm v2, which used Tiller in a cluster.
+In the v2 `auto-deploy-image`, it uses Helm v3 that doesn't require Tiller anymore.
+
+If your Auto DevOps project has an active environment that was deployed with the v1
+`auto-deploy-image`, use the following steps to upgrade to v2, which uses Helm 3:
+
+1. Modify your `.gitlab-ci.yml` with:
+
+   ```yaml
+   include:
+     - template: Auto-DevOps.gitlab-ci.yml
+     - remote: https://gitlab.com/hfyngvason/ci-templates/-/raw/master/Helm-2to3.gitlab-ci.yml
+
+   variables:
+     # If this variable is not present, the migration jobs will not show up
+     MIGRATE_HELM_2TO3: "true"
+
+   .auto-deploy:
+     image: registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v2.0.0-beta.1
+     variables:
+       AUTO_DEVOPS_FORCE_DEPLOY_V2: 1
+   ```
+
+1. Run the `<environment-name>:helm-2to3:migrate` job.
+1. Deploy your environment as usual. This deployment uses Helm 3.
+1. If the deployment succeeds, you can safely run `environment:helm-2to3:cleanup`.
+   This deletes all Helm 2 release data from the namespace.
+
+   If you accidentally delete the Helm 2 releases before you are ready, the `<environment-name>:helm2to3:migrate`
+   job saves a backup for 1 week in a job artifact called `helm-2-release-backups`.
+   The backup is in a Kubernetes manifest file that can be restored using
+   `kubectl apply -f $backup`.
+1. Remove the `MIGRATE_HELM_2TO3` variable.
+
+#### Traffic routing change for canary deployments and incremental rollouts
+
+> [Introduced](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/merge_requests/109) in GitLab 13.4.
+
+Auto Deploy supports advanced deployment strategies such as [canary deployments](customize.md#deploy-policy-for-canary-environments)
+and [incremental rollouts](../../ci/environments/incremental_rollouts.md).
+
+Previously, `auto-deploy-image` created one service to balance the traffic between
+unstable and stable tracks by changing the replica ratio. In the v2 `auto-deploy-image`,
+it controls the traffic with [Canary Ingress](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary).
+
+For more details, see the [v2 `auto-deploy-app` chart resource architecture](#v2-chart-resource-architecture).
+
+If your Auto DevOps project has active `canary` or `rollout` track releases in the
+`production` environment deployed with the v1 `auto-deploy-image`, use the following
+steps to upgrade to v2:
+
+1. Verify your project is [using the v1 `auto-deploy-image`](#verify-dependency-versions).
+   If not, [specify the version](#use-a-specific-version-of-auto-deploy-dependencies).
+1. If you're in the process of deploying `canary` or `rollout` deployments, promote
+   them to `production` first to delete the unstable tracks.
+1. Verify your project is [using the v2 `auto-deploy-image`](#verify-dependency-versions).
+   If not, [specify the version](#use-a-specific-version-of-auto-deploy-dependencies).
+1. Add an `AUTO_DEVOPS_FORCE_DEPLOY_V2` environment variable with a value of `true`
+   in the GitLab CI/CD settings.
+1. Create a new pipeline and run the `production` job to renew the resource architecture
+   with the v2 `auto-deploy-app chart`.
+1. Remove the `AUTO_DEVOPS_FORCE_DEPLOY_V2` environment variable.
+
+### Use a specific version of Auto Deploy dependencies
+
+To use a specifc version of Auto Deploy dependencies, specify the previous Auto Deploy
+stable template that contains the [desired version of `auto-deploy-image` and `auto-deploy-app`](#verify-dependency-versions).
+
+For example, if the template is bundled in GitLab v13.3, change your `.gitlab-ci.yml` to:
+
+```yaml
+include:
+  - template: Auto-DevOps.gitlab-ci.yml
+  - remote: https://gitlab.com/gitlab-org/gitlab/-/blob/v13.3.0-ee/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml
+```
+
+### Ignore warnings and continue deploying
+
+If you are certain that the new chart version is safe to be deployed, you can add
+the `AUTO_DEVOPS_FORCE_DEPLOY_V<major-version-number>` [environment variable](customize.md#build-and-deployment)
+to force the deployment to continue.
+
+For example, if you want to deploy the `v2.0.0` chart on a deployment that previously
+used the `v0.17.0` chart, add `AUTO_DEVOPS_FORCE_DEPLOY_V2`.
+
+## Early adopters
+
+If you want to use the latest beta or unstable version of `auto-deploy-image`, include
+the latest Auto Deploy template into your `.gitlab-ci.yml`:
+
+```yaml
+include:
+  - template: Auto-DevOps.gitlab-ci.yml
+  - remote: https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.latest.gitlab-ci.yml
+```
+
+CAUTION: **Warning:**
+Using a beta or unstable `auto-deploy-image` could cause unrecoverable damage to
+your environments. Do not test it with important projects or environments.
+
+The next stable template update is [planned for GitLab v14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/232788).
+
+## Resource Architectures of the `auto-deploy-app` chart
+
+### v0 and v1 chart resource architecture
+
+```mermaid
+graph TD;
+subgraph gl-managed-app
+Z[Nginx Ingress]
+end
+Z[Nginx Ingress] --> A(Ingress);
+Z[Nginx Ingress] --> B(Ingress);
+subgraph stg namespace
+B[Ingress] --> H(...);
+end
+subgraph prd namespace
+A[Ingress] --> D(Service);
+D[Service] --> E(Deployment:Pods:app:stable);
+D[Service] --> F(Deployment:Pods:app:canary);
+D[Service] --> I(Deployment:Pods:app:rollout);
+E(Deployment:Pods:app:stable)---id1[(Pods:Postgres)]
+F(Deployment:Pods:app:canary)---id1[(Pods:Postgres)]
+I(Deployment:Pods:app:rollout)---id1[(Pods:Postgres)]
+end
+```
+
+### v2 chart resource architecture
+
+```mermaid
+graph TD;
+subgraph gl-managed-app
+Z[Nginx Ingress]
+end
+Z[Nginx Ingress] --> A(Ingress);
+Z[Nginx Ingress] --> B(Ingress);
+Z[Nginx Ingress] --> |If canary is present or incremental rollout/|J(Canary Ingress);
+subgraph stg namespace
+B[Ingress] --> H(...);
+end
+subgraph prd namespace
+subgraph stable track
+A[Ingress] --> D[Service];
+D[Service] --> E(Deployment:Pods:app:stable);
+end
+subgraph canary track
+J(Canary Ingress) --> K[Service]
+K[Service] --> F(Deployment:Pods:app:canary);
+end
+E(Deployment:Pods:app:stable)---id1[(Pods:Postgres)]
+F(Deployment:Pods:app:canary)---id1[(Pods:Postgres)]
+end
+```
+
+## Troubleshooting
+
+### Major version mismatch warning
+
+If deploying a chart that has a major version that is different from the previous one,
+the new chart might not be correctly deployed. This could be due to an architectural
+change. If that happens, the deployment job fails with a message similar to:
+
+```plaintext
+*************************************************************************************
+                                   [WARNING]
+Detected a major version difference between the the chart that is currently deploying (auto-deploy-app-v0.7.0), and the previously deployed chart (auto-deploy-app-v1.0.0).
+A new major version might not be backward compatible with the current release (production). The deployment could fail or be stuck in an unrecoverable status.
+...
+```
+
+To clear this error message and resume deployments, you must do one of the following:
+
+- Manually [upgrade the chart version](#upgrade-guide).
+- [Use a specific chart version](#use-a-specific-version-of-auto-deploy-dependencies).
+
+### Error: `missing key "app.kubernetes.io/managed-by": must be set to "Helm"`
+
+If your cluster has a deployment that was deployed with the v1 `auto-deploy-image`,
+you might encounter the following error:
+
+- `Error: rendered manifests contain a resource that already exists. Unable to continue with install: Secret "production-postgresql" in namespace "<project-name>-production" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "production-postgresql"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "<project-name>-production"`
+
+This is because the previous deployment was deployed with Helm2, which is not compatible with Helm3.
+To resolve the problem, please follow the [upgrade guide](#upgrade-deployments-to-the-v2-auto-deploy-image).

doc/topics/autodevops/upgrading_chart.md

-# Upgrading auto-deploy-app chart for Auto DevOps
+---
+redirect_to: 'upgrading_auto_deploy_dependencies.md'
+---
 
-Auto DevOps provides the auto-deploy-app chart for deploying your application to the
-Kubernetes cluster with Helm/Tiller. Major version changes of this chart could have
-a significantly different resource architecture, and may not be backwards compatible.
-
-This guide provides instructions on how to upgrade your deployments to use the latest
-chart and resource architecture.
-
-## Compatibility
-
-The following table lists the version compatibility between GitLab and [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) (with the [auto-deploy-app chart](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/tree/master/assets/auto-deploy-app)).
-
-| GitLab version   | auto-deploy-image version | Notes                                      |
-|------------------|---------------------------|--------------------------------------------|
-| v10.0 and higher | v0.1.0 and higher         | v0 and v1 charts are backwards compatible. |
-
-## Upgrade Guide
-
-The Auto DevOps project must use the unmodified chart managed by GitLab.
-[Customized charts](customize.md#custom-helm-chart) are unsupported.
-
-### v1 chart
-
-The v1 chart is backward compatible with the v0 chart, so no configuration changes are needed.
-
-## Troubleshooting
-
-### Major version mismatch warning
-
-If deploying a chart that has a major version that is different from the previous one,
-the new chart might not be correctly deployed. This could be due to an architectural
-change. If that happens, the deployment job fails with a message similar to:
-
-```plaintext
-*************************************************************************************
-                                   [WARNING]
-Detected a major version difference between the the chart that is currently deploying (auto-deploy-app-v0.7.0), and the previously deployed chart (auto-deploy-app-v1.0.0).
-A new major version might not be backward compatible with the current release (production). The deployment could fail or be stuck in an unrecoverable status.
-...
-```
-
-To clear this error message and resume deployments, you must do one of the following:
-
-- Manually [upgrade the chart version](#upgrade-guide).
-- [Use a specific chart version](#use-a-specific-chart-version).
-
-#### Use a specific chart version
-
-To use a specific chart version, you must specify a corresponding version of [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image).
-Do this by [customizing the image in your `.gitlab-ci.yml`](customize.md#customizing-gitlab-ciyml).
-
-For example, create the following `.gitlab-ci.yml` file in the project. It configures Auto DevOps
-to use [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) version `v0.17.0`
-for deployment jobs. It will download the chart from [chart repository](https://charts.gitlab.io/):
-
-```yaml
-include:
-  - template: Auto-DevOps.gitlab-ci.yml
-
-.auto-deploy:
-  image: "registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v0.17.0"
-```
-
-#### Ignore warning and continue deploying
-
-If you are certain that the new chart version is safe to be deployed,
-you can add the `AUTO_DEVOPS_FORCE_DEPLOY_V<N>` [environment variable](customize.md#build-and-deployment)
-to force the deployment to continue, where `<N>` is the major version.
-
-For example, if you want to deploy the v2.0.0 chart on a deployment that previously
-used the v0.17.0 chart, add `AUTO_DEVOPS_FORCE_DEPLOY_V2`.
+This document was moved to [another location](upgrading_auto_deploy_dependencies.md).

lib/gitlab/ci/templates/Jobs/Deploy.latest.gitlab-ci.yml

 .auto-deploy:
-  image: "registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v1.0.3"
+  image: "registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v2.0.0-beta.2"
   dependencies: []
 
 review:
@@ -91,7 +91,7 @@ canary:
     - auto-deploy ensure_namespace
     - auto-deploy initialize_tiller
     - auto-deploy create_secret
-    - auto-deploy deploy canary
+    - auto-deploy deploy canary 50
   environment:
     name: production
     url: http://$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN
@@ -114,7 +114,6 @@ canary:
     - auto-deploy create_secret
     - auto-deploy deploy
     - auto-deploy delete canary
-    - auto-deploy delete rollout
     - auto-deploy persist_environment_url
   environment:
     name: production
@@ -163,9 +162,7 @@ production_manual:
     - auto-deploy ensure_namespace
     - auto-deploy initialize_tiller
     - auto-deploy create_secret
-    - auto-deploy deploy rollout $ROLLOUT_PERCENTAGE
-    - auto-deploy scale stable $((100-ROLLOUT_PERCENTAGE))
-    - auto-deploy delete canary
+    - auto-deploy deploy canary $ROLLOUT_PERCENTAGE
     - auto-deploy persist_environment_url
   environment:
     name: production