Merge branch 'bprescott/docs/20210930-batchedmigr' into 'master'

Clarify 14.1 on the upgrade path

See merge request gitlab-org/gitlab!71500

doc/update/index.md

@@ -12,6 +12,11 @@ GitLab version is, if you're upgrading to a major version, and so on.
 
 Make sure to read the whole page as it contains information related to every upgrade method.
 
+NOTE:
+Upgrade GitLab to the latest available patch release, for example `13.8.8` rather than `13.8.0`.
+This includes [versions you must stop at on the upgrade path](#upgrade-paths) as there may
+be fixes for issues relating to the upgrade process.
+
 The [maintenance policy documentation](../policy/maintenance.md)
 has additional information about upgrading, including:
 
@@ -76,22 +81,44 @@ See the guide to [plan your GitLab upgrade](plan_your_upgrade.md).
 
 ## Checking for background migrations before upgrading
 
-Certain major/minor releases may require different migrations to be
-finished before you update to the newer version.
+Certain releases may require different migrations to be
+finished before you update to the newer version. Additionally check
+[batched migrations](#batched-background-migrations) from GitLab 14.0.
 
 Decrease the time required to complete these migrations by increasing the number of
 [Sidekiq workers](../administration/operations/extra_sidekiq_processes.md)
 that can process jobs in the `background_migration` queue.
 
-**For GitLab 14.0 and newer**
+**For Omnibus installations:**
+
+```shell
+sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
+```
+
+**For installations from source:**
+
+```shell
+cd /home/git/gitlab
+sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
+```
+
+### Batched background migrations
+
+GitLab 14.0 introduced [batched background migrations](../user/admin_area/monitoring/background_migrations.md).
+
+Some installations [may need to run GitLab 14.0 for at least a day](#1400) to complete the database changes introduced by that upgrade.
 
-To check the status of [batched background migrations](../user/admin_area/monitoring/background_migrations.md):
+#### Check the status of batched background migrations
+
+To check the status of batched background migrations:
 
 1. On the top bar, select **Menu > Admin**.
 1. On the left sidebar, select **Monitoring > Background Migrations**.
 
    ![queued batched background migrations table](img/batched_background_migrations_queued_v14_0.png)
 
+All migrations must have a `Finished` status before you upgrade GitLab.
+
 The status of batched background migrations can also be queried directly in the database.
 
 1. Log into a `psql` prompt according to the directions for your instance's installation method
@@ -102,20 +129,14 @@ The status of batched background migrations can also be queried directly in the
    select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;
    ```
 
-**For Omnibus installations**
-
-You can also run:
+If the migrations are not finished and you try to update to a later version,
+GitLab prompts you with an error:
 
-```shell
-sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
+```plaintext
+Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
 ```
 
-**For installations from source**
-
-```shell
-cd /home/git/gitlab
-sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
-```
+If you get this error, [check the batched background migration options](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished) to complete the upgrade.
 
 ### What do I do if my background migrations are stuck?
 
@@ -148,6 +169,10 @@ pending_job_classes = scheduled_queue.select { |job| job["class"] == "Background
 pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) }
 ```
 
+**Batched migrations (GitLab 14.0 and newer):**
+
+See [troubleshooting batched background migrations](../user/admin_area/monitoring/background_migrations.md#troubleshooting).
+
 ## Dealing with running CI/CD pipelines and jobs
 
 If you upgrade your GitLab instance while the GitLab Runner is processing jobs, the trace updates fail. When GitLab is back online, the trace updates should self-heal. However, depending on the error, the GitLab Runner either retries or eventually terminates job handling.
@@ -197,7 +222,7 @@ Find where your version sits in the upgrade path below, and upgrade GitLab
 accordingly, while also consulting the
 [version-specific upgrade instructions](#version-specific-upgrading-instructions):
 
-`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [latest `13.12.Z`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/)
+`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [`13.12.15`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.1.Z`](#1410) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/)
 
 The following table, while not exhaustive, shows some examples of the supported
 upgrade paths.
@@ -320,9 +345,11 @@ Git 2.33.x and later is required. We recommend you use the
 
 ### 14.3.0
 
+- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases).
+- Ensure [batched background migrations finish](#batched-background-migrations) before upgrading
+  to 14.3.Z from earlier GitLab 14 releases.
 - Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../install/installation.md#2-ruby)
 for how to proceed.
-
 - GitLab 14.3.0 contains post-deployment migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below:
 
   - [`ci_builds.id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70245)
@@ -343,13 +370,10 @@ for how to proceed.
 
 ### 14.2.0
 
-- Due to an issue where `BatchedBackgroundMigrationWorkers` were
-  [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345)
-  for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106)
-  and a [14.0.Z](#1400) version was released. If you haven't updated to 14.0.5, you need
-  to update to at least 14.1.0 that contains the same fix before you update to 14.2.
+- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases).
+- Ensure [batched background migrations finish](#batched-background-migrations) before upgrading
+  to 14.2.Z from earlier GitLab 14 releases.
 - GitLab 14.2.0 contains background migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below:
-
   - [`ci_build_needs`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65216)
   - [`ci_build_trace_chunks`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66123)
   - [`ci_builds_runner_session`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66433)
@@ -372,35 +396,24 @@ for how to proceed.
 
 ### 14.1.0
 
-Due to an issue where `BatchedBackgroundMigrationWorkers` were
-[not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345)
-for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106)
-and a [14.0.Z](#1400) version was released. If you haven't updated to 14.0.5, you need
-to update to at least 14.1.0 that contains the same fix before you update to
-a later version.
-
-After you update to 14.1.0,
-[batched background migrations need to finish](../user/admin_area/monitoring/background_migrations.md#check-the-status-of-background-migrations)
-before you update to a later version.
-
-If the migrations are not finished and you try to update to a later version,
-you'll see an error like:
-
-```plaintext
-Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
-```
-
-See how to [resolve this error](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished).
+- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases)
+  but can upgrade to 14.1.Z.
+- It is not required for instances already running 14.0.5 (or higher) to stop at 14.1.Z.
+  14.1 is included on the upgrade path for the broadest compatibility
+  with self-managed installations, and ensure 14.0.0-14.0.4 installations do not
+  encounter issues with [batched background migrations](#batched-background-migrations).
 
 ### 14.0.0
 
+- Database changes made by the upgrade to GitLab 14.0 can take hours or days to complete on larger GitLab instances.
+  These [batched background migrations](#batched-background-migrations) update whole database tables to mitigate primary key overflow and must be finished before upgrading to GitLab 14.2 or higher.
 - Due to an issue where `BatchedBackgroundMigrationWorkers` were
   [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345)
   for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106)
-  that requires an update to at least 14.0.5.
+  that requires an update to at least 14.0.5. The fix was also released in [14.1.0](#1410).
 
   After you update to 14.0.5 or a later 14.0 patch version,
-  [batched background migrations need to finish](../user/admin_area/monitoring/background_migrations.md#check-the-status-of-background-migrations)
+  [batched background migrations need to finish](#batched-background-migrations)
   before you update to a later version.
 
   If the migrations are not finished and you try to update to a later version,
@@ -419,6 +432,16 @@ See how to [resolve this error](../user/admin_area/monitoring/background_migrati
   You should instead follow a [supported upgrade path](#upgrade-paths).
 - The support of PostgreSQL 11 [has been dropped](../install/requirements.md#database). Make sure to [update your database](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to version 12 before updating to GitLab 14.0.
 
+#### Upgrading to later 14.Y releases
+
+- Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later,
+  because of [batched background migrations](#batched-background-migrations).
+  1. Upgrade first to either:
+     - 14.0.5 or a later 14.0.Z patch release.
+     - 14.1.0 or a later 14.1.Z patch release.
+  1. [Batched background migrations need to finish](#batched-background-migrations)
+     before you update to a later version [and may take longer than usual](#1400).
+
 ### 13.11.0
 
 Git 2.31.x and later is required. We recommend you use the

doc/user/admin_area/monitoring/background_migrations.md

@@ -24,7 +24,7 @@ prevent integer overflow for some tables.
 ## Check the status of background migrations
 
 All migrations must have a `Finished` status before you [upgrade GitLab](../../../update/index.md).
-You can [check the status of existing migrations](../../../update/index.md#checking-for-background-migrations-before-upgrading).
+You can [check the status of existing migrations](../../../update/index.md#batched-background-migrations).
 
 ## Enable or disable batched background migrations