Merge branch 'pb-clarify-batched-bg-migration-troubleshooting' into 'master'

Update docs for failed batched bg migrations

See merge request gitlab-org/gitlab!71585

doc/update/index.md

@@ -92,6 +92,16 @@ To check the status of [batched background migrations](../user/admin_area/monito
 
    ![queued batched background migrations table](img/batched_background_migrations_queued_v14_0.png)
 
+The status of batched background migrations can also be queried directly in the database.
+
+1. Log into a `psql` prompt according to the directions for your instance's installation method
+(for example, `sudo gitlab-psql` for Omnibus installations).
+1. Run the following query in the `psql` session to see details on incomplete batched background migrations:
+
+   ```sql
+   select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;
+   ```
+
 **For Omnibus installations**
 
 You can also run:

doc/user/admin_area/monitoring/background_migrations.md

@@ -91,13 +91,95 @@ Expected batched background migration for the given configuration to be marked a
   {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"push_event_payloads", :column_name=>"event_id", :job_arguments=>[["event_id"], ["event_id_convert_to_bigint"]]}
 ```
 
-To fix this error:
+First, check if you have followed the [version-specific upgrade instructions for 14.2](../../../update/index.md#1420).
+If you have, you can [manually finish the batched background migration](#manually-finishing-a-batched-background-migration).
+If you haven't, choose one of the following methods:
 
-1. Update to either 14.0.5 or 14.1.
-1. [Check the status](#check-the-status-of-background-migrations) of the batched background migration from the error message, and make sure it is listed as finished. If it is still active, either wait until it is done, or finalize it manually using the command suggested in the error, for example:
+1. [Rollback and upgrade](#roll-back-and-follow-the-required-upgrade-path) through one of the required
+versions before updating to 14.2+.
+1. [Roll forward](#roll-forward-and-finish-the-migrations-on-the-upgraded-version), staying on the current
+version and manually and ensure that the batched migrations complete successfully.
+
+#### Roll back and follow the required upgrade path
+
+1. [Rollback and restore the previously installed version](../../../update/restore_after_failure.md#roll-back-to-an-earlier-version-and-restore-a-backup)
+1. Update to either 14.0.5 or 14.1 **before** updating to 14.2+
+1. [Check the status](#check-the-status-of-background-migrations) of the batched background migrations and
+make sure they are all marked as finished before attempting to upgrade again. If any remain marked as active,
+you can [manually finish them](#manually-finishing-a-batched-background-migration).
+
+#### Roll forward and finish the migrations on the upgraded version
+
+##### For a deployment with downtime
+
+To run all the batched background migrations, it can take a significant amount of time
+depending on the size of your GitLab installation.
+
+1. [Check the status](#check-the-status-of-background-migrations) of the batched background migrations in the
+database, and [manually run them](#manually-finishing-a-batched-background-migration) with the appropriate
+arguments until the status query returns no rows.
+1. When the status of all of all them is marked as complete, re-run migrations for your installation.
+1. [Complete the database migrations](../../../administration/raketasks/maintenance.md#run-incomplete-database-migrations) from your GitLab upgrade:
+
+   ```plaintext
+   sudo gitlab-rake db:migrate
+   ```
+
+1. Run a reconfigure:
+
+     ```plaintext
+   sudo gitlab-ctl reconfigure
+   ```
+
+1. Finish the upgrade for your installation.
+
+##### For a no-downtime deployment
+
+As the failing migrations are post-deployment migrations, you can remain on a running instance of the upgraded
+version and wait for the batched background migrations to finish normally.
+
+1. [Check the status](#check-the-status-of-background-migrations) of the batched background migration from
+the error message, and make sure it is listed as finished. If it is still active, either wait until it is done,
+or [manually finish it](#manually-finishing-a-batched-background-migration).
+1. Re-run migrations for your installation, so the remaining post-deployment migrations finish.
+
+### Manually finishing a batched background migration
+
+If you need to manually finish a batched background migration due to an
+error, you can run:
+
+```shell
+sudo gitlab-rake gitlab:background_migrations:finalize[<job_class_name>,<table_name>,<column_name>,'<job_arguments>']
+```
+
+Replace the values in angle brackets with the correct
+arguments. For example, if you receive an error similar to this:
+
+```plaintext
+StandardError: An error has occurred, all later migrations canceled:
+
+Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
+  {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"push_event_payloads", :column_name=>"event_id", :job_arguments=>[["event_id"], ["event_id_convert_to_bigint"]]}
+```
+
+Plug the arguments from the error message into the command:
 
 ```shell
 sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,push_event_payloads,event_id,'[["event_id"]\, ["event_id_convert_to_bigint"]]']
 ```
 
-1. You can now update to GitLab 14.2 or higher.
+If you need to manually run a batched background migration to continue an upgrade, you can
+[check the status](#check-the-status-of-background-migrations) in the database and get the
+arguments from the query results. For example, if the query returns this:
+
+```plaintext
+            job_class_name             | table_name | column_name |           job_arguments
+---------------------------------------+------------+-------------+------------------------------------
+ CopyColumnUsingBackgroundMigrationJob | events     | id          | [["id"], ["id_convert_to_bigint"]]
+ ```
+
+The results from the query can be plugged into the command:
+
+```shell
+sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
+```