Merge branch 'pokstad1-praefect-docs-reconcile-subcmd' into 'master'

Update Gitaly to 12.9.0-rc5

Closes gitaly#2448

See merge request gitlab-org/gitlab!27631

GITALY_SERVER_VERSION

-12.9.0-rc1
+12.9.0-rc5

changelogs/unreleased/pokstad1-praefect-docs-reconcile-subcmd.yml

+---
+title: Update Gitaly to 12.9.0-rc5
+merge_request: 27631
+author:
+type: added

doc/administration/gitaly/praefect.md

@@ -682,6 +682,29 @@ for example behind a load balancer, `failover_enabled` should be disabled. The r
 is no coordination that currently happens across different Praefect instances, so there could be a situation where
 two Praefect instances think two different Gitaly nodes are the primary.
 
+## Backend Node Recovery
+
+When a Praefect backend node fails and is no longer able to
+replicate changes, the backend node will start to drift from the primary. If
+that node eventually recovers, it will need to be reconciled with the current
+primary. The primary node is considered the single source of truth for the
+state of a shard. The Praefect `reconcile` subcommand allows for the manual
+reconciliation between a backend node and the current primary.
+
+Run the following command on the Praefect server after all placeholders
+(`<virtual-storage>` and `<target-storage>`) have been replaced:
+
+```shell
+sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml reconcile -virtual <virtual-storage> -target <target-storage>
+```
+
+- Replace the placeholder `<virtual-storage>` with the virtual storage containing the backend node storage to be checked.
+- Replace the placeholder `<target-storage>` with the backend storage name.
+
+The command will return a list of repositories that were found to be
+inconsistent against the current primary. Each of these inconsistencies will
+also be logged with an accompanying replication job ID.
+
 ## Grafana
 
 Grafana is included with GitLab, and can be used to monitor your Praefect