Merge branch 'docs-geo-update-patroni-support' into 'master'

Geo - Update Patroni support documentation

See merge request gitlab-org/gitlab!49986

changelogs/unreleased/docs-geo-update-patroni-support.yml

+---
+title: Update documentation for setting up database replication with Patroni on a
+  Geo secondary node
+merge_request: 49986
+author:
+type: other

doc/administration/geo/setup/database.md

@@ -497,54 +497,135 @@ For instructions about how to set up Patroni on the primary node, see the
 If you are currently using `repmgr` on your Geo primary, see [these instructions](#migrating-from-repmgr-to-patroni) for migrating from `repmgr` to Patroni.
 
 A production-ready and secure setup requires at least three Patroni instances on
-the primary, and a similar configuration on the secondary nodes. Be sure to use
-password credentials and other database best practices.
+the primary site, and a similar configuration on the secondary sites. Be sure to
+use password credentials and other database best practices.
 
 Similar to `repmgr`, using Patroni on a secondary node is optional.
 
-To set up database replication with Patroni on a secondary node, configure a
-_permanent replication slot_ on the primary node's Patroni cluster, and ensure
-password authentication is used.
-
-On Patroni instances for the primary node, add the following to the
-`/etc/gitlab/gitlab.rb` file:
-
-```ruby
-# You need one entry for each secondary, with a unique name following PostgreSQL slot_name constraints:
-#
-# Configuration syntax will be: 'unique_slotname' => { 'type' => 'physical' },
-# We don't support setting a permanent replication slot for logical replication type
-patroni['replication_slots'] = {
-    'geo_secondary' => { 'type' => 'physical' }
-}
-
-postgresql['md5_auth_cidr_addresses'] = [
-  'PATRONI_PRIMARY1_IP/32', 'PATRONI_PRIMARY2_IP/32', 'PATRONI_PRIMARY3_IP/32', 'PATRONI_PRIMARY_PGBOUNCER/32',
-  'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32' # we list all secondary instances as they can all become a Standby Leader
-  # any other instance that needs access to the database as per documentation
-]
-
-postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH'
-postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH'
-postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH'
-```
-
-On Patroni instances for the secondary node, add the following to the
-`/etc/gitlab/gitlab.rb` file:
-
-```ruby
-postgresql['md5_auth_cidr_addresses'] = [
-  'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32',
-  # any other instance that needs access to the database as per documentation
-]
-
-patroni['enable'] = true
-patroni['standby_cluster']['enable'] = true
-patroni['standby_cluster']['host'] = 'PATRONI_PRIMARY_LEADER_IP' # this needs to be changed anytime the primary Leader changes
-patroni['standby_cluster']['port'] = 5432
-patroni['standby_cluster']['primary_slot_name'] = 'geo_secondary' # or the unique replication slot name you setup before
-patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD'
-```
+### Step 1. Configure Patroni permanent replication slot on the primary site
+
+To set up database replication with Patroni on a secondary node, we need to
+configure a _permanent replication slot_ on the primary node's Patroni cluster,
+and ensure password authentication is used.
+
+For each Patroni instance on the primary site **starting on the Patroni
+Leader instance**:
+
+1. SSH into your Patroni instance and login as root:
+
+   ```shell
+   sudo -i
+   ```
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following:
+
+   ```ruby
+   consul['enable'] = true
+   consul['configuration'] = {
+     retry_join: %w[CONSUL_PRIMARY1_IP CONSULT_PRIMARY2_IP CONSULT_PRIMARY3_IP]
+   }
+
+   repmgr['enable'] = false
+
+   # You need one entry for each secondary, with a unique name following PostgreSQL slot_name constraints:
+   #
+   # Configuration syntax will be: 'unique_slotname' => { 'type' => 'physical' },
+   # We don't support setting a permanent replication slot for logical replication type
+   patroni['replication_slots'] = {
+     'geo_secondary' => { 'type' => 'physical' }
+   }
+
+   patroni['use_pg_rewind'] = true
+   patroni['postgresql']['max_wal_senders'] = 8 # Use double of the amount of patroni/reserved slots (3 patronis + 1 reserved slot for a Geo secondary).
+   patroni['postgresql']['max_replication_slots'] = 8 # Use double of the amount of patroni/reserved slots (3 patronis + 1 reserved slot for a Geo secondary).
+
+   postgresql['md5_auth_cidr_addresses'] = [
+     'PATRONI_PRIMARY1_IP/32', 'PATRONI_PRIMARY2_IP/32', 'PATRONI_PRIMARY3_IP/32', 'PATRONI_PRIMARY_PGBOUNCER/32',
+     'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32' # We list all secondary instances as they can all become a Standby Leader
+   ]
+
+   postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH'
+   postgresql['sql_replication_password'] = 'POSTGRESQL_REPLICATION_PASSWORD_HASH'
+   postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH'
+   ```
+
+1. Reconfigure GitLab for the changes to take effect:
+
+   ```shell
+   gitlab-ctl reconfigure
+   ```
+
+### Step 2. Configure a Standby cluster on the secondary site
+
+NOTE:
+If you are converting a secondary site to a Patroni Cluster, you must start
+on the PostgreSQL instance. It will become the Patroni Standby Leader instance,
+and then you can switchover to another replica if you need.
+
+For each Patroni instance on the secondary site:
+
+1. SSH into your Patroni node and login as root:
+
+   ```shell
+   sudo -i
+   ```
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following:
+
+   ```ruby
+   roles ['consul_role', 'postgres_role']
+
+   consul['enable'] = true
+   consul['configuration'] = {
+     retry_join: %w[CONSUL_SECONDARY1_IP CONSULT_SECONDARY2_IP CONSULT_SECONDARY3_IP]
+   }
+
+   repmgr['enable'] = false
+
+   postgresql['md5_auth_cidr_addresses'] = [
+     'PATRONI_SECONDARY1_IP/32', 'PATRONI_SECONDARY2_IP/32', 'PATRONI_SECONDARY3_IP/32', 'PATRONI_SECONDARY_PGBOUNCER/32',
+     # Any other instance that needs access to the database as per documentation
+   ]
+
+   patroni['enable'] = false
+   patroni['standby_cluster']['enable'] = true
+   patroni['standby_cluster']['host'] = 'PATRONI_PRIMARY_LEADER_IP' # This needs to be changed anytime the primary Leader changes
+   patroni['standby_cluster']['port'] = 5432
+   patroni['standby_cluster']['primary_slot_name'] = 'geo_secondary' # Or the unique replication slot name you setup before
+   patroni['replication_password'] = 'PLAIN_TEXT_POSTGRESQL_REPLICATION_PASSWORD'
+   patroni['use_pg_rewind'] = true
+   patroni['postgresql']['max_wal_senders'] = 5 # A minimum of three for one replica, plus two for each additional replica
+   patroni['postgresql']['max_replication_slots'] = 5 # A minimum of three for one replica, plus two for each additional replica
+   ```
+
+1. Reconfigure GitLab for the changes to take effect.
+   This is required to bootstrap PostgreSQL users and settings:
+
+   ```shell
+   gitlab-ctl reconfigure
+   ```
+
+1. Remove the PostgreSQL data directory:
+
+   WARNING:
+   If you are converting a secondary site to a Patroni Cluster, you must skip
+   this step on the PostgreSQL instance.
+
+   ```shell
+   rm -rf /var/opt/gitlab/postgresql/data
+   ```
+
+1. Edit `/etc/gitlab/gitlab.rb` to enable Patroni:
+
+   ```ruby
+   patroni['enable'] = true
+   ```
+
+1. Reconfigure GitLab for the changes to take effect:
+
+   ```shell
+   gitlab-ctl reconfigure
+   ```
 
 ## Migrating from repmgr to Patroni