Merge branch 'docs-make-patroni-recommended' into 'master'

Move the pg replication/failover content to make patroni recommended See merge request gitlab-org/gitlab!50101

Merge branch 'docs-make-patroni-recommended' into 'master'
Move the pg replication/failover content to make patroni recommended See merge request gitlab-org/gitlab!50101
c3431964 · Achilleas Pipinellis · 08830d77 · ba55cc76 · c3431964 · c3431964
Commit c3431964 authored Dec 17, 2020 by Achilleas Pipinellis
2 changed files
--- a/changelogs/unreleased/docs-make-patroni-recommended.yml
+++ b/changelogs/unreleased/docs-make-patroni-recommended.yml
+---
+title: Use Patroni as the default in the replication docs
+merge_request: 50101
+author:
+type: changed
--- a/doc/administration/postgresql/replication_and_failover.md
+++ b/doc/administration/postgresql/replication_and_failover.md
@@ -46,22 +46,19 @@ Each database node runs three services:
 `PostgreSQL` - The database itself.
-`repmgrd` - Communicates with other repmgrd services in the cluster and handles
+`Patroni` - Communicates with other patroni services in the cluster and handles
-failover when issues with the master server occurs. The failover procedure
+failover when issues with the leader server occurs. The failover procedure
 consists of:
- Selecting a new master for the cluster.
+- Selecting a new leader for the cluster.
- Promoting the new node to master.
+- Promoting the new node to leader.
- Instructing remaining servers to follow the new master node.
+- Instructing remaining servers to follow the new leader node.
- The old master node is automatically evicted from the cluster and should be
-  rejoined manually once recovered.
-`Consul` agent - Monitors the status of each node in the database cluster and
+`Consul` agent - To communicate with Consul cluster which stores the current Patroni state. The agent monitors the status of each node in the database cluster and tracks its health in a service definition on the Consul cluster.
-tracks its health in a service definition on the Consul cluster.
 ### Consul server node
-The Consul server node runs the Consul server service.
+The Consul server node runs the Consul server service. These nodes must have reached the quorum and elected a leader _before_ Patroni cluster bootstrap otherwise database nodes will wait until such Consul leader is elected.
 ### PgBouncer node
@@ -80,7 +77,7 @@ Each service in the package comes with a set of [default ports](https://docs.git
 - Application servers connect to either PgBouncer directly via its [default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#pgbouncer) or via a configured Internal Load Balancer (TCP) that serves multiple PgBouncers.
 - PgBouncer connects to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql)
- Repmgr connects to the database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql)
+- Patroni actively manages the running PostgreSQL processes and configuration.
 - PostgreSQL secondaries connect to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql)
 - Consul servers and agents connect to each others [Consul default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#consul)
@@ -141,7 +138,7 @@ available database connections.
 In this document we are assuming 3 database nodes, which makes this configuration:
 ```ruby
-postgresql['max_wal_senders'] = 4
+patroni['postgresql']['max_wal_senders'] = 4
 ```
 As previously mentioned, you'll have to prepare the network subnets that will
@@ -186,18 +183,6 @@ Few notes on the service itself:
  - `/etc/gitlab/gitlab.rb`: hashed, and in plain text
  - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed
-#### Repmgr information
-When using default setup, you will only have to prepare the network subnets that will
-be allowed to authenticate with the service.
-Few notes on the service itself:
- The service runs under the same system account as the database
-  - In the package, this is by default `gitlab-psql`
- The service will have a superuser database user account generated for it
-  - This defaults to `gitlab_repmgr`
 ### Installing Omnibus GitLab
 First, make sure to [download/install](https://about.gitlab.com/install/)
@@ -212,72 +197,80 @@ When installing the GitLab package, do not supply `EXTERNAL_URL` value.
 1. Make sure to [configure the Consul nodes](../consul.md).
 1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information), [`POSTGRESQL_PASSWORD_HASH`](#postgresql-information), the [number of db nodes](#postgresql-information), and the [network address](#network-information) before executing the next step.
-1. On the master database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section:
+#### Configuring Patroni cluster
-   ```ruby
+You must enable Patroni explicitly to be able to use it (with `patroni['enable'] = true`). When Patroni is enabled
-   # Disable all components except PostgreSQL and Repmgr and Consul
+repmgr will be disabled automatically.
-   roles ['postgres_role']
-   # PostgreSQL configuration
+Any PostgreSQL configuration item that controls replication, for example `wal_level`, `max_wal_senders`, etc, are strictly
-   postgresql['listen_address'] = '0.0.0.0'
+controlled by Patroni and will override the original settings that you make with the `postgresql[...]` configuration key.
-   postgresql['hot_standby'] = 'on'
+Hence, they are all separated and placed under `patroni['postgresql'][...]`. This behavior is limited to replication.
-   postgresql['wal_level'] = 'replica'
+Patroni honours any other PostgreSQL configuration that was made with the `postgresql[...]` configuration key. For example,
-   postgresql['shared_preload_libraries'] = 'repmgr_funcs'
+`max_wal_senders` by default is set to `5`. If you wish to change this you must set it with the `patroni['postgresql']['max_wal_senders']`
+configuration key.
-   # Disable automatic database migrations
+NOTE:
-   gitlab_rails['auto_migrate'] = false
+The configuration of a Patroni node is very similar to a repmgr but shorter. When Patroni is enabled, first you can ignore
+any replication setting of PostgreSQL (it will be overwritten anyway). Then you can remove any `repmgr[...]` or
+repmgr-specific configuration as well. Especially, make sure that you remove `postgresql['shared_preload_libraries'] = 'repmgr_funcs'`.
-   # Configure the Consul agent
+Here is an example similar to [the one that was done with repmgr](#configuring-repmgr-nodes):
-   consul['services'] = %w(postgresql)
-   # START user configuration
+```ruby
-   # Please set the real values as explained in Required Information section
+# Disable all components except PostgreSQL, Patroni (or Repmgr), and Consul
-   #
+roles['postgres_role']
-   # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value
-   postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH'
-   # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value
-   postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH'
-   # Replace X with value of number of db nodes + 1
-   postgresql['max_wal_senders'] = X
-   postgresql['max_replication_slots'] = X
-   # Replace XXX.XXX.XXX.XXX/YY with Network Address
+# Enable Patroni (which automatically disables Repmgr).
-   postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY)
+patroni['enable'] = true
-   repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY)
-   # Replace placeholders:
+# PostgreSQL configuration
-   #
+postgresql['listen_address'] = '0.0.0.0'
-   # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
-   # with the addresses gathered for CONSUL_SERVER_NODES
-   consul['configuration'] = {
-     retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z)
-   }
-   #
-   # END user configuration
-   ```
-   > `postgres_role` was introduced with GitLab 10.3
+# Disable automatic database migrations
+gitlab_rails['auto_migrate'] = false
-1. On secondary nodes, add all the configuration specified above for primary node
+# Configure the Consul agent
-   to `/etc/gitlab/gitlab.rb`. In addition, append the following configuration
+consul['services'] = %w(postgresql)
-   to inform `gitlab-ctl` that they are standby nodes initially and it need not
-   attempt to register them as primary node
-   ```ruby
+# START user configuration
-   # Specify if a node should attempt to be master on initialization
+# Please set the real values as explained in Required Information section
-   repmgr['master_on_initialization'] = false
+#
-   ```
+# Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value
+postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH'
+# Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value
+postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH'
-1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+# Replace X with value of number of db nodes + 1 (OPTIONAL the default value is 5)
-1. [Enable Monitoring](#enable-monitoring)
+patroni['postgresql']['max_wal_senders'] = X
+patroni['postgresql']['max_replication_slots'] = X
-> Please note:
+# Replace XXX.XXX.XXX.XXX/YY with Network Address
->
+postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY)
-> - If you want your database to listen on a specific interface, change the configuration:
->   `postgresql['listen_address'] = '0.0.0.0'`.
+# Replace placeholders:
-> - If your PgBouncer service runs under a different user account,
+#
->   you also need to specify: `postgresql['pgbouncer_user'] = PGBOUNCER_USERNAME` in
+# Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
->   your configuration.
+# with the addresses gathered for CONSUL_SERVER_NODES
+consul['configuration'] = {
+  retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z)
+}
+#
+# END user configuration
+```
+You do not need an additional or different configuration for replica nodes. As a matter of fact, you don't have to have
+a predetermined primary node. Therefore all database nodes use the same configuration.
+Once the configuration of a node is done, you must [reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure)
+on each node for the changes to take effect.
+Generally, when Consul cluster is ready, the first node that [reconfigures](../restart_gitlab.md#omnibus-gitlab-reconfigure)
+becomes the leader. You do not need to sequence the nodes reconfiguration. You can run them in parallel or in any order.
+If you choose an arbitrary order you do not have any predetermined master.
+NOTE:
+As opposed to repmgr, once the nodes are reconfigured you do not need any further action or additional command to join
+the replicas.
 #### Enable Monitoring
@@ -298,129 +291,6 @@ If you enable Monitoring, it must be enabled on **all** database servers.
 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
-#### Database nodes post-configuration
-##### Primary node
-Select one node as a primary node.
-1. Open a database prompt:
-   ```shell
-   gitlab-psql -d gitlabhq_production
-   ```
-1. Enable the `pg_trgm` extension:
-   ```shell
-   CREATE EXTENSION pg_trgm;
-   ```
-1. Enable the `btree_gist` extension:
-   ```shell
-   CREATE EXTENSION btree_gist;
-   ```
-1. Exit the database prompt by typing `\q` and Enter.
-1. Verify the cluster is initialized with one node:
-   ```shell
-   gitlab-ctl repmgr cluster show
-   ```
-   The output should be similar to the following:
-   ```plaintext
-   Role      | Name     | Upstream | Connection String
-   ----------+----------|----------|----------------------------------------
-   * master  | HOSTNAME |          | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr
-   ```
-1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will
-   refer to the hostname in the next section as `MASTER_NODE_NAME`. If the value
-   is not an IP address, it will need to be a resolvable name (via DNS or
-   `/etc/hosts`)
-##### Secondary nodes
-1. Set up the repmgr standby:
-   ```shell
-   gitlab-ctl repmgr standby setup MASTER_NODE_NAME
-   ```
-   Do note that this will remove the existing data on the node. The command
-   has a wait time.
-   The output should be similar to the following:
-   ```console
-   # gitlab-ctl repmgr standby setup MASTER_NODE_NAME
-   Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data
-   If this is not what you want, hit Ctrl-C now to exit
-   To skip waiting, rerun with the -w option
-   Sleeping for 30 seconds
-   Stopping the database
-   Removing the data
-   Cloning the data
-   Starting the database
-   Registering the node with the cluster
-   ok: run: repmgrd: (pid 19068) 0s
-   ```
-1. Verify the node now appears in the cluster:
-   ```shell
-   gitlab-ctl repmgr cluster show
-   ```
-   The output should be similar to the following:
-   ```plaintext
-   Role      | Name    | Upstream  | Connection String
-   ----------+---------|-----------|------------------------------------------------
-   * master  | MASTER  |           | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr
-     standby | STANDBY | MASTER    | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr
-   ```
-Repeat the above steps on all secondary nodes.
-#### Database checkpoint
-Before moving on, make sure the databases are configured correctly. Run the
-following command on the **primary** node to verify that replication is working
-properly:
-```shell
-gitlab-ctl repmgr cluster show
-```
-The output should be similar to:
-```plaintext
-Role      | Name         | Upstream     | Connection String
----------+--------------|--------------|--------------------------------------------------------------------
-* master  | MASTER  |        | host=MASTER port=5432 user=gitlab_repmgr dbname=gitlab_repmgr
-  standby | STANDBY | MASTER | host=STANDBY port=5432 user=gitlab_repmgr dbname=gitlab_repmgr
-```
-If the 'Role' column for any node says "FAILED", check the
-[Troubleshooting section](#troubleshooting) before proceeding.
-Also, check that the check master command works successfully on each node:
-```shell
-su - gitlab-consul
-gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node'
-```
-This command relies on exit codes to tell Consul whether a particular node is a master
-or secondary. The most important thing here is that this command does not produce errors.
-If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions.
-Check the [Troubleshooting section](#troubleshooting) before proceeding.
 ### Configuring the PgBouncer node
 1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`CONSUL_PASSWORD_HASH`](#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information) before executing the next step.
@@ -605,9 +475,9 @@ Here is a list and description of each machine and the assigned IP:
 - `10.6.0.21`: PgBouncer 1
 - `10.6.0.22`: PgBouncer 2
 - `10.6.0.23`: PgBouncer 3
- `10.6.0.31`: PostgreSQL master
+- `10.6.0.31`: PostgreSQL 1
- `10.6.0.32`: PostgreSQL secondary
+- `10.6.0.32`: PostgreSQL 2
- `10.6.0.33`: PostgreSQL secondary
+- `10.6.0.33`: PostgreSQL 3
 - `10.6.0.41`: GitLab application
 All passwords are set to `toomanysecrets`, please do not use this password or derived hashes and the `external_url` for GitLab is `http://gitlab.example.com`.
@@ -667,29 +537,28 @@ An internal load balancer (TCP) is then required to be setup to serve each PgBou
 #### Example recommended setup for PostgreSQL servers
-##### Primary node
+On database nodes edit `/etc/gitlab/gitlab.rb`:
-On primary node edit `/etc/gitlab/gitlab.rb`:
 ```ruby
-# Disable all components except PostgreSQL and Repmgr and Consul
+# Disable all components except PostgreSQL, Patroni (or Repmgr), and Consul
 roles ['postgres_role']
 # PostgreSQL configuration
 postgresql['listen_address'] = '0.0.0.0'
 postgresql['hot_standby'] = 'on'
 postgresql['wal_level'] = 'replica'
-postgresql['shared_preload_libraries'] = 'repmgr_funcs'
+# Enable Patroni (which automatically disables Repmgr).
+patroni['enable'] = true
 # Disable automatic database migrations
 gitlab_rails['auto_migrate'] = false
 postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c'
 postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f'
-postgresql['max_wal_senders'] = 4
+patroni['postgresql']['max_wal_senders'] = 4
 postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16)
-repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16)
 # Configure the Consul agent
 consul['services'] = %w(postgresql)
@@ -702,61 +571,19 @@ consul['monitoring_service_discovery'] =  true
 [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-##### Secondary nodes
+#### Example recommended setup manual steps
-On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the configuration
+After deploying the configuration follow these steps:
-added to primary node, noted above. In addition, append the following
-configuration:
-```ruby
+1. Find the primary database node:
-# Specify if a node should attempt to be master on initialization
-repmgr['master_on_initialization'] = false
-```
-[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+   ```shell
+   gitlab-ctl get-postgresql-primary
+   ```
-###### Example recommended setup for application server
+1. On the primary database node:
-On the server edit `/etc/gitlab/gitlab.rb`:
+   Enable the `pg_trgm` and `btree_gist` extensions:
-```ruby
-external_url 'http://gitlab.example.com'
-gitlab_rails['db_host'] = '10.6.0.20' # Internal Load Balancer for PgBouncer nodes
-gitlab_rails['db_port'] = 6432
-gitlab_rails['db_password'] = 'toomanysecrets'
-gitlab_rails['auto_migrate'] = false
-postgresql['enable'] = false
-pgbouncer['enable'] = false
-consul['enable'] = true
-# Configure Consul agent
-consul['watchers'] = %w(postgresql)
-pgbouncer['users'] = {
-  'gitlab-consul': {
-    password: '5e0e3263571e3704ad655076301d6ebe'
-  },
-  'pgbouncer': {
-    password: '771a8625958a529132abe6f1a4acb19c'
-  }
-}
-consul['configuration'] = {
-  retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13)
-}
-```
-[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-#### Example recommended setup manual steps
-After deploying the configuration follow these steps:
-1. On `10.6.0.31`, our primary database:
-   Enable the `pg_trgm` and `btree_gist` extensions:
   ```shell
   gitlab-psql -d gitlabhq_production
@@ -767,22 +594,6 @@ After deploying the configuration follow these steps:
   CREATE EXTENSION btree_gist;
   ```
-1. On `10.6.0.32`, our first standby database:
-   Make this node a standby of the primary:
-   ```shell
-   gitlab-ctl repmgr standby setup 10.6.0.21
-   ```
-1. On `10.6.0.33`, our second standby database:
-   Make this node a standby of the primary:
-   ```shell
-   gitlab-ctl repmgr standby setup 10.6.0.21
-   ```
 1. On `10.6.0.41`, our application server:
   Set `gitlab-consul` user's PgBouncer password to `toomanysecrets`:
@@ -802,15 +613,15 @@ After deploying the configuration follow these steps:
 This example uses 3 PostgreSQL servers, and 1 application node (with PgBouncer setup alongside).
 It differs from the [recommended setup](#example-recommended-setup) by moving the Consul servers into the same servers we use for PostgreSQL.
-The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with PostgreSQL [failover](#failover-procedure) and [restore](#restore-procedure) procedures in addition to [Consul outage recovery](../consul.md#outage-recovery) on the same set of machines.
+The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with PostgreSQL [failover](#manual-failover-procedure-for-patroni) procedures in addition to [Consul outage recovery](../consul.md#outage-recovery) on the same set of machines.
 In this example we start with all servers on the same 10.6.0.0/16 private network range, they can connect to each freely other on those addresses.
 Here is a list and description of each machine and the assigned IP:
- `10.6.0.21`: PostgreSQL master
+- `10.6.0.21`: PostgreSQL 1
- `10.6.0.22`: PostgreSQL secondary
+- `10.6.0.22`: PostgreSQL 2
- `10.6.0.23`: PostgreSQL secondary
+- `10.6.0.23`: PostgreSQL 3
 - `10.6.0.31`: GitLab application
 All passwords are set to `toomanysecrets`, please do not use this password or derived hashes.
@@ -821,9 +632,7 @@ Please note that after the initial configuration, if a failover occurs, the Post
 #### Example minimal configuration for database servers
-##### Primary node
+On database nodes edit `/etc/gitlab/gitlab.rb`:
-On primary database node edit `/etc/gitlab/gitlab.rb`:
 ```ruby
 # Disable all components except PostgreSQL, Repmgr, and Consul
@@ -833,7 +642,9 @@ roles ['postgres_role']
 postgresql['listen_address'] = '0.0.0.0'
 postgresql['hot_standby'] = 'on'
 postgresql['wal_level'] = 'replica'
-postgresql['shared_preload_libraries'] = 'repmgr_funcs'
+# Enable Patroni (which automatically disables Repmgr).
+patroni['enable'] = true
 # Disable automatic database migrations
 gitlab_rails['auto_migrate'] = false
@@ -843,10 +654,9 @@ consul['services'] = %w(postgresql)
 postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c'
 postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f'
-postgresql['max_wal_senders'] = 4
+patroni['postgresql']['max_wal_senders'] = 4
 postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16)
-repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16)
 consul['configuration'] = {
  server: true,
@@ -856,16 +666,6 @@ consul['configuration'] = {
 [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
-##### Secondary nodes
-On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the information added
-to primary node, noted above. In addition, append the following configuration
-```ruby
-# Specify if a node should attempt to be master on initialization
-repmgr['master_on_initialization'] = false
-```
 #### Example minimal configuration for application server
 On the server edit `/etc/gitlab/gitlab.rb`:
@@ -908,555 +708,678 @@ consul['configuration'] = {
 The manual steps for this configuration are the same as for the [example recommended setup](#example-recommended-setup-manual-steps).
-### Failover procedure
+### Manual failover procedure for Patroni
-By default, if the master database fails, `repmgrd` should promote one of the
+While Patroni supports automatic failover, you also have the ability to perform
-standby nodes to master automatically, and Consul will update PgBouncer with
+a manual one, where you have two slightly different options:
-the new master.
-If you need to failover manually, you have two options:
+- **Failover**: allows you to perform a manual failover when there are no healthy nodes.
+  You can perform this action in any PostgreSQL node:
-**Shutdown the current master database**
+  ```shell
+  sudo gitlab-ctl patroni failover
+  ```
-Run:
+- **Switchover**: only works when the cluster is healthy and allows you to schedule a switchover (it can happen immediately).
+  You can perform this action in any PostgreSQL node:
-```shell
+  ```shell
-gitlab-ctl stop postgresql
+  sudo gitlab-ctl patroni switchover
-```
+  ```
-The automated failover process will see this and failover to one of the
+For further details on this subject, see the
-standby nodes.
+[Patroni documentation](https://patroni.readthedocs.io/en/latest/rest_api.html#switchover-and-failover-endpoints).
-**Or perform a manual failover**
+## Patroni
-1. Ensure the old master node is not still active.
+NOTE:
-1. Login to the server that should become the new master and run:
+Using Patroni instead of Repmgr is supported for PostgreSQL 11 and required for PostgreSQL 12.
-   ```shell
+Patroni is an opinionated solution for PostgreSQL high-availability. It takes the control of PostgreSQL, overrides its
-   gitlab-ctl repmgr standby promote
+configuration and manages its lifecycle (start, stop, restart). This is a more active approach when compared to repmgr.
-   ```
+Both repmgr and Patroni are both supported and available. But Patroni will be the default (and perhaps the only) option
+for PostgreSQL 12 clustering and cascading replication for Geo deployments.
-1. If there are any other standby servers in the cluster, have them follow
+The [architecture](#example-recommended-setup-manual-steps) (that was mentioned above) does not change for Patroni.
-   the new master server:
+You do not need any special consideration for Patroni while provisioning your database nodes. Patroni heavily relies on
+Consul to store the state of the cluster and elect a leader. Any failure in Consul cluster and its leader election will
+propagate to Patroni cluster as well.
-   ```shell
+Similar to repmgr, Patroni monitors the cluster and handles failover. When the primary node fails it works with Consul
-   gitlab-ctl repmgr standby follow NEW_MASTER
+to notify PgBouncer. However, as opposed to repmgr, on failure, Patroni handles the transitioning of the old primary to
-   ```
+a replica and rejoins it to the cluster automatically. So you do not need any manual operation for recovering the
+cluster as you do with repmgr.
-#### Geo secondary site considerations
+With Patroni the connection flow is slightly different. Patroni on each node connects to Consul agent to join the
+cluster. Only after this point it decides if the node is the primary or a replica. Based on this decision, it configures
+and starts PostgreSQL which it communicates with directly over a Unix socket. This implies that if Consul cluster is not
+functional or does not have a leader, Patroni and by extension PostgreSQL will not start. Patroni also exposes a REST
+API which can be accessed via its [default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#patroni)
+on each node.
-When a Geo secondary site is replicating from a primary site that uses `repmgr` and `PgBouncer`, [replicating through PgBouncer is not supported](https://github.com/pgbouncer/pgbouncer/issues/382#issuecomment-517911529) and the secondary must replicate directly from the leader node in the `repmgr` cluster. Therefore, when there is a failover in the `repmgr` cluster, you will need to manually re-point your secondary site to replicate from the new leader with:
+### Database authorization for Patroni
+Patroni uses Unix socket to manage PostgreSQL instance. Therefore, the connection from the `local` socket must be trusted.
+Also, replicas use the replication user (`gitlab_replicator` by default) to communicate with the leader. For this user,
+you can choose between `trust` and `md5` authentication. If you set `postgresql['sql_replication_password']`,
+Patroni will use `md5` authentication, otherwise it falls back to `trust`. You must to specify the cluster CIDR in
+`postgresql['md5_auth_cidr_addresses']` or `postgresql['trust_auth_cidr_addresses']` respectively.
+### Interacting with Patroni cluster
+You can use `gitlab-ctl patroni members` to check the status of the cluster members. To check the status of each node
+`gitlab-ctl patroni` provides two additional sub-commands, `check-leader` and `check-replica` which indicate if a node
+is the primary or a replica.
+When Patroni is enabled, you don't have direct control over `postgresql` service. Patroni will signal PostgreSQL's startup,
+shutdown, and restart. For example, for shutting down PostgreSQL on a node, you must shutdown Patroni on the same node
+with:
 ```shell
-sudo gitlab-ctl replicate-geo-database --host=<new_leader_ip> --replication-slot=<slot_name>
+sudo gitlab-ctl stop patroni
 ```
-Otherwise, the replication will not happen anymore, even if the original node gets re-added as a follower node. This will re-sync your secondary site database and may take a long time depending on the amount of data to sync.
+Note that stopping or restarting Patroni service on the leader node will trigger the automatic failover. If you
+want to signal Patroni to reload its configuration or restart PostgreSQL process without triggering the failover, you
+must use the `reload` or `restart` sub-commands of `gitlab-ctl patroni` instead. These two sub-commands are wrappers of
+the same `patronictl` commands.
-### Restore procedure
+### Recovering the Patroni cluster
-If a node fails, it can be removed from the cluster, or added back as a standby
+To recover the old primary and rejoin it to the cluster as a replica, you can simply start Patroni with:
-after it has been restored to service.
-#### Remove a standby from the cluster
+```shell
+sudo gitlab-ctl start patroni
+```
-  From any other node in the cluster, run:
+No further configuration or intervention is needed.
-  ```shell
+### Maintenance procedure for Patroni
-  gitlab-ctl repmgr standby unregister --node=X
-  ```
-  where X is the value of node in `repmgr.conf` on the old server.
+With Patroni enabled, you can run a planned maintenance. If you want to do some maintenance work on one node and you
+don't want Patroni to manage it, you can use put it into maintenance mode:
-  To find this, you can use:
+```shell
+sudo gitlab-ctl patroni pause
+```
-  ```shell
+When Patroni runs in a paused mode, it does not change the state of PostgreSQL. Once you are done you can resume Patroni:
-  awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf
-  ```
-  It will output something like:
+```shell
+sudo gitlab-ctl patroni resume
+```
-  ```plaintext
+For further details, see [Patroni documentation on this subject](https://patroni.readthedocs.io/en/latest/pause.html).
-  959789412
-  ```
-  Then you will use this ID to unregister the node:
+### Switching from repmgr to Patroni
-  ```shell
+WARNING:
-  gitlab-ctl repmgr standby unregister --node=959789412
+Although switching from repmgr to Patroni is fairly straightforward the other way around is not. Rolling back from
-  ```
+Patroni to repmgr can be complicated and may involve deletion of data directory. If you need to do that, please contact
+GitLab support.
-#### Add a node as a standby server
+You can switch an exiting database cluster to use Patroni instead of repmgr with the following steps:
-  From the standby node, run:
+1. Stop repmgr on all replica nodes and lastly with the primary node:
   ```shell
-  gitlab-ctl repmgr standby follow NEW_MASTER
+   sudo gitlab-ctl stop repmgrd
-  gitlab-ctl restart repmgrd
   ```
-  WARNING:
+1. Stop PostgreSQL on all replica nodes:
-  When the server is brought back online, and before
-  you switch it to a standby node, repmgr will report that there are two masters.
-  If there are any clients that are still attempting to write to the old master,
-  this will cause a split, and the old master will need to be resynced from
-  scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`.
-#### Add a failed master back into the cluster as a standby node
-  Once `repmgrd` and PostgreSQL are running, the node will need to follow the new
-  as a standby node.
   ```shell
-  gitlab-ctl repmgr standby follow NEW_MASTER
+   sudo gitlab-ctl stop postgresql
   ```
-  Once the node is following the new master as a standby, the node needs to be
+   NOTE:
-  [unregistered from the cluster on the new master node](#remove-a-standby-from-the-cluster).
+   Ensure that there is no `walsender` process running on the primary node.
+   `ps aux | grep walsender` must not show any running process.
-  Once the old master node has been unregistered from the cluster, it will need
+1. On the primary node, [configure Patroni](#configuring-patroni-cluster). Remove `repmgr` and any other
-  to be setup as a new standby:
+   repmgr-specific configuration. Also remove any configuration that is related to PostgreSQL replication.
+1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on the primary node. It will become
+   the leader. You can check this with:
   ```shell
-  gitlab-ctl repmgr standby setup NEW_MASTER
+   sudo gitlab-ctl tail patroni
   ```
-  Failure to unregister and read the old master node can lead to subsequent failovers
+1. Repeat the last two steps for all replica nodes. `gitlab.rb` should look the same on all nodes.
-  not working.
+1. Optional: You can remove `gitlab_repmgr` database and role on the primary.
-### Alternate configurations
+### Upgrading PostgreSQL major version in a Patroni cluster
-#### Database authorization
+As of GitLab 13.3, PostgreSQL 11.7 and 12.3 are both shipped with Omnibus GitLab, and as of GitLab 13.7
+PostgreSQL 12 is used by default. If you want to upgrade to PostgreSQL 12 in versions prior to GitLab 13.7,
+you must ask for it explicitly.
-By default, we give any host on the database network the permission to perform
+WARNING:
-repmgr operations using PostgreSQL's `trust` method. If you do not want this
+The procedure for upgrading PostgreSQL in a Patroni cluster is different than when upgrading using repmgr.
-level of trust, there are alternatives.
+The following outlines the key differences and important considerations that need to be accounted for when
+upgrading PostgreSQL.
-You can trust only the specific nodes that will be database clusters, or you
+Here are a few key facts that you must consider before upgrading PostgreSQL:
-can require md5 authentication.
-#### Trust specific addresses
+- The main point is that you will have to **shut down the Patroni cluster**. This means that your
+  GitLab deployment will be down for the duration of database upgrade or, at least, as long as your leader
+  node is upgraded. This can be **a significant downtime depending on the size of your database**.
-If you know the IP address, or FQDN of all database and PgBouncer nodes in the
+- Upgrading PostgreSQL creates a new data directory with a new control data. From Patroni's perspective
-cluster, you can trust only those nodes.
+  this is a new cluster that needs to be bootstrapped again. Therefore, as part of the upgrade procedure,
+  the cluster state, which is stored in Consul, will be wiped out. Once the upgrade is completed, Patroni
+  will be instructed to bootstrap a new cluster. **Note that this will change your _cluster ID_**.
-In `/etc/gitlab/gitlab.rb` on all of the database nodes, set
+- The procedures for upgrading leader and replicas are not the same. That is why it is important to use the
-`repmgr['trust_auth_cidr_addresses']` to an array of strings containing all of
+  right procedure on each node.
-the addresses.
-If setting to a node's FQDN, they must have a corresponding PTR record in DNS.
+- Upgrading a replica node **deletes the data directory and resynchronizes it** from the leader using the
-If setting to a node's IP address, specify it as `XXX.XXX.XXX.XXX/32`.
+  configured replication method (currently `pg_basebackup` is the only available option). It might take some
+  time for replica to catch up with the leader, depending on the size of your database.
-For example:
+- An overview of the upgrade procedure is outlined in [Patoni's documentation](https://patroni.readthedocs.io/en/latest/existing_data.html#major-upgrade-of-postgresql-version).
+  You can still use `gitlab-ctl pg-upgrade` which implements this procedure with a few adjustments.
-```ruby
+Considering these, you should carefully plan your PostgreSQL upgrade:
-repmgr['trust_auth_cidr_addresses'] = %w(192.168.1.44/32 db2.example.com)
-```
-#### MD5 Authentication
+1. Find out which node is the leader and which node is a replica:
-If you are running on an untrusted network, repmgr can use md5 authentication
+   ```shell
-with a [`.pgpass` file](https://www.postgresql.org/docs/11/libpq-pgpass.html)
+   gitlab-ctl patroni members
-to authenticate.
+   ```
-You can specify by IP address, FQDN, or by subnet, using the same format as in
+   NOTE:
-the previous section:
+   `gitlab-ctl pg-upgrade` tries to detect the role of the node. If for any reason the auto-detection
+   does not work or you believe it did not detect the role correctly, you can use the `--leader` or `--replica`
+   arguments to manually override it.
-1. On the current master node, create a password for the `gitlab` and
+1. Stop Patroni **only on replicas**.
-   `gitlab_repmgr` user:
   ```shell
-   gitlab-psql -d template1
+   sudo gitlab-ctl stop patroni
-   template1=# \password gitlab_repmgr
-   Enter password: ****
-   Confirm password: ****
-   template1=# \password gitlab
   ```
-1. On each database node:
+1. Enable the maintenance mode on the **application node**:
-   1. Edit `/etc/gitlab/gitlab.rb`:
+   ```shell
-      1. Ensure `repmgr['trust_auth_cidr_addresses']` is **not** set
+   sudo gitlab-ctl deploy-page up
-      1. Set `postgresql['md5_auth_cidr_addresses']` to the desired value
+   ```
-      1. Set `postgresql['sql_replication_user'] = 'gitlab_repmgr'`
-      1. Reconfigure with `gitlab-ctl reconfigure`
-      1. Restart PostgreSQL with `gitlab-ctl restart postgresql`
-   1. Create a `.pgpass` file. Enter the `gitlab_repmgr` password twice to
+1. Upgrade PostgreSQL on **the leader node** and make sure that the upgrade is completed successfully:
-      when asked:
   ```shell
-      gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*'
+   sudo gitlab-ctl pg-upgrade -V 12
   ```
-1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`:
+1. Check the status of the leader and cluster. You can only proceed if you have a healthy leader:
-   1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for
-      the `gitlab` database user
-   1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect
-## Troubleshooting
+   ```shell
+   gitlab-ctl patroni check-leader
-### Consul and PostgreSQL changes not taking effect
+   # OR
-Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading.
+   gitlab-ctl patroni members
+   ```
-To restart either service, run `gitlab-ctl restart SERVICE`
+1. You can now disable the maintenance mode on the **application node**:
-For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. To be safe, you can stop `repmgrd` on the standby nodes first with `gitlab-ctl stop repmgrd`, then start afterwards with `gitlab-ctl start repmgrd`.
+   ```shell
+   sudo gitlab-ctl deploy-page down
+   ```
-On the Consul server nodes, it is important to [restart the Consul service](../consul.md#restart-consul) in a controlled manner.
+1. Upgrade PostgreSQL **on replicas** (you can do this in parallel on all of them):
-### `gitlab-ctl repmgr-check-master` command produces errors
+   ```shell
+   sudo gitlab-ctl pg-upgrade -V 12
+   ```
-If this command displays errors about database permissions it is likely that something failed during
+NOTE:
-install, resulting in the `gitlab-consul` database user getting incorrect permissions. Follow these
+Reverting PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the same considerations as
-steps to fix the problem:
+`gitlab-ctl pg-upgrade`. You should follow the same procedure by first stopping the replicas,
+then reverting the leader, and finally reverting the replicas.
-1. On the master database node, connect to the database prompt - `gitlab-psql -d template1`
+## Repmgr
-1. Delete the `gitlab-consul` user - `DROP USER "gitlab-consul";`
-1. Exit the database prompt - `\q`
-1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) and the user will be re-added with the proper permissions.
-1. Change to the `gitlab-consul` user - `su - gitlab-consul`
-1. Try the check command again - `gitlab-ctl repmgr-check-master`.
-Now there should not be errors. If errors still occur then there is another problem.
+NOTE:
+Using Patroni instead of Repmgr is supported for PostgreSQL 11 and required for PostgreSQL 12.
-### PgBouncer error `ERROR: pgbouncer cannot connect to server`
+### Configuring Repmgr Nodes
-You may get this error when running `gitlab-rake gitlab:db:configure` or you
+1. On the master database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section:
-may see the error in the PgBouncer log file.
-```plaintext
+   ```ruby
-PG::ConnectionBad: ERROR:  pgbouncer cannot connect to server
+   # Disable all components except PostgreSQL and Repmgr and Consul
-```
+   roles ['postgres_role']
-The problem may be that your PgBouncer node's IP address is not included in the
+   # PostgreSQL configuration
-`trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes.
+   postgresql['listen_address'] = '0.0.0.0'
+   postgresql['hot_standby'] = 'on'
+   postgresql['wal_level'] = 'replica'
+   postgresql['shared_preload_libraries'] = 'repmgr_funcs'
-You can confirm that this is the issue by checking the PostgreSQL log on the master
+   # Disable automatic database migrations
-database node. If you see the following error then `trust_auth_cidr_addresses`
+   gitlab_rails['auto_migrate'] = false
-is the problem.
-```plaintext
+   # Configure the Consul agent
-2018-03-29_13:59:12.11776 FATAL:  no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off
+   consul['services'] = %w(postgresql)
-```
-To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`.
+   # START user configuration
+   # Please set the real values as explained in Required Information section
+   #
+   # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value
+   postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH'
+   # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value
+   postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH'
+   # Replace X with value of number of db nodes + 1
+   postgresql['max_wal_senders'] = X
+   postgresql['max_replication_slots'] = X
-```ruby
+   # Replace XXX.XXX.XXX.XXX/YY with Network Address
-postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>)
+   postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY)
-```
+   repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY)
-[Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+   # Replace placeholders:
+   #
+   # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
+   # with the addresses gathered for CONSUL_SERVER_NODES
+   consul['configuration'] = {
+     retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z)
+   }
+   #
+   # END user configuration
+   ```
-### Issues with other components
+   > `postgres_role` was introduced with GitLab 10.3
-If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page:
+1. On secondary nodes, add all the configuration specified above for primary node
+   to `/etc/gitlab/gitlab.rb`. In addition, append the following configuration
+   to inform `gitlab-ctl` that they are standby nodes initially and it need not
+   attempt to register them as primary node
- [Consul](../consul.md#troubleshooting-consul)
+   ```ruby
- [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting)
+   # Specify if a node should attempt to be master on initialization
+   repmgr['master_on_initialization'] = false
+   ```
-## Patroni
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+1. [Enable Monitoring](#enable-monitoring)
-NOTE:
+> Please note:
-Starting from GitLab 13.1, Patroni is available for **experimental** use to replace repmgr. Due to its
+>
-experimental nature, Patroni support is **subject to change without notice.**
+> - If you want your database to listen on a specific interface, change the configuration:
+>   `postgresql['listen_address'] = '0.0.0.0'`.
+> - If your PgBouncer service runs under a different user account,
+>   you also need to specify: `postgresql['pgbouncer_user'] = PGBOUNCER_USERNAME` in
+>   your configuration.
-Patroni is an opinionated solution for PostgreSQL high-availability. It takes the control of PostgreSQL, overrides its
+#### Database nodes post-configuration
-configuration and manages its lifecycle (start, stop, restart). This is a more active approach when compared to repmgr.
-Both repmgr and Patroni are both supported and available. But Patroni will be the default (and perhaps the only) option
-for PostgreSQL 12 clustering and cascading replication for Geo deployments.
-The [architecture](#example-recommended-setup-manual-steps) (that was mentioned above) does not change for Patroni.
+##### Primary node
-You do not need any special consideration for Patroni while provisioning your database nodes. Patroni heavily relies on
-Consul to store the state of the cluster and elect a leader. Any failure in Consul cluster and its leader election will
-propagate to Patroni cluster as well.
-Similar to repmgr, Patroni monitors the cluster and handles failover. When the primary node fails it works with Consul
+Select one node as a primary node.
-to notify PgBouncer. However, as opposed to repmgr, on failure, Patroni handles the transitioning of the old primary to
-a replica and rejoins it to the cluster automatically. So you do not need any manual operation for recovering the
-cluster as you do with repmgr.
-With Patroni the connection flow is slightly different. Patroni on each node connects to Consul agent to join the
+1. Open a database prompt:
-cluster. Only after this point it decides if the node is the primary or a replica. Based on this decision, it configures
-and starts PostgreSQL which it communicates with directly over a Unix socket. This implies that if Consul cluster is not
-functional or does not have a leader, Patroni and by extension PostgreSQL will not start. Patroni also exposes a REST
-API which can be accessed via its [default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#patroni)
-on each node.
-### Configuring Patroni cluster
+   ```shell
+   gitlab-psql -d gitlabhq_production
+   ```
-You must enable Patroni explicitly to be able to use it (with `patroni['enable'] = true`). When Patroni is enabled
+1. Enable the `pg_trgm` extension:
-repmgr will be disabled automatically.
-Any PostgreSQL configuration item that controls replication, for example `wal_level`, `max_wal_senders`, etc, are strictly
+   ```shell
-controlled by Patroni and will override the original settings that you make with the `postgresql[...]` configuration key.
+   CREATE EXTENSION pg_trgm;
-Hence, they are all separated and placed under `patroni['postgresql'][...]`. This behavior is limited to replication.
+   ```
-Patroni honours any other PostgreSQL configuration that was made with the `postgresql[...]` configuration key. For example,
-`max_wal_senders` by default is set to `5`. If you wish to change this you must set it with the `patroni['postgresql']['max_wal_senders']`
-configuration key.
-The configuration of Patroni node is very similar to a repmgr but shorter. When Patroni is enabled, first you can ignore
+1. Enable the `btree_gist` extension:
-any replication setting of PostgreSQL (it will be overwritten anyway). Then you can remove any `repmgr[...]` or
-repmgr-specific configuration as well. Especially, make sure that you remove `postgresql['shared_preload_libraries'] = 'repmgr_funcs'`.
-Here is an example similar to [the one that was done with repmgr](#configuring-the-database-nodes):
+   ```shell
+   CREATE EXTENSION btree_gist;
+   ```
-```ruby
+1. Exit the database prompt by typing `\q` and Enter.
-# Disable all components except PostgreSQL and Repmgr and Consul
-roles['postgres_role']
-# Enable Patroni
+1. Verify the cluster is initialized with one node:
-patroni['enable'] = true
-# PostgreSQL configuration
+   ```shell
-postgresql['listen_address'] = '0.0.0.0'
+   gitlab-ctl repmgr cluster show
+   ```
-# Disable automatic database migrations
+   The output should be similar to the following:
-gitlab_rails['auto_migrate'] = false
-# Configure the Consul agent
+   ```plaintext
-consul['services'] = %w(postgresql)
+   Role      | Name     | Upstream | Connection String
+   ----------+----------|----------|----------------------------------------
+   * master  | HOSTNAME |          | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr
+   ```
-# START user configuration
+1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will
-# Please set the real values as explained in Required Information section
+   refer to the hostname in the next section as `MASTER_NODE_NAME`. If the value
-#
+   is not an IP address, it will need to be a resolvable name (via DNS or
-# Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value
+   `/etc/hosts`)
-postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH'
-# Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value
-postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH'
-# Replace X with value of number of db nodes + 1 (OPTIONAL the default value is 5)
+##### Secondary nodes
-patroni['postgresql']['max_wal_senders'] = X
-patroni['postgresql']['max_replication_slots'] = X
-# Replace XXX.XXX.XXX.XXX/YY with Network Address
+1. Set up the repmgr standby:
-postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY)
-# Replace placeholders:
+   ```shell
-#
+   gitlab-ctl repmgr standby setup MASTER_NODE_NAME
-# Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
+   ```
-# with the addresses gathered for CONSUL_SERVER_NODES
-consul['configuration'] = {
-  retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z)
-}
-#
-# END user configuration
-```
-You do not need an additional or different configuration for replica nodes. As a matter of fact, you don't have to have
+   Do note that this will remove the existing data on the node. The command
-a predetermined primary node. Therefore all database nodes use the same configuration.
+   has a wait time.
-Once the configuration of a node is done, you must [reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure)
+   The output should be similar to the following:
-on each node for the changes to take effect.
-Generally, when Consul cluster is ready, the first node that [reconfigures](../restart_gitlab.md#omnibus-gitlab-reconfigure)
+   ```console
-becomes the leader. You do not need to sequence the nodes reconfiguration. You can run them in parallel or in any order.
+   # gitlab-ctl repmgr standby setup MASTER_NODE_NAME
-If you choose an arbitrary order you do not have any predetermined master.
+   Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data
+   If this is not what you want, hit Ctrl-C now to exit
+   To skip waiting, rerun with the -w option
+   Sleeping for 30 seconds
+   Stopping the database
+   Removing the data
+   Cloning the data
+   Starting the database
+   Registering the node with the cluster
+   ok: run: repmgrd: (pid 19068) 0s
+   ```
-As opposed to repmgr, once the nodes are reconfigured you do not need any further action or additional command to join
+1. Verify the node now appears in the cluster:
-the replicas.
-#### Database authorization for Patroni
+   ```shell
+   gitlab-ctl repmgr cluster show
+   ```
-Patroni uses Unix socket to manage PostgreSQL instance. Therefore, the connection from the `local` socket must be trusted.
+   The output should be similar to the following:
-Also, replicas use the replication user (`gitlab_replicator` by default) to communicate with the leader. For this user,
+   ```plaintext
-you can choose between `trust` and `md5` authentication. If you set `postgresql['sql_replication_password']`,
+   Role      | Name    | Upstream  | Connection String
-Patroni will use `md5` authentication, otherwise it falls back to `trust`. You must to specify the cluster CIDR in
+   ----------+---------|-----------|------------------------------------------------
-`postgresql['md5_auth_cidr_addresses']` or `postgresql['trust_auth_cidr_addresses']` respectively.
+   * master  | MASTER  |           | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr
+     standby | STANDBY | MASTER    | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr
+   ```
-### Interacting with Patroni cluster
+Repeat the above steps on all secondary nodes.
-You can use `gitlab-ctl patroni members` to check the status of the cluster members. To check the status of each node
+#### Database checkpoint
-`gitlab-ctl patroni` provides two additional sub-commands, `check-leader` and `check-replica` which indicate if a node
-is the primary or a replica.
-When Patroni is enabled, you don't have direct control over `postgresql` service. Patroni will signal PostgreSQL's startup,
+Before moving on, make sure the databases are configured correctly. Run the
-shutdown, and restart. For example, for shutting down PostgreSQL on a node, you must shutdown Patroni on the same node
+following command on the **primary** node to verify that replication is working
-with:
+properly:
 ```shell
-sudo gitlab-ctl stop patroni
+gitlab-ctl repmgr cluster show
 ```
-Note that stopping or restarting Patroni service on the leader node will trigger the automatic failover. If you
+The output should be similar to:
-want to signal Patroni to reload its configuration or restart PostgreSQL process without triggering the failover, you
-must use the `reload` or `restart` sub-commands of `gitlab-ctl patroni` instead. These two sub-commands are wrappers of
-the same `patronictl` commands.
-### Manual failover procedure for Patroni
+```plaintext
+Role      | Name         | Upstream     | Connection String
+----------+--------------|--------------|--------------------------------------------------------------------
+* master  | MASTER  |        | host=MASTER port=5432 user=gitlab_repmgr dbname=gitlab_repmgr
+  standby | STANDBY | MASTER | host=STANDBY port=5432 user=gitlab_repmgr dbname=gitlab_repmgr
+```
-While Patroni supports automatic failover, you also have the ability to perform
+If the 'Role' column for any node says "FAILED", check the
-a manual one, where you have two slightly different options:
+[Troubleshooting section](#troubleshooting) before proceeding.
+Also, check that the check master command works successfully on each node:
+```shell
+su - gitlab-consul
+gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node'
+```
+This command relies on exit codes to tell Consul whether a particular node is a master
+or secondary. The most important thing here is that this command does not produce errors.
+If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions.
+Check the [Troubleshooting section](#troubleshooting) before proceeding.
+### Repmgr failover procedure
+By default, if the master database fails, `repmgrd` should promote one of the
+standby nodes to master automatically, and Consul will update PgBouncer with
+the new master.
+If you need to failover manually, you have two options:
+**Shutdown the current master database**
+Run:
+```shell
+gitlab-ctl stop postgresql
+```
+The automated failover process will see this and failover to one of the
+standby nodes.
+**Or perform a manual failover**
- **Failover**: allows you to perform a manual failover when there are no healthy nodes.
+1. Ensure the old master node is not still active.
-  You can perform this action in any PostgreSQL node:
+1. Login to the server that should become the new master and run:
   ```shell
-  sudo gitlab-ctl patroni failover
+   gitlab-ctl repmgr standby promote
   ```
- **Switchover**: only works when the cluster is healthy and allows you to schedule a switchover (it can happen immediately).
+1. If there are any other standby servers in the cluster, have them follow
-  You can perform this action in any PostgreSQL node:
+   the new master server:
   ```shell
-  sudo gitlab-ctl patroni switchover
+   gitlab-ctl repmgr standby follow NEW_MASTER
   ```
-For further details on this subject, see the
+#### Geo secondary site considerations
-[Patroni documentation](https://patroni.readthedocs.io/en/latest/rest_api.html#switchover-and-failover-endpoints).
-### Recovering the Patroni cluster
-To recover the old primary and rejoin it to the cluster as a replica, you can simply start Patroni with:
+When a Geo secondary site is replicating from a primary site that uses `repmgr` and `PgBouncer`, [replicating through PgBouncer is not supported](https://github.com/pgbouncer/pgbouncer/issues/382#issuecomment-517911529) and the secondary must replicate directly from the leader node in the `repmgr` cluster. Therefore, when there is a failover in the `repmgr` cluster, you will need to manually re-point your secondary site to replicate from the new leader with:
 ```shell
-sudo gitlab-ctl start patroni
+sudo gitlab-ctl replicate-geo-database --host=<new_leader_ip> --replication-slot=<slot_name>
 ```
-No further configuration or intervention is needed.
+Otherwise, the replication will not happen anymore, even if the original node gets re-added as a follower node. This will re-sync your secondary site database and may take a long time depending on the amount of data to sync.
-### Maintenance procedure for Patroni
+### Repmgr Restore procedure
-With Patroni enabled, you can run a planned maintenance. If you want to do some maintenance work on one node and you
+If a node fails, it can be removed from the cluster, or added back as a standby
-don't want Patroni to manage it, you can use put it into maintenance mode:
+after it has been restored to service.
-```shell
+#### Remove a standby from the cluster
-sudo gitlab-ctl patroni pause
-```
-When Patroni runs in a paused mode, it does not change the state of PostgreSQL. Once you are done you can resume Patroni:
+  From any other node in the cluster, run:
-```shell
+  ```shell
-sudo gitlab-ctl patroni resume
+  gitlab-ctl repmgr standby unregister --node=X
-```
+  ```
-For further details, see [Patroni documentation on this subject](https://patroni.readthedocs.io/en/latest/pause.html).
+  where X is the value of node in `repmgr.conf` on the old server.
-### Switching from repmgr to Patroni
+  To find this, you can use:
-WARNING:
+  ```shell
-Although switching from repmgr to Patroni is fairly straightforward the other way around is not. Rolling back from
+  awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf
-Patroni to repmgr can be complicated and may involve deletion of data directory. If you need to do that, please contact
+  ```
-GitLab support.
-You can switch an exiting database cluster to use Patroni instead of repmgr with the following steps:
+  It will output something like:
-1. Stop repmgr on all replica nodes and lastly with the primary node:
+  ```plaintext
+  959789412
+  ```
+  Then you will use this ID to unregister the node:
  ```shell
-   sudo gitlab-ctl stop repmgrd
+  gitlab-ctl repmgr standby unregister --node=959789412
  ```
-1. Stop PostgreSQL on all replica nodes:
+#### Add a node as a standby server
+  From the standby node, run:
  ```shell
-   sudo gitlab-ctl stop postgresql
+  gitlab-ctl repmgr standby follow NEW_MASTER
+  gitlab-ctl restart repmgrd
  ```
-   NOTE:
+  WARNING:
-   Ensure that there is no `walsender` process running on the primary node.
+  When the server is brought back online, and before
-   `ps aux | grep walsender` must not show any running process.
+  you switch it to a standby node, repmgr will report that there are two masters.
+  If there are any clients that are still attempting to write to the old master,
+  this will cause a split, and the old master will need to be resynced from
+  scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`.
-1. On the primary node, [configure Patroni](#configuring-patroni-cluster). Remove `repmgr` and any other
+#### Add a failed master back into the cluster as a standby node
-   repmgr-specific configuration. Also remove any configuration that is related to PostgreSQL replication.
-1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on the primary node. It will become
+  Once `repmgrd` and PostgreSQL are running, the node will need to follow the new
-   the leader. You can check this with:
+  as a standby node.
  ```shell
-   sudo gitlab-ctl tail patroni
+  gitlab-ctl repmgr standby follow NEW_MASTER
  ```
-1. Repeat the last two steps for all replica nodes. `gitlab.rb` should look the same on all nodes.
+  Once the node is following the new master as a standby, the node needs to be
-1. Optional: You can remove `gitlab_repmgr` database and role on the primary.
+  [unregistered from the cluster on the new master node](#remove-a-standby-from-the-cluster).
-### Upgrading PostgreSQL major version in a Patroni cluster
+  Once the old master node has been unregistered from the cluster, it will need
+  to be setup as a new standby:
-As of GitLab 13.3, PostgreSQL 11.7 and 12.3 are both shipped with Omnibus GitLab. GitLab still
+  ```shell
-uses PostgreSQL 11 by default. Therefore `gitlab-ctl pg-upgrade` does not automatically upgrade
+  gitlab-ctl repmgr standby setup NEW_MASTER
-to PostgreSQL 12. If you want to upgrade to PostgreSQL 12, you must ask for it explicitly.
+  ```
-WARNING:
+  Failure to unregister and read the old master node can lead to subsequent failovers
-The procedure for upgrading PostgreSQL in a Patroni cluster is different than when upgrading using repmgr.
+  not working.
-The following outlines the key differences and important considerations that need to be accounted for when
-upgrading PostgreSQL.
-Here are a few key facts that you must consider before upgrading PostgreSQL:
+### Alternate configurations
- The main point is that you will have to **shut down the Patroni cluster**. This means that your
+#### Database authorization
-  GitLab deployment will be down for the duration of database upgrade or, at least, as long as your leader
-  node is upgraded. This can be **a significant downtime depending on the size of your database**.
- Upgrading PostgreSQL creates a new data directory with a new control data. From Patroni's perspective
+By default, we give any host on the database network the permission to perform
-  this is a new cluster that needs to be bootstrapped again. Therefore, as part of the upgrade procedure,
+repmgr operations using PostgreSQL's `trust` method. If you do not want this
-  the cluster state, which is stored in Consul, will be wiped out. Once the upgrade is completed, Patroni
+level of trust, there are alternatives.
-  will be instructed to bootstrap a new cluster. **Note that this will change your _cluster ID_**.
- The procedures for upgrading leader and replicas are not the same. That is why it is important to use the
+You can trust only the specific nodes that will be database clusters, or you
-  right procedure on each node.
+can require md5 authentication.
- Upgrading a replica node **deletes the data directory and resynchronizes it** from the leader using the
+#### Trust specific addresses
-  configured replication method (currently `pg_basebackup` is the only available option). It might take some
-  time for replica to catch up with the leader, depending on the size of your database.
- An overview of the upgrade procedure is outlined in [Patoni's documentation](https://patroni.readthedocs.io/en/latest/existing_data.html#major-upgrade-of-postgresql-version).
+If you know the IP address, or FQDN of all database and PgBouncer nodes in the
-  You can still use `gitlab-ctl pg-upgrade` which implements this procedure with a few adjustments.
+cluster, you can trust only those nodes.
-Considering these, you should carefully plan your PostgreSQL upgrade:
+In `/etc/gitlab/gitlab.rb` on all of the database nodes, set
+`repmgr['trust_auth_cidr_addresses']` to an array of strings containing all of
+the addresses.
-1. Find out which node is the leader and which node is a replica:
+If setting to a node's FQDN, they must have a corresponding PTR record in DNS.
+If setting to a node's IP address, specify it as `XXX.XXX.XXX.XXX/32`.
-   ```shell
+For example:
-   gitlab-ctl patroni members
-   ```
-   NOTE:
+```ruby
-   `gitlab-ctl pg-upgrade` tries to detect the role of the node. If for any reason the auto-detection
+repmgr['trust_auth_cidr_addresses'] = %w(192.168.1.44/32 db2.example.com)
-   does not work or you believe it did not detect the role correctly, you can use the `--leader` or `--replica`
+```
-   arguments to manually override it.
-1. Stop Patroni **only on replicas**.
+#### MD5 Authentication
-   ```shell
+If you are running on an untrusted network, repmgr can use md5 authentication
-   sudo gitlab-ctl stop patroni
+with a [`.pgpass` file](https://www.postgresql.org/docs/11/libpq-pgpass.html)
-   ```
+to authenticate.
-1. Enable the maintenance mode on the **application node**:
+You can specify by IP address, FQDN, or by subnet, using the same format as in
+the previous section:
+1. On the current master node, create a password for the `gitlab` and
+   `gitlab_repmgr` user:
   ```shell
-   sudo gitlab-ctl deploy-page up
+   gitlab-psql -d template1
+   template1=# \password gitlab_repmgr
+   Enter password: ****
+   Confirm password: ****
+   template1=# \password gitlab
   ```
-1. Upgrade PostgreSQL on **the leader node** and make sure that the upgrade is completed successfully:
+1. On each database node:
+   1. Edit `/etc/gitlab/gitlab.rb`:
+      1. Ensure `repmgr['trust_auth_cidr_addresses']` is **not** set
+      1. Set `postgresql['md5_auth_cidr_addresses']` to the desired value
+      1. Set `postgresql['sql_replication_user'] = 'gitlab_repmgr'`
+      1. Reconfigure with `gitlab-ctl reconfigure`
+      1. Restart PostgreSQL with `gitlab-ctl restart postgresql`
+   1. Create a `.pgpass` file. Enter the `gitlab_repmgr` password twice to
+      when asked:
      ```shell
-   sudo gitlab-ctl pg-upgrade -V 12
+      gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*'
      ```
-1. Check the status of the leader and cluster. You can only proceed if you have a healthy leader:
+1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`:
+   1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for
+      the `gitlab` database user
+   1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect
-   ```shell
+## Troubleshooting
-   gitlab-ctl patroni check-leader
-   # OR
+### Consul and PostgreSQL changes not taking effect
-   gitlab-ctl patroni members
+Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading.
-   ```
-1. You can now disable the maintenance mode on the **application node**:
+To restart either service, run `gitlab-ctl restart SERVICE`
-   ```shell
+For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. To be safe, you can stop `repmgrd` on the standby nodes first with `gitlab-ctl stop repmgrd`, then start afterwards with `gitlab-ctl start repmgrd`.
-   sudo gitlab-ctl deploy-page down
-   ```
-1. Upgrade PostgreSQL **on replicas** (you can do this in parallel on all of them):
+On the Consul server nodes, it is important to [restart the Consul service](../consul.md#restart-consul) in a controlled manner.
-   ```shell
+### `gitlab-ctl repmgr-check-master` command produces errors
-   sudo gitlab-ctl pg-upgrade -V 12
-   ```
-NOTE:
+If this command displays errors about database permissions it is likely that something failed during
-Reverting PostgreSQL upgrade with `gitlab-ctl revert-pg-upgrade` has the same considerations as
+install, resulting in the `gitlab-consul` database user getting incorrect permissions. Follow these
-`gitlab-ctl pg-upgrade`. You should follow the same procedure by first stopping the replicas,
+steps to fix the problem:
-then reverting the leader, and finally reverting the replicas.
+1. On the master database node, connect to the database prompt - `gitlab-psql -d template1`
+1. Delete the `gitlab-consul` user - `DROP USER "gitlab-consul";`
+1. Exit the database prompt - `\q`
+1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) and the user will be re-added with the proper permissions.
+1. Change to the `gitlab-consul` user - `su - gitlab-consul`
+1. Try the check command again - `gitlab-ctl repmgr-check-master`.
+Now there should not be errors. If errors still occur then there is another problem.
+### PgBouncer error `ERROR: pgbouncer cannot connect to server`
+You may get this error when running `gitlab-rake gitlab:db:configure` or you
+may see the error in the PgBouncer log file.
+```plaintext
+PG::ConnectionBad: ERROR:  pgbouncer cannot connect to server
+```
+The problem may be that your PgBouncer node's IP address is not included in the
+`trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes.
+You can confirm that this is the issue by checking the PostgreSQL log on the master
+database node. If you see the following error then `trust_auth_cidr_addresses`
+is the problem.
+```plaintext
+2018-03-29_13:59:12.11776 FATAL:  no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off
+```
+To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`.
+```ruby
+postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>)
+```
+[Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
+### Issues with other components
+If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page:
+- [Consul](../consul.md#troubleshooting-consul)
+- [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting)