Skip to content

G
gitlab-ce
Commit f55d8570 authored by Evan Read's avatar Evan Read
Browse files
Options

Merge branch 'docs-geo-markdown-1' into 'master' 

Docs: Clean up markdown spacing in geo docs

See merge request gitlab-org/gitlab-ce!30112
parents 4771ad2c f3ed4ecc
Hide whitespace changes
Inline Side-by-side
Showing with 1092 additions and 1091 deletions
doc/administration/geo/disaster_recovery/background_verification.md
View file @ f55d8570
......@@ -143,11 +143,13 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch
1. On the project admin page get the **Gitaly storage name**, and **Gitaly relative path**:
![Project admin page](img/checksum-differences-admin-project-page.png)
1. Navigate to the project's repository directory on both **primary** and **secondary** nodes (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure.
1. Navigate to the project's repository directory on both **primary** and **secondary** nodes
(the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs`
is customized, check the directory layout on your server to be sure.
```sh
cd /var/opt/gitlab/git-data/repositories
```
```sh
cd /var/opt/gitlab/git-data/repositories
```
1. Run the following command on the **primary** node, redirecting the output to a file:
......
doc/administration/geo/disaster_recovery/bring_primary_back.md
View file @ f55d8570
......@@ -21,20 +21,20 @@ To bring the former **primary** node up to date:
1. SSH into the former **primary** node that has fallen behind.
1. Make sure all the services are up:
```sh
sudo gitlab-ctl start
```
> **Note 1:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary],
> you need to undo those steps now. For Debian/Ubuntu you just need to run
> `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install
> the GitLab instance from scratch and set it up as a **secondary** node by
> following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step.
>
> **Note 2:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record)
> for this node during disaster recovery procedure you may need to [block
> all the writes to this node](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/gitlab-geo/planned-failover.md#block-primary-traffic)
> during this procedure.
```sh
sudo gitlab-ctl start
```
NOTE: **Note:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary],
you need to undo those steps now. For Debian/Ubuntu you just need to run
`sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install
the GitLab instance from scratch and set it up as a **secondary** node by
following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step.
NOTE: **Note:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record)
for this node during disaster recovery procedure you may need to [block
all the writes to this node](planned_failover.md#prevent-updates-to-the-primary-node)
during this procedure.
1. [Setup database replication][database-replication]. Note that in this
case, **primary** node refers to the current **primary** node, and **secondary** node refers to the
......
doc/administration/geo/disaster_recovery/index.md
View file @ f55d8570
......@@ -39,48 +39,50 @@ must disable the **primary** node.
1. SSH into the **primary** node to stop and disable GitLab, if possible:
```sh
sudo gitlab-ctl stop
```
```sh
sudo gitlab-ctl stop
```
Prevent GitLab from starting up again if the server unexpectedly reboots:
Prevent GitLab from starting up again if the server unexpectedly reboots:
```sh
sudo systemctl disable gitlab-runsvdir
```
```sh
sudo systemctl disable gitlab-runsvdir
```
> **CentOS only**: In CentOS 6 or older, there is no easy way to prevent GitLab from being
> started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]).
> It may be safest to uninstall the GitLab package completely:
NOTE: **Note:**
(**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being
started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]).
It may be safest to uninstall the GitLab package completely:
```sh
yum remove gitlab-ee
```
```sh
yum remove gitlab-ee
```
> **Ubuntu 14.04 LTS**: If you are using an older version of Ubuntu
> or any other distro based on the Upstart init system, you can prevent GitLab
> from starting if the machine reboots by doing the following:
NOTE: **Note:**
(**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu
or any other distro based on the Upstart init system, you can prevent GitLab
from starting if the machine reboots by doing the following:
```sh
initctl stop gitlab-runsvvdir
echo 'manual' > /etc/init/gitlab-runsvdir.override
initctl reload-configuration
```
```sh
initctl stop gitlab-runsvvdir
echo 'manual' > /etc/init/gitlab-runsvdir.override
initctl reload-configuration
```
1. If you do not have SSH access to the **primary** node, take the machine offline and
prevent it from rebooting by any means at your disposal.
Since there are many ways you may prefer to accomplish this, we will avoid a
single recommendation. You may need to:
- Reconfigure the load balancers.
- Change DNS records (e.g., point the primary DNS record to the **secondary**
node in order to stop usage of the **primary** node).
- Stop the virtual servers.
- Block traffic through a firewall.
- Revoke object storage permissions from the **primary** node.
- Physically disconnect a machine.
1. If you plan to
[update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record),
prevent it from rebooting by any means at your disposal.
Since there are many ways you may prefer to accomplish this, we will avoid a
single recommendation. You may need to:
- Reconfigure the load balancers.
- Change DNS records (e.g., point the primary DNS record to the **secondary**
node in order to stop usage of the **primary** node).
- Stop the virtual servers.
- Block traffic through a firewall.
- Revoke object storage permissions from the **primary** node.
- Physically disconnect a machine.
1. If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record),
you may wish to lower the TTL now to speed up propagation.
### Step 3. Promoting a **secondary** node
......@@ -94,26 +96,26 @@ the **secondary** to the **primary**.
1. SSH in to your **secondary** node and login as root:
```sh
sudo -i
```
```sh
sudo -i
```
1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by
removing any lines that enabled the `geo_secondary_role`:
```ruby
## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
geo_secondary_role['enable'] = true
```ruby
## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
geo_secondary_role['enable'] = true
## In 11.5+ documentation, the role was enabled as follows. Remove this line.
roles ['geo_secondary_role']
```
## In 11.5+ documentation, the role was enabled as follows. Remove this line.
roles ['geo_secondary_role']
```
1. Promote the **secondary** node to the **primary** node. Execute:
```sh
gitlab-ctl promote-to-primary-node
```
```sh
gitlab-ctl promote-to-primary-node
```
1. Verify you can connect to the newly promoted **primary** node using the URL used
previously for the **secondary** node.
......@@ -129,31 +131,31 @@ do this manually.
1. SSH in to the database node in the **secondary** and trigger PostgreSQL to
promote to read-write:
```bash
sudo gitlab-pg-ctl promote
```
```bash
sudo gitlab-pg-ctl promote
```
1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to
reflect its new status as **primary** by removing any lines that enabled the
`geo_secondary_role`:
```ruby
## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
geo_secondary_role['enable'] = true
```ruby
## In pre-11.5 documentation, the role was enabled as follows. Remove this line.
geo_secondary_role['enable'] = true
## In 11.5+ documentation, the role was enabled as follows. Remove this line.
roles ['geo_secondary_role']
```
## In 11.5+ documentation, the role was enabled as follows. Remove this line.
roles ['geo_secondary_role']
```
After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each
machine so the changes take effect.
After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each
machine so the changes take effect.
1. Promote the **secondary** to **primary**. SSH into a single application
server and execute:
```bash
sudo gitlab-rake geo:set_secondary_as_primary
```
```bash
sudo gitlab-rake geo:set_secondary_as_primary
```
1. Verify you can connect to the newly promoted **primary** using the URL used
previously for the **secondary**.
......@@ -167,37 +169,37 @@ secondary domain, like changing Git remotes and API URLs.
1. SSH into the **secondary** node and login as root:
```sh
sudo -i
```
```sh
sudo -i
```
1. Update the primary domain's DNS record. After updating the primary domain's
DNS records to point to the **secondary** node, edit `/etc/gitlab/gitlab.rb` on the
**secondary** node to reflect the new URL:
```ruby
# Change the existing external_url configuration
external_url 'https://<new_external_url>'
```
```ruby
# Change the existing external_url configuration
external_url 'https://<new_external_url>'
```
NOTE: **Note**
Changing `external_url` won't prevent access via the old secondary URL, as
long as the secondary DNS records are still intact.
NOTE: **Note**
Changing `external_url` won't prevent access via the old secondary URL, as
long as the secondary DNS records are still intact.
1. Reconfigure the **secondary** node for the change to take effect:
```sh
gitlab-ctl reconfigure
```
```sh
gitlab-ctl reconfigure
```
1. Execute the command below to update the newly promoted **primary** node URL:
```sh
gitlab-rake geo:update_primary_node_url
```
```sh
gitlab-rake geo:update_primary_node_url
```
This command will use the changed `external_url` configuration defined
in `/etc/gitlab/gitlab.rb`.
This command will use the changed `external_url` configuration defined
in `/etc/gitlab/gitlab.rb`.
1. Verify you can connect to the newly promoted **primary** using its URL.
If you updated the DNS records for the primary domain, these changes may
......@@ -231,62 +233,61 @@ and after that you also need two extra steps.
1. SSH into the new **primary** node and login as root:
```sh
sudo -i
```
```sh
sudo -i
```
1. Edit `/etc/gitlab/gitlab.rb`
```ruby
## Enable a Geo Primary role (if you haven't yet)
roles ['geo_primary_role']
```ruby
## Enable a Geo Primary role (if you haven't yet)
roles ['geo_primary_role']
##
# Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be
# public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32']
##
postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32']
##
# Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be
# public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32']
##
postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32']
# Every secondary server needs to have its own slot so specify the number of secondary nodes you're going to have
postgresql['max_replication_slots'] = 1
# Every secondary server needs to have its own slot so specify the number of secondary nodes you're going to have
postgresql['max_replication_slots'] = 1
##
## Disable automatic database migrations temporarily
## (until PostgreSQL is restarted and listening on the private address).
##
gitlab_rails['auto_migrate'] = false
##
## Disable automatic database migrations temporarily
## (until PostgreSQL is restarted and listening on the private address).
##
gitlab_rails['auto_migrate'] = false
```
```
(For more details about these settings you can read [Configure the primary server][configure-the-primary-server])
(For more details about these settings you can read [Configure the primary server][configure-the-primary-server])
1. Save the file and reconfigure GitLab for the database listen changes and
the replication slot changes to be applied.
```sh
gitlab-ctl reconfigure
```
```sh
gitlab-ctl reconfigure
```
Restart PostgreSQL for its changes to take effect:
Restart PostgreSQL for its changes to take effect:
```sh
gitlab-ctl restart postgresql
```
```sh
gitlab-ctl restart postgresql
```
1. Re-enable migrations now that PostgreSQL is restarted and listening on the
private address.
Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`:
Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`:
```ruby
gitlab_rails['auto_migrate'] = true
```
```ruby
gitlab_rails['auto_migrate'] = true
```
Save the file and reconfigure GitLab:
Save the file and reconfigure GitLab:
```sh
gitlab-ctl reconfigure
```
```sh
gitlab-ctl reconfigure
```
### Step 2. Initiate the replication process
......
doc/administration/geo/disaster_recovery/planned_failover.md
View file @ f55d8570
......@@ -143,26 +143,26 @@ access to the **primary** node during the maintenance window.
all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and
the **secondary** node's IP.
For instance, you might run the following commands on the server(s) making up your **primary** node:
For instance, you might run the following commands on the server(s) making up your **primary** node:
```sh
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT
sudo iptables -A INPUT --destination-port 22 -j REJECT
```sh
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT
sudo iptables -A INPUT --destination-port 22 -j REJECT
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT
sudo iptables -A INPUT --tcp-dport 80 -j REJECT
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT
sudo iptables -A INPUT --tcp-dport 80 -j REJECT
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT
sudo iptables -A INPUT --tcp-dport 443 -j REJECT
```
sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT
sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT
sudo iptables -A INPUT --tcp-dport 443 -j REJECT
```
From this point, users will be unable to view their data or make changes on the
**primary** node. They will also be unable to log in to the **secondary** node.
However, existing sessions will work for the remainder of the maintenance period, and
public data will be accessible throughout.
From this point, users will be unable to view their data or make changes on the
**primary** node. They will also be unable to log in to the **secondary** node.
However, existing sessions will work for the remainder of the maintenance period, and
public data will be accessible throughout.
1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via
another IP. The server should refuse connection.
......@@ -187,10 +187,11 @@ access to the **primary** node during the maintenance window.
before it is completed will cause the work to be lost.
1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the
following conditions to be true of the **secondary** node you are failing over to:
- All replication meters to each 100% replicated, 0% failures.
- All verification meters reach 100% verified, 0% failures.
- Database replication lag is 0ms.
- The Geo log cursor is up to date (0 events behind).
- All replication meters to each 100% replicated, 0% failures.
- All verification meters reach 100% verified, 0% failures.
- Database replication lag is 0ms.
- The Geo log cursor is up to date (0 events behind).
1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues**
and wait for all the `geo` queues to drop to 0 queued and 0 running jobs.
......
doc/administration/geo/replication/configuration.md
View file @ f55d8570
......@@ -16,11 +16,10 @@ The basic steps of configuring a **secondary** node are to:
You are encouraged to first read through all the steps before executing them
in your testing/production environment.
> **Notes:**
> - **Do not** setup any custom authentication for the **secondary** nodes. This will be
handled by the **primary** node.
> - Any change that requires access to the **Admin Area** needs to be done in the
**primary** node because the **secondary** node is a read-only replica.
NOTE: **Note:**
**Do not** set up any custom authentication for the **secondary** nodes. This will be handled by the **primary** node.
Any change that requires access to the **Admin Area** needs to be done in the
**primary** node because the **secondary** node is a read-only replica.
### Step 1. Manually replicate secret GitLab values
......@@ -31,47 +30,47 @@ they must be manually replicated to the **secondary** node.
1. SSH into the **primary** node, and execute the command below:
```sh
sudo cat /etc/gitlab/gitlab-secrets.json
```
```sh
sudo cat /etc/gitlab/gitlab-secrets.json
```
This will display the secrets that need to be replicated, in JSON format.
This will display the secrets that need to be replicated, in JSON format.
1. SSH into the **secondary** node and login as the `root` user:
```sh
sudo -i
```
```sh
sudo -i
```
1. Make a backup of any existing secrets:
```sh
mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F`
```
```sh
mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F`
```
1. Copy `/etc/gitlab/gitlab-secrets.json` from the **primary** node to the **secondary** node, or
copy-and-paste the file contents between nodes:
```sh
sudo editor /etc/gitlab/gitlab-secrets.json
```sh
sudo editor /etc/gitlab/gitlab-secrets.json
# paste the output of the `cat` command you ran on the primary
# save and exit
```
# paste the output of the `cat` command you ran on the primary
# save and exit
```
1. Ensure the file permissions are correct:
```sh
chown root:root /etc/gitlab/gitlab-secrets.json
chmod 0600 /etc/gitlab/gitlab-secrets.json
```
```sh
chown root:root /etc/gitlab/gitlab-secrets.json
chmod 0600 /etc/gitlab/gitlab-secrets.json
```
1. Reconfigure the **secondary** node for the change to take effect:
```sh
gitlab-ctl reconfigure
gitlab-ctl restart
```
```sh
gitlab-ctl reconfigure
gitlab-ctl restart
```
### Step 2. Manually replicate the **primary** node's SSH host keys
......@@ -89,80 +88,80 @@ keys must be manually replicated to the **secondary** node.
1. SSH into the **secondary** node and login as the `root` user:
```sh
sudo -i
```
```sh
sudo -i
```
1. Make a backup of any existing SSH host keys:
```sh
find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \;
```
```sh
find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \;
```
1. Copy OpenSSH host keys from the **primary** node:
If you can access your **primary** node using the **root** user:
If you can access your **primary** node using the **root** user:
```sh
# Run this from the secondary node, change `<primary_node_fqdn>` for the IP or FQDN of the server
scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh
```
```sh
# Run this from the secondary node, change `<primary_node_fqdn>` for the IP or FQDN of the server
scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh
```
If you only have access through a user with **sudo** privileges:
If you only have access through a user with **sudo** privileges:
```sh
# Run this from your primary node:
sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key*
```sh
# Run this from your primary node:
sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key*
# Run this from your secondary node:
scp <user_with_sudo>@<primary_node_fqdn>:geo-host-key.tar.gz .
tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh
```
# Run this from your secondary node:
scp <user_with_sudo>@<primary_node_fqdn>:geo-host-key.tar.gz .
tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh
```
1. On your **secondary** node, ensure the file permissions are correct:
```sh
chown root:root /etc/ssh/ssh_host_*_key*
chmod 0600 /etc/ssh/ssh_host_*_key*
```
```sh
chown root:root /etc/ssh/ssh_host_*_key*
chmod 0600 /etc/ssh/ssh_host_*_key*
```
1. To verify key fingerprint matches, execute the following command on both nodes:
```sh
for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done
```
```sh
for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done
```
You should get an output similar to this one and they should be identical on both nodes:
You should get an output similar to this one and they should be identical on both nodes:
```sh
1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA)
256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA)
256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519)
2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA)
```
```sh
1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA)
256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA)
256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519)
2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA)
```
1. Verify that you have the correct public keys for the existing private keys:
```sh
# This will print the fingerprint for private keys:
for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done
```sh
# This will print the fingerprint for private keys:
for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done
# This will print the fingerprint for public keys:
for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done
```
# This will print the fingerprint for public keys:
for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done
```
NOTE: **Note**:
The output for private keys and public keys command should generate the same fingerprint.
NOTE: **Note:**
The output for private keys and public keys command should generate the same fingerprint.
1. Restart sshd on your **secondary** node:
```sh
# Debian or Ubuntu installations
sudo service ssh reload
```sh
# Debian or Ubuntu installations
sudo service ssh reload
# CentOS installations
sudo service sshd reload
```
# CentOS installations
sudo service sshd reload
```
### Step 3. Add the **secondary** node
......@@ -176,22 +175,22 @@ keys must be manually replicated to the **secondary** node.
1. Click the **Add node** button.
1. SSH into your GitLab **secondary** server and restart the services:
```sh
gitlab-ctl restart
```
```sh
gitlab-ctl restart
```
Check if there are any common issue with your Geo setup by running:
Check if there are any common issue with your Geo setup by running:
```sh
gitlab-rake gitlab:geo:check
```
```sh
gitlab-rake gitlab:geo:check
```
1. SSH into your **primary** server and login as root to verify the
**secondary** node is reachable or there are any common issue with your Geo setup:
```sh
gitlab-rake gitlab:geo:check
```
```sh
gitlab-rake gitlab:geo:check
```
Once added to the admin panel and restarted, the **secondary** node will automatically start
replicating missing data from the **primary** node in a process known as **backfill**.
......@@ -250,9 +249,8 @@ The two most obvious issues that can become apparent in the dashboard are:
1. Database replication not working well.
1. Instance to instance notification not working. In that case, it can be
something of the following:
- You are using a custom certificate or custom CA (see the
[troubleshooting document]).
- The instance is firewalled (check your firewall rules).
- You are using a custom certificate or custom CA (see the [troubleshooting document](troubleshooting.md)).
- The instance is firewalled (check your firewall rules).
Please note that disabling a **secondary** node will stop the synchronization process.
......@@ -304,5 +302,4 @@ See the [troubleshooting document](troubleshooting.md).
[gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789
[gitlab-com/infrastructure#2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821
[omnibus-ssl]: https://docs.gitlab.com/omnibus/settings/ssl.html
[troubleshooting document]: troubleshooting.md
[using-geo]: using_a_geo_server.md
doc/administration/geo/replication/database.md
View file @ f55d8570
......@@ -52,186 +52,188 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
1. SSH into your GitLab **primary** server and login as root:
```sh
sudo -i
```
```sh
sudo -i
```
1. Execute the command below to define the node as **primary** node:
```sh
gitlab-ctl set-geo-primary-node
```
```sh
gitlab-ctl set-geo-primary-node
```
This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`.
This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`.
1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined:
Generate a MD5 hash of the desired password:
Generate a MD5 hash of the desired password:
```sh
gitlab-ctl pg-password-md5 gitlab
# Enter password: <your_password_here>
# Confirm password: <your_password_here>
# fca0b89a972d69f00eb3ec98a5838484
```
```sh
gitlab-ctl pg-password-md5 gitlab
# Enter password: <your_password_here>
# Confirm password: <your_password_here>
# fca0b89a972d69f00eb3ec98a5838484
```
Edit `/etc/gitlab/gitlab.rb`:
Edit `/etc/gitlab/gitlab.rb`:
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
# Every node that runs Unicorn or Sidekiq needs to have the database
# password specified as below. If you have a high-availability setup, this
# must be present in all application nodes.
gitlab_rails['db_password'] = '<your_password_here>'
```
# Every node that runs Unicorn or Sidekiq needs to have the database
# password specified as below. If you have a high-availability setup, this
# must be present in all application nodes.
gitlab_rails['db_password'] = '<your_password_here>'
```
1. Omnibus GitLab already has a [replication user]
called `gitlab_replicator`. You must set the password for this user manually.
You will be prompted to enter a password:
```sh
gitlab-ctl set-replication-password
```
```sh
gitlab-ctl set-replication-password
```
This command will also read the `postgresql['sql_replication_user']` Omnibus
setting in case you have changed `gitlab_replicator` username to something
else.
This command will also read the `postgresql['sql_replication_user']` Omnibus
setting in case you have changed `gitlab_replicator` username to something
else.
If you are using an external database not managed by Omnibus GitLab, you need
to create the replicator user and define a password to it manually:
If you are using an external database not managed by Omnibus GitLab, you need
to create the replicator user and define a password to it manually:
```sql
--- Create a new user 'replicator'
CREATE USER gitlab_replicator;
```sql
--- Create a new user 'replicator'
CREATE USER gitlab_replicator;
--- Set/change a password and grants replication privilege
ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>';
```
--- Set/change a password and grants replication privilege
ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>';
```
1. Configure PostgreSQL to listen on network interfaces:
For security reasons, PostgreSQL does not listen on any network interfaces
by default. However, Geo requires the **secondary** node to be able to
connect to the **primary** node's database. For this reason, we need the address of
each node. Note: For external PostgreSQL instances, see [additional instructions](external_database.md).
If you are using a cloud provider, you can lookup the addresses for each
Geo node through your cloud provider's management console.
To lookup the address of a Geo node, SSH in to the Geo node and execute:
```sh
##
## Private address
##
ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}'
##
## Public address
##
echo "External address: $(curl --silent ipinfo.io/ip)"
```
In most cases, the following addresses will be used to configure GitLab
Geo:
| Configuration | Address |
|:----------------------------------------|:------------------------------------------------------|
| `postgresql['listen_address']` | **Primary** node's public or VPC private address. |
| `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. |
If you are using Google Cloud Platform, SoftLayer, or any other vendor that
provides a virtual private cloud (VPC) you can use the **secondary** node's private
address (corresponds to "internal address" for Google Cloud Platform) for
`postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`.
The `listen_address` option opens PostgreSQL up to network connections
with the interface corresponding to the given address. See [the PostgreSQL
documentation][pg-docs-runtime-conn] for more details.
Depending on your network configuration, the suggested addresses may not
be correct. If your **primary** node and **secondary** nodes connect over a local
area network, or a virtual network connecting availability zones like
[Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/)
you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`.
Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP
addresses with addresses appropriate to your network configuration:
```ruby
##
## Geo Primary role
## - configure dependent flags automatically to enable Geo
##
roles ['geo_primary_role']
##
## Primary address
## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node
##
postgresql['listen_address'] = '<primary_node_ip>'
##
# Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be
# public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32']
##
postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32']
##
## Replication settings
## - set this to be the number of Geo secondary nodes you have
##
postgresql['max_replication_slots'] = 1
# postgresql['max_wal_senders'] = 10
# postgresql['wal_keep_segments'] = 10
##
## Disable automatic database migrations temporarily
## (until PostgreSQL is restarted and listening on the private address).
##
gitlab_rails['auto_migrate'] = false
```
For security reasons, PostgreSQL does not listen on any network interfaces
by default. However, Geo requires the **secondary** node to be able to
connect to the **primary** node's database. For this reason, we need the address of
each node.
NOTE: **Note:** For external PostgreSQL instances, see [additional instructions](external_database.md).
If you are using a cloud provider, you can lookup the addresses for each
Geo node through your cloud provider's management console.
To lookup the address of a Geo node, SSH in to the Geo node and execute:
```sh
##
## Private address
##
ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}'
##
## Public address
##
echo "External address: $(curl --silent ipinfo.io/ip)"
```
In most cases, the following addresses will be used to configure GitLab
Geo:
| Configuration | Address |
|:----------------------------------------|:------------------------------------------------------|
| `postgresql['listen_address']` | **Primary** node's public or VPC private address. |
| `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. |
If you are using Google Cloud Platform, SoftLayer, or any other vendor that
provides a virtual private cloud (VPC) you can use the **secondary** node's private
address (corresponds to "internal address" for Google Cloud Platform) for
`postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`.
The `listen_address` option opens PostgreSQL up to network connections
with the interface corresponding to the given address. See [the PostgreSQL
documentation][pg-docs-runtime-conn] for more details.
Depending on your network configuration, the suggested addresses may not
be correct. If your **primary** node and **secondary** nodes connect over a local
area network, or a virtual network connecting availability zones like
[Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/)
you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`.
Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP
addresses with addresses appropriate to your network configuration:
```ruby
##
## Geo Primary role
## - configure dependent flags automatically to enable Geo
##
roles ['geo_primary_role']
##
## Primary address
## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node
##
postgresql['listen_address'] = '<primary_node_ip>'
##
# Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be
# public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32']
##
postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32']
##
## Replication settings
## - set this to be the number of Geo secondary nodes you have
##
postgresql['max_replication_slots'] = 1
# postgresql['max_wal_senders'] = 10
# postgresql['wal_keep_segments'] = 10
##
## Disable automatic database migrations temporarily
## (until PostgreSQL is restarted and listening on the private address).
##
gitlab_rails['auto_migrate'] = false
```
1. Optional: If you want to add another **secondary** node, the relevant setting would look like:
```ruby
postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32']
```
```ruby
postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32']
```
You may also want to edit the `wal_keep_segments` and `max_wal_senders` to
match your database replication requirements. Consult the [PostgreSQL -
Replication documentation][pg-docs-runtime-replication]
for more information.
You may also want to edit the `wal_keep_segments` and `max_wal_senders` to
match your database replication requirements. Consult the [PostgreSQL -
Replication documentation][pg-docs-runtime-replication]
for more information.
1. Save the file and reconfigure GitLab for the database listen changes and
the replication slot changes to be applied:
```sh
gitlab-ctl reconfigure
```
```sh
gitlab-ctl reconfigure
```
Restart PostgreSQL for its changes to take effect:
Restart PostgreSQL for its changes to take effect:
```sh
gitlab-ctl restart postgresql
```
```sh
gitlab-ctl restart postgresql
```
1. Re-enable migrations now that PostgreSQL is restarted and listening on the
private address.
Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`:
Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`:
```ruby
gitlab_rails['auto_migrate'] = true
```
```ruby
gitlab_rails['auto_migrate'] = true
```
Save the file and reconfigure GitLab:
Save the file and reconfigure GitLab:
```sh
gitlab-ctl reconfigure
```
```sh
gitlab-ctl reconfigure
```
1. Now that the PostgreSQL server is set up to accept remote connections, run
`netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port
......@@ -241,143 +243,143 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
will be used automatically to protect your PostgreSQL traffic from
eavesdroppers, but to protect against active ("man-in-the-middle") attackers,
the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL
`server.crt` file on the **primary** node by running this command:
`server.crt` file on the **primary** node by running this command:
```sh
cat ~gitlab-psql/data/server.crt
```
```sh
cat ~gitlab-psql/data/server.crt
```
Copy the output into a clipboard or into a local file. You
will need it when setting up the **secondary** node! The certificate is not sensitive
data.
Copy the output into a clipboard or into a local file. You
will need it when setting up the **secondary** node! The certificate is not sensitive
data.
### Step 2. Configure the **secondary** server
1. SSH into your GitLab **secondary** server and login as root:
```
sudo -i
```
```
sudo -i
```
1. Stop application server and Sidekiq
```
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq
```
```
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq
```
NOTE: **Note**:
This step is important so we don't try to execute anything before the node is fully configured.
NOTE: **Note:**
This step is important so we don't try to execute anything before the node is fully configured.
1. [Check TCP connectivity][rake-maintenance] to the **primary** node's PostgreSQL server:
```sh
gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432]
```
```sh
gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432]
```
NOTE: **Note**:
If this step fails, you may be using the wrong IP address, or a firewall may
be preventing access to the server. Check the IP address, paying close
attention to the difference between public and private addresses and ensure
that, if a firewall is present, the **secondary** node is permitted to connect to the
**primary** node on port 5432.
NOTE: **Note:**
If this step fails, you may be using the wrong IP address, or a firewall may
be preventing access to the server. Check the IP address, paying close
attention to the difference between public and private addresses and ensure
that, if a firewall is present, the **secondary** node is permitted to connect to the
**primary** node on port 5432.
1. Create a file `server.crt` in the **secondary** server, with the content you got on the last step of the **primary** node's setup:
```
editor server.crt
```
```
editor server.crt
```
1. Set up PostgreSQL TLS verification on the **secondary** node:
Install the `server.crt` file:
Install the `server.crt` file:
```sh
install \
-D \
-o gitlab-psql \
-g gitlab-psql \
-m 0400 \
-T server.crt ~gitlab-psql/.postgresql/root.crt
```
```sh
install \
-D \
-o gitlab-psql \
-g gitlab-psql \
-m 0400 \
-T server.crt ~gitlab-psql/.postgresql/root.crt
```
PostgreSQL will now only recognize that exact certificate when verifying TLS
connections. The certificate can only be replicated by someone with access
to the private key, which is **only** present on the **primary** node.
PostgreSQL will now only recognize that exact certificate when verifying TLS
connections. The certificate can only be replicated by someone with access
to the private key, which is **only** present on the **primary** node.
1. Test that the `gitlab-psql` user can connect to the **primary** node's database
(the default Omnibus database name is gitlabhq_production):
```sh
sudo \
-u gitlab-psql /opt/gitlab/embedded/bin/psql \
--list \
-U gitlab_replicator \
-d "dbname=gitlabhq_production sslmode=verify-ca" \
-W \
-h <primary_node_ip>
```
```sh
sudo \
-u gitlab-psql /opt/gitlab/embedded/bin/psql \
--list \
-U gitlab_replicator \
-d "dbname=gitlabhq_production sslmode=verify-ca" \
-W \
-h <primary_node_ip>
```
When prompted enter the password you set in the first step for the
`gitlab_replicator` user. If all worked correctly, you should see
the list of **primary** node's databases.
When prompted enter the password you set in the first step for the
`gitlab_replicator` user. If all worked correctly, you should see
the list of **primary** node's databases.
A failure to connect here indicates that the TLS configuration is incorrect.
Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node
match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node.
A failure to connect here indicates that the TLS configuration is incorrect.
Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node
match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node.
1. Configure PostgreSQL to enable FDW support:
This step is similar to how we configured the **primary** instance.
We need to enable this, to enable FDW support, even if using a single node.
Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP
addresses with addresses appropriate to your network configuration:
```ruby
##
## Geo Secondary role
## - configure dependent flags automatically to enable Geo
##
roles ['geo_secondary_role']
##
## Secondary address
## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node
##
postgresql['listen_address'] = '<secondary_node_ip>'
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']
##
## Database credentials password (defined previously in primary node)
## - replicate same values here as defined in primary node
##
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
gitlab_rails['db_password'] = '<your_password_here>'
##
## Enable FDW support for the Geo Tracking Database (improves performance)
##
geo_secondary['db_fdw'] = true
```
For external PostgreSQL instances, see [additional instructions](external_database.md).
If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`.
This step is similar to how we configured the **primary** instance.
We need to enable this, to enable FDW support, even if using a single node.
Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP
addresses with addresses appropriate to your network configuration:
```ruby
##
## Geo Secondary role
## - configure dependent flags automatically to enable Geo
##
roles ['geo_secondary_role']
##
## Secondary address
## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node
##
postgresql['listen_address'] = '<secondary_node_ip>'
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']
##
## Database credentials password (defined previously in primary node)
## - replicate same values here as defined in primary node
##
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
gitlab_rails['db_password'] = '<your_password_here>'
##
## Enable FDW support for the Geo Tracking Database (improves performance)
##
geo_secondary['db_fdw'] = true
```
For external PostgreSQL instances, see [additional instructions](external_database.md).
If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`.
1. Reconfigure GitLab for the changes to take effect:
```sh
gitlab-ctl reconfigure
```
```sh
gitlab-ctl reconfigure
```
1. Restart PostgreSQL for the IP change to take effect and reconfigure again:
```sh
gitlab-ctl restart postgresql
gitlab-ctl reconfigure
```
```sh
gitlab-ctl restart postgresql
gitlab-ctl reconfigure
```
This last reconfigure will provision the FDW configuration and enable it.
This last reconfigure will provision the FDW configuration and enable it.
### Step 3. Initiate the replication process
......@@ -394,9 +396,9 @@ data before running `pg_basebackup`.
1. SSH into your GitLab **secondary** server and login as root:
```sh
sudo -i
```
```sh
sudo -i
```
1. Choose a database-friendly name to use for your **secondary** node to
use as the replication slot name. For example, if your domain is
......@@ -404,38 +406,40 @@ data before running `pg_basebackup`.
name as shown in the commands below.
1. Execute the command below to start a backup/restore and begin the replication
CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name.
Using the same slot name between two secondaries will break PostgreSQL replication.
```sh
gitlab-ctl replicate-geo-database \
--slot-name=<secondary_node_name> \
--host=<primary_node_ip>
```
When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator`
user in the first step.
This command also takes a number of additional options. You can use `--help`
to list them all, but here are a couple of tips:
- If PostgreSQL is listening on a non-standard port, add `--port=` as well.
- If your database is too large to be transferred in 30 minutes, you will need
to increase the timeout, e.g., `--backup-timeout=3600` if you expect the
initial replication to take under an hour.
- Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether
(e.g., you know the network path is secure, or you are using a site-to-site
VPN). This is **not** safe over the public Internet!
- You can read more details about each `sslmode` in the
[PostgreSQL documentation][pg-docs-ssl];
the instructions above are carefully written to ensure protection against
both passive eavesdroppers and active "man-in-the-middle" attackers.
- Change the `--slot-name` to the name of the replication slot
to be used on the **primary** database. The script will attempt to create the
replication slot automatically if it does not exist.
- If you're repurposing an old server into a Geo **secondary** node, you'll need to
add `--force` to the command line.
- When not in a production machine you can disable backup step if you
really sure this is what you want by adding `--skip-backup`
CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name.
Using the same slot name between two secondaries will break PostgreSQL replication.
```sh
gitlab-ctl replicate-geo-database \
--slot-name=<secondary_node_name> \
--host=<primary_node_ip>
```
When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator`
user in the first step.
This command also takes a number of additional options. You can use `--help`
to list them all, but here are a couple of tips:
- If PostgreSQL is listening on a non-standard port, add `--port=` as well.
- If your database is too large to be transferred in 30 minutes, you will need
to increase the timeout, e.g., `--backup-timeout=3600` if you expect the
initial replication to take under an hour.
- Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether
(e.g., you know the network path is secure, or you are using a site-to-site
VPN). This is **not** safe over the public Internet!
- You can read more details about each `sslmode` in the
[PostgreSQL documentation][pg-docs-ssl];
the instructions above are carefully written to ensure protection against
both passive eavesdroppers and active "man-in-the-middle" attackers.
- Change the `--slot-name` to the name of the replication slot
to be used on the **primary** database. The script will attempt to create the
replication slot automatically if it does not exist.
- If you're repurposing an old server into a Geo **secondary** node, you'll need to
add `--force` to the command line.
- When not in a production machine you can disable backup step if you
really sure this is what you want by adding `--skip-backup`
The replication process is now complete.
......@@ -452,42 +456,42 @@ it will need a separate read-only user to make [PostgreSQL FDW queries][FDW]
work:
1. On the **primary** Geo database, enter the PostgreSQL on the console as an
admin user. If you are using an Omnibus-managed database, log onto the **primary**
node that is running the PostgreSQL database (the default Omnibus database name is gitlabhq_production):
admin user. If you are using an Omnibus-managed database, log onto the **primary**
node that is running the PostgreSQL database (the default Omnibus database name is gitlabhq_production):
```sh
sudo \
-u gitlab-psql /opt/gitlab/embedded/bin/psql \
-h /var/opt/gitlab/postgresql gitlabhq_production
```
```sh
sudo \
-u gitlab-psql /opt/gitlab/embedded/bin/psql \
-h /var/opt/gitlab/postgresql gitlabhq_production
```
1. Then create the read-only user:
```sql
-- NOTE: Use the password defined earlier
CREATE USER gitlab_geo_fdw WITH password 'mypassword';
GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw;
GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw;
```sql
-- NOTE: Use the password defined earlier
CREATE USER gitlab_geo_fdw WITH password 'mypassword';
GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw;
GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw;
-- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw"
-- automatically.
ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw;
ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw;
```
-- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw"
-- automatically.
ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw;
ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw;
```
1. On the **secondary** nodes, change `/etc/gitlab/gitlab.rb`:
```
geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw'
```
```
geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw'
```
1. Save the file and reconfigure GitLab for the changes to be applied:
```sh
gitlab-ctl reconfigure
```
```sh
gitlab-ctl reconfigure
```
## Troubleshooting
......
doc/administration/geo/replication/external_database.md
View file @ f55d8570
......@@ -4,7 +4,7 @@ This document is relevant if you are using a PostgreSQL instance that is *not
managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or
manually installed and configured PostgreSQL instances.
NOTE: **Note**:
NOTE: **Note:**
We strongly recommend running Omnibus-managed instances as they are actively
developed and tested. We aim to be compatible with most external
(not managed by Omnibus) databases but we do not guarantee compatibility.
......@@ -13,17 +13,17 @@ developed and tested. We aim to be compatible with most external
1. SSH into a GitLab **primary** application server and login as root:
```sh
sudo -i
```
```sh
sudo -i
```
1. Execute the command below to define the node as **primary** node:
```sh
gitlab-ctl set-geo-primary-node
```
```sh
gitlab-ctl set-geo-primary-node
```
This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`.
This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`.
### Configure the external database to be replicated
......@@ -101,26 +101,27 @@ To configure the connection to the external read-replica database and enable Log
1. SSH into a GitLab **secondary** application server and login as root:
```bash
sudo -i
```
```bash
sudo -i
```
1. Edit `/etc/gitlab/gitlab.rb` and add the following
```ruby
##
## Geo Secondary role
## - configure dependent flags automatically to enable Geo
##
roles ['geo_secondary_role']
```ruby
##
## Geo Secondary role
## - configure dependent flags automatically to enable Geo
##
roles ['geo_secondary_role']
# note this is shared between both databases,
# make sure you define the same password in both
gitlab_rails['db_password'] = '<your_password_here>'
# note this is shared between both databases,
# make sure you define the same password in both
gitlab_rails['db_password'] = '<your_password_here>'
gitlab_rails['db_username'] = 'gitlab'
gitlab_rails['db_host'] = '<database_read_replica_host>'
```
gitlab_rails['db_username'] = 'gitlab'
gitlab_rails['db_host'] = '<database_read_replica_host>'
```
1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure)
### Configure the tracking database
......@@ -147,73 +148,72 @@ the tracking database on port 5432.
1. SSH into a GitLab **secondary** server and login as root:
```bash
sudo -i
```
```bash
sudo -i
```
1. Edit `/etc/gitlab/gitlab.rb` with the connection params and credentials for
the machine with the PostgreSQL instance:
the machine with the PostgreSQL instance:
```ruby
geo_secondary['db_username'] = 'gitlab_geo'
geo_secondary['db_password'] = '<your_password_here>'
```ruby
geo_secondary['db_username'] = 'gitlab_geo'
geo_secondary['db_password'] = '<your_password_here>'
geo_secondary['db_host'] = '<tracking_database_host>'
geo_secondary['db_port'] = <tracking_database_port> # change to the correct port
geo_secondary['db_fdw'] = true # enable FDW
geo_postgresql['enable'] = false # don't use internal managed instance
```
geo_secondary['db_host'] = '<tracking_database_host>'
geo_secondary['db_port'] = <tracking_database_port> # change to the correct port
geo_secondary['db_fdw'] = true # enable FDW
geo_postgresql['enable'] = false # don't use internal managed instance
```
1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure)
1. Run the tracking database migrations:
```bash
gitlab-rake geo:db:create
gitlab-rake geo:db:migrate
```
1. Configure the
[PostgreSQL FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html)
connection and credentials:
Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection
params to match your environment. Execute it to set up the FDW connection.
```bash
#!/bin/bash
# Secondary Database connection params:
DB_HOST="<public_ip_or_vpc_private_ip>"
DB_NAME="gitlabhq_production"
DB_USER="gitlab"
DB_PASS="<your_password_here>"
DB_PORT="5432"
# Tracking Database connection params:
GEO_DB_HOST="<public_ip_or_vpc_private_ip>"
GEO_DB_NAME="gitlabhq_geo_production"
GEO_DB_USER="gitlab_geo"
GEO_DB_PORT="5432"
query_exec () {
gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}"
}
query_exec "CREATE EXTENSION postgres_fdw;"
query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');"
query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}', password '${DB_PASS}');"
query_exec "CREATE SCHEMA gitlab_secondary;"
query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};"
```
NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine,
but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using
`psql` for AWS RDS.
```bash
gitlab-rake geo:db:create
gitlab-rake geo:db:migrate
```
1. Configure the [PostgreSQL FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html)
connection and credentials:
Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection
params to match your environment. Execute it to set up the FDW connection.
```bash
#!/bin/bash
# Secondary Database connection params:
DB_HOST="<public_ip_or_vpc_private_ip>"
DB_NAME="gitlabhq_production"
DB_USER="gitlab"
DB_PASS="<your_password_here>"
DB_PORT="5432"
# Tracking Database connection params:
GEO_DB_HOST="<public_ip_or_vpc_private_ip>"
GEO_DB_NAME="gitlabhq_geo_production"
GEO_DB_USER="gitlab_geo"
GEO_DB_PORT="5432"
query_exec () {
gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}"
}
query_exec "CREATE EXTENSION postgres_fdw;"
query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');"
query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}', password '${DB_PASS}');"
query_exec "CREATE SCHEMA gitlab_secondary;"
query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};"
```
NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine,
but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using
`psql` for AWS RDS.
1. Save the file and [restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart)
1. Populate the FDW tables:
```bash
gitlab-rake geo:db:refresh_foreign_tables
```
```bash
gitlab-rake geo:db:refresh_foreign_tables
```
doc/administration/geo/replication/high_availability.md
View file @ f55d8570
......@@ -50,17 +50,17 @@ The following steps enable a GitLab cluster to serve as the **primary** node.
1. Edit `/etc/gitlab/gitlab.rb` and add the following:
```ruby
##
## Enable the Geo primary role
##
roles ['geo_primary_role']
##
## Disable automatic migrations
##
gitlab_rails['auto_migrate'] = false
```
```ruby
##
## Enable the Geo primary role
##
roles ['geo_primary_role']
##
## Disable automatic migrations
##
gitlab_rails['auto_migrate'] = false
```
After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect.
......@@ -107,36 +107,36 @@ Configure the [**secondary** database](database.md) as a read-only replica of
the **primary** database. Use the following as a guide.
1. Edit `/etc/gitlab/gitlab.rb` in the replica database machine, and add the
following:
```ruby
##
## Configure the PostgreSQL role
##
roles ['postgres_role']
##
## Secondary address
## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node
## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node
##
postgresql['listen_address'] = '<secondary_node_ip>'
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32']
##
## Database credentials password (defined previously in primary node)
## - replicate same values here as defined in primary node
##
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
gitlab_rails['db_password'] = '<your_password_here>'
##
## When running the Geo tracking database on a separate machine, disable it
## here and allow connections from the tracking database host. And ensure
## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above.
##
geo_postgresql['enable'] = false
```
following:
```ruby
##
## Configure the PostgreSQL role
##
roles ['postgres_role']
##
## Secondary address
## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node
## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node
##
postgresql['listen_address'] = '<secondary_node_ip>'
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32']
##
## Database credentials password (defined previously in primary node)
## - replicate same values here as defined in primary node
##
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
gitlab_rails['db_password'] = '<your_password_here>'
##
## When running the Geo tracking database on a separate machine, disable it
## here and allow connections from the tracking database host. And ensure
## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above.
##
geo_postgresql['enable'] = false
```
After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect.
......@@ -151,47 +151,47 @@ only a single machine, rather than as a PostgreSQL cluster.
Configure the tracking database.
1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the
following:
```ruby
##
## Enable the Geo secondary tracking database
##
geo_postgresql['enable'] = true
geo_postgresql['listen_address'] = '<ip_address_of_this_host>'
geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>'
##
## Configure FDW connection to the replica database
##
geo_secondary['db_fdw'] = true
geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>'
geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32']
gitlab_rails['db_host'] = '<replica_database_ip>'
# Prevent reconfigure from attempting to run migrations on the replica DB
gitlab_rails['auto_migrate'] = false
##
## Disable all other services that aren't needed, since we don't have a role
## that does this.
##
alertmanager['enable'] = false
consul['enable'] = false
gitaly['enable'] = false
gitlab_monitor['enable'] = false
gitlab_workhorse['enable'] = false
nginx['enable'] = false
node_exporter['enable'] = false
pgbouncer_exporter['enable'] = false
postgresql['enable'] = false
prometheus['enable'] = false
redis['enable'] = false
redis_exporter['enable'] = false
repmgr['enable'] = false
sidekiq['enable'] = false
unicorn['enable'] = false
```
following:
```ruby
##
## Enable the Geo secondary tracking database
##
geo_postgresql['enable'] = true
geo_postgresql['listen_address'] = '<ip_address_of_this_host>'
geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>'
##
## Configure FDW connection to the replica database
##
geo_secondary['db_fdw'] = true
geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>'
geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32']
gitlab_rails['db_host'] = '<replica_database_ip>'
# Prevent reconfigure from attempting to run migrations on the replica DB
gitlab_rails['auto_migrate'] = false
##
## Disable all other services that aren't needed, since we don't have a role
## that does this.
##
alertmanager['enable'] = false
consul['enable'] = false
gitaly['enable'] = false
gitlab_monitor['enable'] = false
gitlab_workhorse['enable'] = false
nginx['enable'] = false
node_exporter['enable'] = false
pgbouncer_exporter['enable'] = false
postgresql['enable'] = false
prometheus['enable'] = false
redis['enable'] = false
redis_exporter['enable'] = false
repmgr['enable'] = false
sidekiq['enable'] = false
unicorn['enable'] = false
```
After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect.
......@@ -211,50 +211,50 @@ following modifications:
1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary**
cluster, and add the following:
```ruby
##
## Enable the Geo secondary role
##
roles ['geo_secondary_role', 'application_role']
##
## Disable automatic migrations
##
gitlab_rails['auto_migrate'] = false
##
## Configure the connection to the tracking DB. And disable application
## servers from running tracking databases.
##
geo_secondary['db_host'] = '<geo_tracking_db_host>'
geo_secondary['db_password'] = '<geo_tracking_db_password>'
geo_postgresql['enable'] = false
##
## Configure connection to the streaming replica database, if you haven't
## already
##
gitlab_rails['db_host'] = '<replica_database_host>'
gitlab_rails['db_password'] = '<replica_database_password>'
##
## Configure connection to Redis, if you haven't already
##
gitlab_rails['redis_host'] = '<redis_host>'
gitlab_rails['redis_password'] = '<redis_password>'
##
## If you are using custom users not managed by Omnibus, you need to specify
## UIDs and GIDs like below, and ensure they match between servers in a
## cluster to avoid permissions issues
##
user['uid'] = 9000
user['gid'] = 9000
web_server['uid'] = 9001
web_server['gid'] = 9001
registry['uid'] = 9002
registry['gid'] = 9002
```
```ruby
##
## Enable the Geo secondary role
##
roles ['geo_secondary_role', 'application_role']
##
## Disable automatic migrations
##
gitlab_rails['auto_migrate'] = false
##
## Configure the connection to the tracking DB. And disable application
## servers from running tracking databases.
##
geo_secondary['db_host'] = '<geo_tracking_db_host>'
geo_secondary['db_password'] = '<geo_tracking_db_password>'
geo_postgresql['enable'] = false
##
## Configure connection to the streaming replica database, if you haven't
## already
##
gitlab_rails['db_host'] = '<replica_database_host>'
gitlab_rails['db_password'] = '<replica_database_password>'
##
## Configure connection to Redis, if you haven't already
##
gitlab_rails['redis_host'] = '<redis_host>'
gitlab_rails['redis_password'] = '<redis_password>'
##
## If you are using custom users not managed by Omnibus, you need to specify
## UIDs and GIDs like below, and ensure they match between servers in a
## cluster to avoid permissions issues
##
user['uid'] = 9000
user['gid'] = 9000
web_server['uid'] = 9001
web_server['gid'] = 9001
registry['uid'] = 9002
registry['gid'] = 9002
```
NOTE: **Note:**
If you had set up PostgreSQL cluster using the omnibus package and you had set
......
doc/administration/geo/replication/index.md
View file @ f55d8570
......@@ -80,8 +80,8 @@ In this diagram:
- If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](../disaster_recovery/index.md) scenarios.
- A **secondary** node performs different type of synchronizations against the **primary** node, using a special
authorization protected by JWT:
- Repositories are cloned/updated via Git over HTTPS.
- Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint.
- Repositories are cloned/updated via Git over HTTPS.
- Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint.
From the perspective of a user performing Git operations:
......@@ -107,8 +107,8 @@ The following are required to run Geo:
- An operating system that supports OpenSSH 6.9+ (needed for
[fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md))
The following operating systems are known to ship with a current version of OpenSSH:
- [CentOS](https://www.centos.org) 7.4+
- [Ubuntu](https://www.ubuntu.com) 16.04+
- [CentOS](https://www.centos.org) 7.4+
- [Ubuntu](https://www.ubuntu.com) 16.04+
- PostgreSQL 9.6+ with [FDW](https://www.postgresql.org/docs/9.6/postgres-fdw.html) support and [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication)
- Git 2.9+
......
doc/administration/geo/replication/remove_geo_node.md
View file @ f55d8570
......@@ -10,41 +10,42 @@ Once removed from the Geo admin page, you must stop and uninstall the **secondar
1. On the **secondary** node, stop GitLab:
```bash
sudo gitlab-ctl stop
```
```bash
sudo gitlab-ctl stop
```
1. On the **secondary** node, uninstall GitLab:
```bash
# Stop gitlab and remove its supervision process
sudo gitlab-ctl uninstall
```bash
# Stop gitlab and remove its supervision process
sudo gitlab-ctl uninstall
# Debian/Ubuntu
sudo dpkg --remove gitlab-ee
# Debian/Ubuntu
sudo dpkg --remove gitlab-ee
# Redhat/Centos
sudo rpm --erase gitlab-ee
```
# Redhat/Centos
sudo rpm --erase gitlab-ee
```
Once GitLab has been uninstalled from the **secondary** node, the replication slot must be dropped from the **primary** node's database as follows:
1. On the **primary** node, start a PostgreSQL console session:
```bash
sudo gitlab-psql
```
```bash
sudo gitlab-psql
```
NOTE: **Note:**
Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions.
NOTE: **Note:**
Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions.
1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`.
```sql
SELECT * FROM pg_replication_slots;
```
```sql
SELECT * FROM pg_replication_slots;
```
1. Remove the replication slot for the **secondary** node:
```sql
SELECT pg_drop_replication_slot('<name_of_slot>');
```
```sql
SELECT pg_drop_replication_slot('<name_of_slot>');
```
doc/administration/geo/replication/troubleshooting.md
View file @ f55d8570
......@@ -184,17 +184,17 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou
1. Start a PostgreSQL console session:
```sh
sudo gitlab-psql gitlabhq_production
```
```sh
sudo gitlab-psql gitlabhq_production
```
> Note that using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions.
Note: **Note:** Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions.
1. View your replication slots with:
```sql
SELECT * FROM pg_replication_slots;
```
```sql
SELECT * FROM pg_replication_slots;
```
Slots where `active` is `f` are not active.
......@@ -204,9 +204,9 @@ Slots where `active` is `f` are not active.
- If you are no longer using the slot (e.g. you no longer have Geo enabled), you can remove it with in the
PostgreSQL console session:
```sql
SELECT pg_drop_replication_slot('<name_of_extra_slot>');
```
```sql
SELECT pg_drop_replication_slot('<name_of_extra_slot>');
```
### Very large repositories never successfully synchronize on the **secondary** node
......@@ -237,82 +237,82 @@ to start again from scratch, there are a few steps that can help you:
1. Stop Sidekiq and the Geo LogCursor
It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and
wait until the current jobs to finish processing.
It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and
wait until the current jobs to finish processing.
You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM**
when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands.
You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM**
when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands.
```sh
gitlab-ctl status sidekiq
# run: sidekiq: (pid 10180) <- this is the PID you will use
kill -TSTP 10180 # change to the correct PID
```sh
gitlab-ctl status sidekiq
# run: sidekiq: (pid 10180) <- this is the PID you will use
kill -TSTP 10180 # change to the correct PID
gitlab-ctl stop sidekiq
gitlab-ctl stop geo-logcursor
```
gitlab-ctl stop sidekiq
gitlab-ctl stop geo-logcursor
```
You can watch sidekiq logs to know when sidekiq jobs processing have finished:
You can watch sidekiq logs to know when sidekiq jobs processing have finished:
```sh
gitlab-ctl tail sidekiq
```
```sh
gitlab-ctl tail sidekiq
```
1. Rename repository storage folders and create new ones
```sh
mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old
mkdir -p /var/opt/gitlab/git-data/repositories
chown git:git /var/opt/gitlab/git-data/repositories
```
```sh
mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old
mkdir -p /var/opt/gitlab/git-data/repositories
chown git:git /var/opt/gitlab/git-data/repositories
```
TIP: **Tip**
You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future
as soon as you confirmed that you don't need it anymore, to save disk space.
TIP: **Tip**
You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future
as soon as you confirmed that you don't need it anymore, to save disk space.
1. _(Optional)_ Rename other data folders and create new ones
CAUTION: **Caution**:
You may still have files on the **secondary** node that have been removed from **primary** node but
removal have not been reflected. If you skip this step, they will never be removed
from this Geo node.
CAUTION: **Caution**:
You may still have files on the **secondary** node that have been removed from **primary** node but
removal have not been reflected. If you skip this step, they will never be removed
from this Geo node.
Any uploaded content like file attachments, avatars or LFS objects are stored in a
subfolder in one of the two paths below:
Any uploaded content like file attachments, avatars or LFS objects are stored in a
subfolder in one of the two paths below:
1. /var/opt/gitlab/gitlab-rails/shared
1. /var/opt/gitlab/gitlab-rails/uploads
- /var/opt/gitlab/gitlab-rails/shared
- /var/opt/gitlab/gitlab-rails/uploads
To rename all of them:
To rename all of them:
```sh
gitlab-ctl stop
```sh
gitlab-ctl stop
mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old
mkdir -p /var/opt/gitlab/gitlab-rails/shared
mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old
mkdir -p /var/opt/gitlab/gitlab-rails/shared
mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old
mkdir -p /var/opt/gitlab/gitlab-rails/uploads
```
mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old
mkdir -p /var/opt/gitlab/gitlab-rails/uploads
```
Reconfigure in order to recreate the folders and make sure permissions and ownership
are correctly
Reconfigure in order to recreate the folders and make sure permissions and ownership
are correctly
```sh
gitlab-ctl reconfigure
```
```sh
gitlab-ctl reconfigure
```
1. Reset the Tracking Database
```sh
gitlab-rake geo:db:reset
```
```sh
gitlab-rake geo:db:reset
```
1. Restart previously stopped services
```sh
gitlab-ctl start
```
```sh
gitlab-ctl start
```
## Fixing Foreign Data Wrapper errors
......@@ -345,108 +345,106 @@ To check the configuration:
1. Enter the database console:
```sh
gitlab-geo-psql
```
```sh
gitlab-geo-psql
```
1. Check whether any tables are present. If everything is working, you
should see something like this:
```sql
gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables;
foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_n
ame
-------------------------+----------------------+-------------------------------------------------+-------------------------+-----------------
----
gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary
gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary
gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary
gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary
<snip>
```
However, if the query returns with `0 rows`, then continue onto the next steps.
```sql
gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables;
foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_name
-------------------------+----------------------+-------------------------------------------------+-------------------------+---------------------
gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary
gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary
gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary
gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary
<snip>
```
However, if the query returns with `0 rows`, then continue onto the next steps.
1. Check that the foreign server mapping is correct via `\des+`. The
results should look something like this:
```sql
gitlabhq_geo_production=# \des+
List of foreign servers
-[ RECORD 1 ]--------+------------------------------------------------------------
Name | gitlab_secondary
Owner | gitlab-psql
Foreign-data wrapper | postgres_fdw
Access privileges | "gitlab-psql"=U/"gitlab-psql" +
| gitlab_geo=U/"gitlab-psql"
Type |
Version |
FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production')
Description |
```
NOTE: **Note:** Pay particular attention to the host and port under
FDW options. That configuration should point to the Geo secondary
database.
If you need to experiment with changing the host or password, the
following queries demonstrate how:
```sql
ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>');
ALTER SERVER gitlab_secondary OPTIONS (SET port 5432);
```
If you change the host and/or port, you will also have to adjust the
following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl
reconfigure`:
- `gitlab_rails['db_host']`
- `gitlab_rails['db_port']`
```sql
gitlabhq_geo_production=# \des+
List of foreign servers
-[ RECORD 1 ]--------+------------------------------------------------------------
Name | gitlab_secondary
Owner | gitlab-psql
Foreign-data wrapper | postgres_fdw
Access privileges | "gitlab-psql"=U/"gitlab-psql" +
| gitlab_geo=U/"gitlab-psql"
Type |
Version |
FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production')
Description |
```
NOTE: **Note:** Pay particular attention to the host and port under
FDW options. That configuration should point to the Geo secondary
database.
If you need to experiment with changing the host or password, the
following queries demonstrate how:
```sql
ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>');
ALTER SERVER gitlab_secondary OPTIONS (SET port 5432);
```
If you change the host and/or port, you will also have to adjust the
following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl
reconfigure`:
- `gitlab_rails['db_host']`
- `gitlab_rails['db_port']`
1. Check that the user mapping is configured properly via `\deu+`:
```sql
gitlabhq_geo_production=# \deu+
List of user mappings
Server | User name | FDW Options
------------------+------------+--------------------------------------------------------------------------------
gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE')
(1 row)
```
```sql
gitlabhq_geo_production=# \deu+
List of user mappings
Server | User name | FDW Options
------------------+------------+--------------------------------------------------------------------------------
gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE')
(1 row)
```
Make sure the password is correct. You can test that logins work by running `psql`:
Make sure the password is correct. You can test that logins work by running `psql`:
```sh
# Connect to the tracking database as the `gitlab_geo` user
sudo \
-u git /opt/gitlab/embedded/bin/psql \
-h /var/opt/gitlab/geo-postgresql \
-p 5431 \
-U gitlab_geo \
-W \
-d gitlabhq_geo_production
```
```sh
# Connect to the tracking database as the `gitlab_geo` user
sudo \
-u git /opt/gitlab/embedded/bin/psql \
-h /var/opt/gitlab/geo-postgresql \
-p 5431 \
-U gitlab_geo \
-W \
-d gitlabhq_geo_production
```
If you need to correct the password, the following query shows how:
If you need to correct the password, the following query shows how:
```sql
ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>');
```
```sql
ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>');
```
If you change the user or password, you will also have to adjust the
following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl
reconfigure`:
If you change the user or password, you will also have to adjust the
following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl
reconfigure`:
- `gitlab_rails['db_username']`
- `gitlab_rails['db_password']`
- `gitlab_rails['db_username']`
- `gitlab_rails['db_password']`
If you are using [PgBouncer in front of the secondary
database](database.md#pgbouncer-support-optional), be sure to update
the following settings:
If you are using [PgBouncer in front of the secondary
database](database.md#pgbouncer-support-optional), be sure to update
the following settings:
- `geo_postgresql['fdw_external_user']`
- `geo_postgresql['fdw_external_password']`
- `geo_postgresql['fdw_external_user']`
- `geo_postgresql['fdw_external_password']`
#### Manual reload of FDW schema
......@@ -456,34 +454,34 @@ reload of the FDW schema. To manually reload the FDW schema:
1. On the node running the Geo tracking database, enter the PostgreSQL console via
the `gitlab_geo` user:
```sh
sudo \
-u git /opt/gitlab/embedded/bin/psql \
-h /var/opt/gitlab/geo-postgresql \
-p 5431 \
-U gitlab_geo \
-W \
-d gitlabhq_geo_production
```
```sh
sudo \
-u git /opt/gitlab/embedded/bin/psql \
-h /var/opt/gitlab/geo-postgresql \
-p 5431 \
-U gitlab_geo \
-W \
-d gitlabhq_geo_production
```
Be sure to adjust the port and hostname for your configuration. You
may be asked to enter a password.
Be sure to adjust the port and hostname for your configuration. You
may be asked to enter a password.
1. Reload the schema via:
```sql
DROP SCHEMA IF EXISTS gitlab_secondary CASCADE;
CREATE SCHEMA gitlab_secondary;
GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo;
IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary;
```
```sql
DROP SCHEMA IF EXISTS gitlab_secondary CASCADE;
CREATE SCHEMA gitlab_secondary;
GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo;
IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary;
```
1. Test that queries work:
```sql
SELECT * from information_schema.foreign_tables;
SELECT * FROM gitlab_secondary.projects limit 1;
```
```sql
SELECT * from information_schema.foreign_tables;
SELECT * FROM gitlab_secondary.projects limit 1;
```
[database-start-replication]: database.md#step-3-initiate-the-replication-process
[database-pg-replication]: database.md#postgresql-replication
......
doc/administration/geo/replication/updating_the_geo_nodes.md
View file @ f55d8570
......@@ -32,68 +32,68 @@ authentication method.
1. **[primary]** Login to your **primary** node and run:
```sh
gitlab-ctl pg-password-md5 gitlab
# Enter password: <your_password_here>
# Confirm password: <your_password_here>
# fca0b89a972d69f00eb3ec98a5838484
```
```sh
gitlab-ctl pg-password-md5 gitlab
# Enter password: <your_password_here>
# Confirm password: <your_password_here>
# fca0b89a972d69f00eb3ec98a5838484
```
Copy the generated hash and edit `/etc/gitlab/gitlab.rb`:
Copy the generated hash and edit `/etc/gitlab/gitlab.rb`:
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
# Every node that runs Unicorn or Sidekiq needs to have the database
# password specified as below. If you have a high-availability setup, this
# must be present in all application nodes.
gitlab_rails['db_password'] = '<your_password_here>'
```
# Every node that runs Unicorn or Sidekiq needs to have the database
# password specified as below. If you have a high-availability setup, this
# must be present in all application nodes.
gitlab_rails['db_password'] = '<your_password_here>'
```
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
```ruby
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this
```
```ruby
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this
```
1. **[primary]** Reconfigure and restart:
```sh
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
```
```sh
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
```
1. **[secondary]** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`:
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
```ruby
# Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab`
postgresql['sql_user_password'] = '<md5_hash_of_your_password>'
# Every node that runs Unicorn or Sidekiq needs to have the database
# password specified as below. If you have a high-availability setup, this
# must be present in all application nodes.
gitlab_rails['db_password'] = '<your_password_here>'
# Every node that runs Unicorn or Sidekiq needs to have the database
# password specified as below. If you have a high-availability setup, this
# must be present in all application nodes.
gitlab_rails['db_password'] = '<your_password_here>'
# Enable Foreign Data Wrapper
geo_secondary['db_fdw'] = true
# Enable Foreign Data Wrapper
geo_secondary['db_fdw'] = true
# Secondary address in CIDR format, for example '5.6.7.8/32'
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']
```
# Secondary address in CIDR format, for example '5.6.7.8/32'
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']
```
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
Still in the configuration file, locate and remove the `trust_auth_cidr_address`:
```ruby
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this
```
```ruby
postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this
```
1. **[secondary]** Reconfigure and restart:
```sh
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
```
```sh
sudo gitlab-ctl reconfigure
sudo gitlab-ctl restart
```
## Upgrading to GitLab 10.5
......@@ -171,9 +171,9 @@ the now-unused SSH keys from your secondaries, as they may cause problems if the
1. **[secondary]** Login to **all** your **secondary** nodes and run:
```ruby
sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub
```
```ruby
sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub
```
### Hashed Storage
......@@ -236,12 +236,12 @@ instructions below.
When in doubt, it does not hurt to do a resync. The easiest way to do this in
Omnibus is the following:
1. Make sure you have Omnibus GitLab on the **primary** server.
1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database.
1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`.
1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations.
1. Install GitLab on the **secondary** server.
1. Re-run the [database replication process][database-replication].
1. Make sure you have Omnibus GitLab on the **primary** server.
1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database.
1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`.
1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations.
1. Install GitLab on the **secondary** server.
1. Re-run the [database replication process][database-replication].
## Special update notes for 9.0.x
......@@ -262,16 +262,16 @@ is prepended with the relevant node for better clarity:
1. **[secondary]** Login to **all** your **secondary** nodes and stop all services:
```ruby
sudo gitlab-ctl stop
```
```ruby
sudo gitlab-ctl stop
```
1. **[secondary]** Make a backup of the `recovery.conf` file on **all**
**secondary** nodes to preserve PostgreSQL's credentials:
```sh
sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/
```
```sh
sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/
```
1. **[primary]** Update the **primary** node to GitLab 9.0 following the
[regular update docs][update]. At the end of the update, the **primary** node
......@@ -281,136 +281,133 @@ is prepended with the relevant node for better clarity:
stop all services except `postgresql` as we will use it to re-initialize the
**secondary** node's database:
```sh
sudo gitlab-ctl stop
sudo gitlab-ctl start postgresql
```
```sh
sudo gitlab-ctl stop
sudo gitlab-ctl start postgresql
```
1. **[secondary]** Run the following steps on each of the **secondary** nodes:
1. **[secondary]** Stop all services:
1. **[secondary]** Stop all services:
```sh
sudo gitlab-ctl stop
```
```sh
sudo gitlab-ctl stop
```
1. **[secondary]** Prevent running database migrations:
1. **[secondary]** Prevent running database migrations:
```sh
sudo touch /etc/gitlab/skip-auto-migrations
```
```sh
sudo touch /etc/gitlab/skip-auto-migrations
```
1. **[secondary]** Move the old database to another directory:
1. **[secondary]** Move the old database to another directory:
```sh
sudo mv /var/opt/gitlab/postgresql{,.bak}
```
```sh
sudo mv /var/opt/gitlab/postgresql{,.bak}
```
1. **[secondary]** Update to GitLab 9.0 following the [regular update docs][update].
At the end of the update, the node will be running with PostgreSQL 9.6.
1. **[secondary]** Update to GitLab 9.0 following the [regular update docs][update].
At the end of the update, the node will be running with PostgreSQL 9.6.
1. **[secondary]** Make sure all services are up:
1. **[secondary]** Make sure all services are up:
```sh
sudo gitlab-ctl start
```
```sh
sudo gitlab-ctl start
```
1. **[secondary]** Reconfigure GitLab:
1. **[secondary]** Reconfigure GitLab:
```sh
sudo gitlab-ctl reconfigure
```
```sh
sudo gitlab-ctl reconfigure
```
1. **[secondary]** Run the PostgreSQL upgrade command:
1. **[secondary]** Run the PostgreSQL upgrade command:
```sh
sudo gitlab-ctl pg-upgrade
```
```sh
sudo gitlab-ctl pg-upgrade
```
1. **[secondary]** See the stored credentials for the database that you will
need to re-initialize the replication:
1. **[secondary]** See the stored credentials for the database that you will
need to re-initialize the replication:
```sh
sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf
```
```sh
sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf
```
1. **[secondary]** Create the `replica.sh` script as described in the
[database configuration document][database-source-replication].
1. **[secondary]** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the
embedded paths if necessary:
1. 1. **[secondary]** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the
embedded paths if necessary:
```
#!/bin/bash
```
#!/bin/bash
PORT="5432"
USER="gitlab_replicator"
echo ---------------------------------------------------------------
echo WARNING: Make sure this script is run from the secondary server
echo ---------------------------------------------------------------
echo
echo Enter the IP or FQDN of the primary PostgreSQL server
read HOST
echo Enter the password for $USER@$HOST
read -s PASSWORD
echo Enter the required sslmode
read SSLMODE
PORT="5432"
USER="gitlab_replicator"
echo ---------------------------------------------------------------
echo WARNING: Make sure this script is run from the secondary server
echo ---------------------------------------------------------------
echo
echo Enter the IP or FQDN of the primary PostgreSQL server
read HOST
echo Enter the password for $USER@$HOST
read -s PASSWORD
echo Enter the required sslmode
read SSLMODE
echo Stopping PostgreSQL and all GitLab services
sudo service gitlab stop
sudo service postgresql stop
echo Stopping PostgreSQL and all GitLab services
sudo service gitlab stop
sudo service postgresql stop
echo Backing up postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/
echo Backing up postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/
echo Cleaning up old cluster directory
sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data
echo Cleaning up old cluster directory
sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data
echo Starting base backup as the replicator user
echo Enter the password for $USER@$HOST
sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P
echo Starting base backup as the replicator user
echo Enter the password for $USER@$HOST
sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P
echo Writing recovery.conf file
sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_
standby_mode = 'on'
primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE'
_EOF1_
"
echo Writing recovery.conf file
sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_
standby_mode = 'on'
primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE'
_EOF1_
"
echo Restoring postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/
echo Restoring postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/
echo Starting PostgreSQL
sudo service postgresql start
```
echo Starting PostgreSQL
sudo service postgresql start
```
1. **[secondary]** Run the recovery script using the credentials from the
previous step:
1. **[secondary]** Run the recovery script using the credentials from the
previous step:
```sh
sudo bash /tmp/replica.sh
```
```sh
sudo bash /tmp/replica.sh
```
1. **[secondary]** Reconfigure GitLab:
1. **[secondary]** Reconfigure GitLab:
```sh
sudo gitlab-ctl reconfigure
```
```sh
sudo gitlab-ctl reconfigure
```
1. **[secondary]** Start all services:
1. **[secondary]** Start all services:
```sh
sudo gitlab-ctl start
```
```sh
sudo gitlab-ctl start
```
1. **[secondary]** Repeat the steps for the remaining **secondary** nodes.
1. **[secondary]** Repeat the steps for the remaining **secondary** nodes.
1. **[primary]** After all **secondary** nodes are updated, start all services in
**primary** node:
```sh
sudo gitlab-ctl start
```
```sh
sudo gitlab-ctl start
```
## Check status after updating
......@@ -419,9 +416,9 @@ everything is working correctly:
1. Run the Geo raketask on all nodes, everything should be green:
```sh
sudo gitlab-rake gitlab:geo:check
```
```sh
sudo gitlab-rake gitlab:geo:check
```
1. Check the **primary** node's Geo dashboard for any errors.
1. Test the data replication by pushing code to the **primary** node and see if it
......@@ -435,9 +432,9 @@ and it is required since 10.0.
1. Run database migrations on tracking database:
```sh
sudo gitlab-rake geo:db:migrate
```
```sh
sudo gitlab-rake geo:db:migrate
```
1. Repeat this step for each **secondary** node.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7