@@ -77,6 +77,7 @@ check the [Troubleshooting section](#troubleshooting) before proceeding.
### Checking cluster membership
To see which nodes are part of the cluster, run the following on any member in the cluster
```
# /opt/gitlab/embedded/bin/consul members
Node Address Status Type Build Protocol DC
...
...
@@ -112,7 +113,6 @@ You will see messages like the following in `gitlab-ctl tail consul` output if y
2017-09-25_19:53:41.74356 2017/09/25 19:53:41 [ERR] agent: failed to sync remote state: No cluster leader
```
To fix this:
1. Pick an address on each node that all of the other nodes can reach this node through.
...
...
@@ -124,6 +124,7 @@ To fix this:
bind_addr: 'IP ADDRESS'
}
```
1. Run `gitlab-ctl reconfigure`
If you still see the errors, you may have to [erase the consul database and reinitialize](#recreate-from-scratch) on the affected node.
...
...
@@ -150,6 +151,7 @@ To fix this:
bind_addr: 'IP ADDRESS'
}
```
1. Run `gitlab-ctl reconfigure`
### Outage recovery
...
...
@@ -157,6 +159,7 @@ To fix this:
If you lost enough server agents in the cluster to break quorum, then the cluster is considered failed, and it will not function without manual intervenetion.
#### Recreate from scratch
By default, GitLab does not store anything in the consul cluster that cannot be recreated. To erase the consul database and reinitialize
```
...
...
@@ -168,4 +171,5 @@ By default, GitLab does not store anything in the consul cluster that cannot be
After this, the cluster should start back up, and the server agents rejoin. Shortly after that, the client agents should rejoin as well.
#### Recover a failed cluster
If you have taken advantage of consul to store other data, and want to restore the failed cluster, please follow the [Consul guide](https://www.consul.io/docs/guides/outage.html) to recover a failed cluster.
@@ -288,7 +288,6 @@ Make sure you install the necessary dependencies from step 1,
add GitLab package repository from step 2.
When installing the GitLab package, do not supply `EXTERNAL_URL` value.
#### Configuring the Database nodes
1. Make sure to [configure the Consul nodes](consul.md).
...
...
@@ -343,6 +342,7 @@ When installing the GitLab package, do not supply `EXTERNAL_URL` value.
to `/etc/gitlab/gitlab.rb`. In addition, append the following configuration
to inform gitlab-ctl that they are standby nodes initially and it need not
attempt to register them as primary node
```
# HA setting to specify if a node should attempt to be master on initialization
repmgr['master_on_initialization'] = false
...
...
@@ -977,7 +977,7 @@ If you enable Monitoring, it must be enabled on **all** database servers.
## Troubleshooting
### Consul and PostgreSQL changes not taking effect.
### Consul and PostgreSQL changes not taking effect
Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading.
- For non-NetApp devices, disable NFSv4 `idmapping` by performing opposite of [enable NFSv4 idmapper](https://wiki.archlinux.org/index.php/NFS#Enabling_NFSv4_idmapping)
- For non-NetApp devices, disable NFSv4 `idmapping` by performing opposite of [enable NFSv4 idmapper](https://wiki.archlinux.org/index.php/NFS#Enabling_NFSv4_idmapping)