gitlab.md 9.22 KB
Newer Older
1
# Configuring GitLab Scaling and High Availability
Drew Blessing's avatar
Drew Blessing committed
2 3

> **Note:** There is some additional configuration near the bottom for
4
  additional GitLab application servers. It's important to read and understand
Drew Blessing's avatar
Drew Blessing committed
5 6 7 8 9
  these additional steps before proceeding with GitLab installation.

1. If necessary, install the NFS client utility packages using the following
   commands:

10 11 12
   ```
   # Ubuntu/Debian
   apt-get install nfs-common
Drew Blessing's avatar
Drew Blessing committed
13

14 15 16
   # CentOS/Red Hat
   yum install nfs-utils nfs-utils-lib
   ```
Drew Blessing's avatar
Drew Blessing committed
17 18 19 20 21 22

1. Specify the necessary NFS shares. Mounts are specified in
   `/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose
   to configure your NFS server. See [NFS documentation](nfs.md) for the various
   options. Here is an example snippet to add to `/etc/fstab`:

23 24 25 26 27 28 29
   ```
   10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
   10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
   10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
   10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
   10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2
   ```
Drew Blessing's avatar
Drew Blessing committed
30 31 32 33

1. Create the shared directories. These may be different depending on your NFS
   mount locations.

34 35 36
   ```
   mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data
   ```
Drew Blessing's avatar
Drew Blessing committed
37 38 39 40 41 42 43 44 45

1. Download/install GitLab Omnibus using **steps 1 and 2** from
   [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other
   steps on the download page.
1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration.
   Be sure to change the `external_url` to match your eventual GitLab front-end
   URL. Depending your the NFS configuration, you may need to change some GitLab
   data locations. See [NFS documentation](nfs.md) for `/etc/gitlab/gitlab.rb`
   configuration values for various scenarios. The example below assumes you've
46 47
   added NFS mounts in the default data locations. Additionally the UID and GIDs
   given are just examples and you should configure with your preferred values.
48

49 50
   ```ruby
   external_url 'https://gitlab.example.com'
Drew Blessing's avatar
Drew Blessing committed
51

52 53
   # Prevent GitLab from starting if NFS data mounts are not available
   high_availability['mountpoint'] = '/var/opt/gitlab/git-data'
54

55 56 57
   # Disable components that will not be on the GitLab application server
   roles ['application_role']
   nginx['enable'] = true
58

59 60 61 62 63
   # PostgreSQL connection details
   gitlab_rails['db_adapter'] = 'postgresql'
   gitlab_rails['db_encoding'] = 'unicode'
   gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server
   gitlab_rails['db_password'] = 'DB password'
64

65 66 67 68
   # Redis connection details
   gitlab_rails['redis_port'] = '6379'
   gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server
   gitlab_rails['redis_password'] = 'Redis Password'
Evan Read's avatar
Evan Read committed
69

70 71 72 73 74 75 76 77
   # Ensure UIDs and GIDs match between servers for permissions via NFS
   user['uid'] = 9000
   user['gid'] = 9000
   web_server['uid'] = 9001
   web_server['gid'] = 9001
   registry['uid'] = 9002
   registry['gid'] = 9002
   ```
78

79 80
1. [Enable monitoring](#enable-monitoring)

81 82 83 84 85 86 87 88 89 90 91 92 93 94
   > **Note:** To maintain uniformity of links across HA clusters, the `external_url`
   on the first application server as well as the additional application
   servers should point to the external url that users will use to access GitLab.
   In a typical HA setup, this will be the url of the load balancer which will
   route traffic to all GitLab application servers in the HA cluster.
   >
   > **Note:** When you specify `https` in the `external_url`, as in the example
   above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If
   certificates are not present, Nginx will fail to start. See
   [Nginx documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https)
   for more information.
   >
   > **Note:** It is best to set the `uid` and `gid`s prior to the initial reconfigure
   of GitLab. Omnibus will not recursively `chown` directories if set after the initial reconfigure.
Drew Blessing's avatar
Drew Blessing committed
95

96
## First GitLab application server
Drew Blessing's avatar
Drew Blessing committed
97

98 99
As a final step, run the setup rake task **only on** the first GitLab application server.
Do not run this on additional application servers.
Drew Blessing's avatar
Drew Blessing committed
100 101

1. Initialize the database by running `sudo gitlab-rake gitlab:setup`.
102
1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
Drew Blessing's avatar
Drew Blessing committed
103 104 105 106

> **WARNING:** Only run this setup task on **NEW** GitLab instances because it
  will wipe any existing data.

107
## Extra configuration for additional GitLab application servers
Drew Blessing's avatar
Drew Blessing committed
108

109 110
Additional GitLab servers (servers configured **after** the first GitLab server)
need some extra configuration.
Drew Blessing's avatar
Drew Blessing committed
111 112

1. Configure shared secrets. These values can be obtained from the primary
113 114 115
   GitLab server in `/etc/gitlab/gitlab-secrets.json`. Copy this file to the
   secondary servers **prior to** running the first `reconfigure` in the steps
   above.
Drew Blessing's avatar
Drew Blessing committed
116

117 118 119 120 121 122
   ```ruby
   gitlab_shell['secret_token'] = 'fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860'
   gitlab_rails['otp_key_base'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa'
   gitlab_rails['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d'
   gitlab_rails['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964'
   ```
Drew Blessing's avatar
Drew Blessing committed
123

124
1. Run `touch /etc/gitlab/skip-auto-reconfigure` to prevent database migrations
Drew Blessing's avatar
Drew Blessing committed
125 126 127
   from running on upgrade. Only the primary GitLab application server should
   handle migrations.

128
1. **Recommended** Configure host keys. Copy the contents (primary and public keys) of `/etc/ssh/` on
Davin Walker's avatar
Davin Walker committed
129 130 131 132
   the primary application server to `/etc/ssh` on all secondary servers. This
   prevents false man-in-the-middle-attack alerts when accessing servers in your
   High Availability cluster behind a load balancer.

133 134
1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.

135 136 137 138 139 140
## Enable Monitoring

> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.

If you enable Monitoring, it must be enabled on **all** GitLab servers.

141 142
1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z`

143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162
1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration:

   ```ruby
   # Enable service discovery for Prometheus
   consul['enable'] = true
   consul['monitoring_service_discovery'] =  true

   # Replace placeholders
   # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
   # with the addresses of the Consul server nodes
   consul['configuration'] = {
      retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
   }

   # Set the network addresses that the exporters will listen on
   node_exporter['listen_address'] = '0.0.0.0:9100'
   gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229'
   sidekiq['listen_address'] = "0.0.0.0"
   unicorn['listen'] = '0.0.0.0'

163 164 165 166 167
   # Add the monitoring node's IP address to the monitoring whitelist and allow it to
   # scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with
   # the address and/or subnets gathered from the monitoring node(s).
   gitlab_rails['monitoring_whitelist'] = ['monitoring.gitlab.example.com', '127.0.0.0/8']
   nginx['status']['options']['allow'] = ['monitoring.gitlab.example.com', '127.0.0.0/8']
168 169 170 171 172 173 174 175
   ```

1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.

> **Warning:** After changing `unicorn['listen']` in `gitlab.rb`, and running `sudo gitlab-ctl reconfigure`,
  it can take an extended period of time for unicorn to complete reloading after receiving a `HUP`.
  For more information, see the [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4401).

Drew Blessing's avatar
Drew Blessing committed
176 177 178 179 180 181 182 183 184 185 186 187 188
## Troubleshooting

- `mount: wrong fs type, bad option, bad superblock on`

You have not installed the necessary NFS client utilities. See step 1 above.

- `mount: mount point /var/opt/gitlab/... does not exist`

This particular directory does not exist on the NFS server. Ensure
the share is exported and exists on the NFS server and try to remount.

---

189 190 191 192 193 194 195 196
## Upgrading GitLab HA

GitLab HA installations can be upgraded with no downtime, but the
upgrade process must be carefully coordinated to avoid failures. See the
[Omnibus GitLab multi-node upgrade
document](https://docs.gitlab.com/omnibus/update/#multi-node--ha-deployment)
for more details.

Drew Blessing's avatar
Drew Blessing committed
197 198 199 200 201 202
Read more on high-availability configuration:

1. [Configure the database](database.md)
1. [Configure Redis](redis.md)
1. [Configure NFS](nfs.md)
1. [Configure the load balancers](load_balancer.md)