Skip to content

G
gitlab-ce
Commit a9f7ce47 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis
Browse files
Options

Merge branch 'doc/sentinel' of gitlab.com:gitlab-org/gitlab-ce into doc/sentinel

parents 95f6cf33 850405b4
Hide whitespace changes
Inline Side-by-side
Showing with 1107 additions and 330 deletions
doc/administration/high_availability/README.md
View file @ a9f7ce47
...@@ -7,19 +7,10 @@ highly available. ...@@ -7,19 +7,10 @@ highly available.
## Architecture ## Architecture
### Active/Passive There are two kinds of setups:
For pure high-availability/failover with no scaling you can use an
active/passive configuration. This utilizes DRBD (Distributed Replicated
Block Device) to keep all data in sync. DRBD requires a low latency link to
remain in sync. It is not advisable to attempt to run DRBD between data centers
or in different cloud availability zones.
Components/Servers Required: - active/active
- active/passive
- 2 servers/virtual machines (one active/one passive)
![Active/Passive HA Diagram](../img/high_availability/active-passive-diagram.png)
### Active/Active ### Active/Active
...@@ -28,12 +19,24 @@ user requests simultaneously. The database, Redis, and GitLab application are ...@@ -28,12 +19,24 @@ user requests simultaneously. The database, Redis, and GitLab application are
all deployed on separate servers. The configuration is **only** highly-available all deployed on separate servers. The configuration is **only** highly-available
if the database, Redis and storage are also configured as such. if the database, Redis and storage are also configured as such.
![Active/Active HA Diagram](../img/high_availability/active-active-diagram.png) Follow the steps below to configure an active/active setup:
**Steps to configure active/active:**
1. [Configure the database](database.md) 1. [Configure the database](database.md)
1. [Configure Redis](redis.md) 1. [Configure Redis](redis.md)
1. [Configure NFS](nfs.md) 1. [Configure NFS](nfs.md)
1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the GitLab application servers](gitlab.md)
1. [Configure the load balancers](load_balancer.md) 1. [Configure the load balancers](load_balancer.md)
![Active/Active HA Diagram](../img/high_availability/active-active-diagram.png)
### Active/Passive
For pure high-availability/failover with no scaling you can use an
active/passive configuration. This utilizes DRBD (Distributed Replicated
Block Device) to keep all data in sync. DRBD requires a low latency link to
remain in sync. It is not advisable to attempt to run DRBD between data centers
or in different cloud availability zones.
Components/Servers Required: 2 servers/virtual machines (one active/one passive)
![Active/Passive HA Diagram](../img/high_availability/active-passive-diagram.png)
doc/administration/high_availability/redis.md
View file @ a9f7ce47
# Configuring Redis for GitLab HA # Configuring Redis for GitLab HA
You can choose to install and manage Redis yourself, or you can use the one High Availability with [Redis] is possible using a **Master** x **Slave**
that comes bundled with GitLab Omnibus packages. topology with a [Redis Sentinel][sentinel] service to watch and automatically
start the failover procedure.
> **Note:** Redis does not require authentication by default. See You can choose to install and manage Redis and Sentinel yourself, use
a hosted cloud solution or you can use the one that comes bundled with
Omnibus GitLab packages.
> **Notes:**
- Redis requires authentication for High Availability. See
[Redis Security](http://redis.io/topics/security) documentation for more [Redis Security](http://redis.io/topics/security) documentation for more
information. We recommend using a combination of a Redis password and tight information. We recommend using a combination of a Redis password and tight
firewall rules to secure your Redis service. firewall rules to secure your Redis service.
- You are highly encouraged to read the [Redis Sentinel][sentinel] documentation
before configuring Redis HA with GitLab to fully understand the topology and
architecture.
- This is the documentation for the Omnibus GitLab packages. For installations
from source, follow the [Redis HA source installation](redis_source.md) guide.
- Redis Sentinel daemon is bundled with Omnibus GitLab Enterprise Edition only.
For configuring Sentinel with the Omnibus GitLab Community Edition and
installations from source, read the
[Available configuration setups](#available-configuration-setups) section
below.
<!-- START doctoc generated TOC please keep comment here to allow auto update --> <!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> <!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents** **Table of Contents**
- [Configure your own Redis server](#configure-your-own-redis-server) - [Overview](#overview)
- [Configure Redis using Omnibus](#configure-redis-using-omnibus) - [High Availability with Sentinel](#high-availability-with-sentinel)
- [Experimental Redis Sentinel support](#experimental-redis-sentinel-support) - [Recommended setup](#recommended-setup)
- [Redis setup](#redis-setup) - [Redis setup overview](#redis-setup-overview)
- [Source install](#source-install) - [Sentinel setup overview](#sentinel-setup-overview)
- [Omnibus Install](#omnibus-install) - [Available configuration setups](#available-configuration-setups)
- [Troubleshooting Replication](#troubleshooting-replication) - [Configuring Redis HA](#configuring-redis-ha)
- [Sentinel](#sentinel) - [Prerequisites](#prerequisites)
- [Sentinel setup (Community Edition)](#sentinel-setup-community-edition) - [Step 1. Configuring the master Redis instance](#step-1-configuring-the-master-redis-instance)
- [Sentinel setup (EE Only)](#sentinel-setup-ee-only) - [Step 2. Configuring the slave Redis instances](#step-2-configuring-the-slave-redis-instances)
- [GitLab setup](#gitlab-setup) - [Step 3. Configuring the Redis Sentinel instances](#step-3-configuring-the-redis-sentinel-instances)
- [Sentinel troubleshooting](#sentinel-troubleshooting) - [Step 4. Configuring the GitLab application](#step-4-configuring-the-gitlab-application)
- [Omnibus install](#omnibus-install) - [Switching from an existing single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha)
- [Source install](#source-install-1) - [Example of a minimal configuration with 1 master, 2 slaves and 3 Sentinels](#example-of-a-minimal-configuration-with-1-master-2-slaves-and-3-sentinels)
- [Example configuration for Redis master and Sentinel 1](#example-configuration-for-redis-master-and-sentinel-1)
- [Example configuration for Redis slave 1 and Sentinel 2](#example-configuration-for-redis-slave-1-and-sentinel-2)
- [Example configuration for Redis slave 2 and Sentinel 3](#example-configuration-for-redis-slave-2-and-sentinel-3)
- [Example configuration for the GitLab application](#example-configuration-for-the-gitlab-application)
- [Advanced configuration](#advanced-configuration)
- [Control running services](#control-running-services)
- [Troubleshooting](#troubleshooting)
- [Troubleshooting Redis replication](#troubleshooting-redis-replication)
- [Troubleshooting Sentinel](#troubleshooting-sentinel)
- [Changelog](#changelog)
- [Experimental Redis Sentinel support](#experimental-redis-sentinel-support)
- [Further reading](#further-reading)
<!-- END doctoc generated TOC please keep comment here to allow auto update --> <!-- END doctoc generated TOC please keep comment here to allow auto update -->
## Configure your own Redis server ## Overview
If you're hosting GitLab on a cloud provider, you can optionally use a Before diving into the details of setting up Redis and Redis Sentinel for HA,
managed service for Redis. For example, AWS offers a managed ElastiCache service make sure you read this Overview section to better understand how the components
that runs Redis. are tied together.
## Configure Redis using Omnibus You need at least `3` independent machines: physical, or VMs running into
distinct physical machines. It is essential that all master and slaves Redis
instances run in different machines. If you fail to provision the machines in
that specific way, any issue with the shared environment can bring your entire
setup down.
If you don't want to bother setting up your own Redis server, you can use the It is OK to run a Sentinel along with a master or slave Redis instance.
one bundled with Omnibus. In this case, you should disable all services except No more than one Sentinel in the same machine though.
Redis.
1. Download/install GitLab Omnibus using **steps 1 and 2** from You also need to take in consideration the underlying network topology,
[GitLab downloads](https://about.gitlab.com/downloads). Do not complete other making sure you have redundant connectivity between Redis / Sentinel and
steps on the download page. GitLab instances, otherwise the networks will become a single point of
1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. failure.
Be sure to change the `external_url` to match your eventual GitLab front-end
URL:
```ruby Make sure that you read this document once as a whole before configuring the
external_url 'https://gitlab.example.com' components below.
# Disable all services except Redis
redis['enable'] = true
bootstrap['enable'] = false
nginx['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
postgresql['enable'] = false
gitlab_rails['enable'] = false
gitlab_workhorse['enable'] = false
mailroom['enable'] = false
# Redis configuration
redis['port'] = 6379
redis['bind'] = '0.0.0.0'
# If you wish to use Redis authentication (recommended) ### High Availability with Sentinel
redis['password'] = 'redis-password-goes-here'
```
1. Run `sudo touch /etc/gitlab/skip-auto-migrations` to prevent database migrations >**Notes:**
from running on upgrade. Only the primary GitLab application server should - Starting with GitLab `8.11`, you can configure a list of Redis Sentinel
handle migrations. servers that will monitor a group of Redis servers to provide failover support.
- Starting with GitLab `8.14`, the Omnibus GitLab Enterprise Edition package
comes with Redis Sentinel daemon built-in.
1. Run `sudo gitlab-ctl reconfigure` to install and configure Redis. High Availability with Redis requires a few things:
> **Note**: This `reconfigure` step will result in some errors. - Multiple Redis instances
That's OK - don't be alarmed. - Run Redis in a **Master** x **Slave** topology
- Multiple Sentinel instances
- Application support and visibility to all Sentinel and Redis instances
## Experimental Redis Sentinel support Redis Sentinel can handle the most important tasks in an HA environment and that's
to help keep servers online with minimal to no downtime. Redis Sentinel:
> [Introduced][ce-1877] in GitLab 8.11, improved in 8.13. - Monitors **Master** and **Slaves** instances to see if they are available
- Promotes a **Slave** to **Master** when the **Master** fails
- Demotes a **Master** to **Slave** when the failed **Master** comes back online
(to prevent data-partitioning)
- Can be queried by clients to always connect to the current **Master** server
Since GitLab 8.11, you can configure a list of Redis Sentinel servers that When a **Master** fails to respond, it's the client's responsibility to handle
will monitor a group of Redis servers to provide you with a standard failover timeout and reconnect (querying a **Sentinel** for a new **Master**).
support.
To get a better understanding on how to correctly setup Sentinel, please read To get a better understanding on how to correctly setup Sentinel, please read
the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as the [Redis Sentinel documentation](http://redis.io/topics/sentinel) first, as
failing to configure it correctly can lead to data loss. failing to configure it correctly can lead to data loss or can bring your
whole cluster down, invalidating the failover effort.
### Recommended setup
For a minimal setup, you will install the Omnibus GitLab package in `3`
**independent** machines, both with **Redis** and **Sentinel**:
Redis Sentinel can handle the most important tasks in a HA environment to help - Redis Master + Sentinel
keep servers online with minimal to no downtime: - Redis Slave + Sentinel
- Redis Slave + Sentinel
- Monitors master and slave instances to see if they are available If you are not sure or don't understand why and where the amount of nodes come
- Promote a slave to master when the master fails. from, read [Redis setup overview](#redis-setup-overview) and
- Demote a master to slave when failed master comes back online (to prevent [Sentinel setup overview](#sentinel-setup-overview).
data-partitioning).
- Can be queried by clients to always connect to the correct master server.
There is currently one exception to the Sentinel support: `mail_room`, the For a recommended setup that can resist more failures, you will install
component that processes incoming emails. It doesn't support Sentinel yet, but the Omnibus GitLab package in `5` **independent** machines, both with
we hope to integrate a future release that does support it soon. **Redis** and **Sentinel**:
The configuration consists of three parts: - Redis Master + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
- Setup Redis Master and Slave nodes ### Redis setup overview
- Setup Sentinel nodes
- Setup GitLab
> **IMPORTANT**: You need at least 3 independent machines: physical, or VMs You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they
running into distinct physical machines. If you fail to provision the need to be each in a independent machine (see explanation above).
machines in that specific way, any issue with the shared environment can
bring your entire setup down.
Read carefully how to configure those components below. You can have additional Redis nodes, that will help survive a situation
where more nodes goes down. Whenever there is only `2` nodes online, a failover
will not be initiated.
### Redis setup As an example, if you have `6` Redis nodes, a maximum of `3` can be
simultaneously down.
You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they need to Please note that there are different requirements for Sentinel nodes.
be each in a independent machine (see explanation above). If you host them in the same Redis machines, you may need to take
that restrictions into consideration when calculating the amount of
nodes to be provisioned. See [Sentinel setup overview](#sentinel-setup-overview)
documentation for more information.
They should be configured the same way and with similar server specs, as All Redis nodes should be configured the same way and with similar server specs, as
in a failover situation, any `Slave` can be elected as the new `Master` by in a failover situation, any **Slave** can be promoted as the new **Master** by
the Sentinel servers. the Sentinel servers.
With Sentinel, you must define a password to protect the access as both The replication requires authentication, so you need to define a password to
Sentinel instances and other redis instances should be able to talk to protect all Redis nodes and the Sentinels. They will all share the same
password, and all instances must be able to talk to
each other over the network. each other over the network.
You'll need to define both `requirepass` and `masterauth` in all ### Sentinel setup overview
nodes because they can be re-configured at any time by the Sentinels
during a failover, and change it's status as `Master` or `Slave`.
Initial `Slave` nodes will have in `redis.conf` an additional `slaveof` line Sentinels watch both other Sentinels and Redis nodes. Whenever a Sentinel
pointing to the initial `Master`. detects that a Redis node is not responding, it will announce that to the
other Sentinels. They have to reach the **quorum**, that is the minimum amount
of Sentinels that agrees a node is down, in order to be able to start a failover.
#### Source install Whenever the **quorum** is met, the **majority** of all known Sentinel nodes
need to be available and reachable, so that they can elect the Sentinel **leader**
who will take all the decisions to restore the service availability by:
**Master Redis instance** - Promoting a new **Master**
- Reconfiguring the other **Slaves** and make them point to the new **Master**
- Announce the new **Master** to every other Sentinel peer
- Reconfigure the old **Master** and demote to **Slave** when it comes back online
You need to make the following changes in `redis.conf`: You must have at least `3` Redis Sentinel servers, and they need to
be each in a independent machine (that are believed to fail independently),
ideally in different geographical areas.
1. Define a `bind` address pointing to a local IP that your other machines You can configure them in the same machines where you've configured the other
can reach you. If you really need to bind to an external acessible IP, make Redis servers, but understand that if a whole node goes down, you loose both
sure you add extra firewall rules to prevent unauthorized access: a Sentinel and a Redis instance.
```conf The number of sentinels should ideally always be an **odd** number, for the
# By default, if no "bind" configuration directive is specified, Redis listens consensus algorithm to be effective in the case of a failure.
# for connections from all the network interfaces available on the server.
# It is possible to listen to just one or multiple selected interfaces using
# the "bind" configuration directive, followed by one or more IP addresses.
#
# Examples:
#
# bind 192.168.1.100 10.0.0.1
# bind 127.0.0.1 ::1
bind 0.0.0.0 # This will bind to all interfaces
```
1. Define a `port` to force redis to listin on TCP so other machines can In a `3` nodes topology, you can only afford `1` Sentinel node going down.
connect to it: Whenever the **majority** of the Sentinels goes down, the network partition
protection prevents destructive actions and a failover **will not be started**.
```conf Here are some examples:
# Accept connections on the specified port, default is 6379 (IANA #815344).
# If port 0 is specified Redis will not listen on a TCP socket.
port 6379
```
1. Set up password authentication (use the same password in all nodes) - With `5` or `6` sentinels, a maximum of `2` can go down for a failover begin.
- With `7` sentinels, a maximum of `3` nodes can go down.
```conf The **Leader** election can sometimes fail the voting round when **consensus**
requirepass "redis-password-goes-here" is not achieved (see the odd number of nodes requirement above). In that case,
masterauth "redis-password-goes-here" a new attempt will be made after the amount of time defined in
``` `sentinel['failover_timeout']` (in milliseconds).
1. Restart the Redis services for the changes to take effect. >**Note:**
We will see where `sentinel['failover_timeout']` is defined later.
The `failover_timeout` variable has a lot of different use cases. According to
the official documentation:
- The time needed to re-start a failover after a previous failover was
already tried against the same master by a given Sentinel, is two
times the failover timeout.
- The time needed for a slave replicating to a wrong master according
to a Sentinel current configuration, to be forced to replicate
with the right master, is exactly the failover timeout (counting since
the moment a Sentinel detected the misconfiguration).
- The time needed to cancel a failover that is already in progress but
did not produced any configuration change (SLAVEOF NO ONE yet not
acknowledged by the promoted slave).
- The maximum time a failover in progress waits for all the slaves to be
reconfigured as slaves of the new master. However even after this time
the slaves will be reconfigured by the Sentinels anyway, but not with
the exact parallel-syncs progression as specified.
### Available configuration setups
Based on your infrastructure setup and how you have installed GitLab, there are
multiple ways to configure Redis HA. Omnibus GitLab packages have Redis and/or
Redis Sentinel bundled with them so you only need to focus on configuration.
Pick the one that suits your needs.
- [Installations from source][source]: You need to install Redis and Sentinel
yourself. Use the [Redis HA installation from source](redis_source.md)
documentation.
- [Omnibus GitLab **Community Edition** (CE) package][ce]: Redis is bundled, so you
can use the package with only the Redis service enabled as described in steps
1 and 2 of this document (works for both master and slave setups). To install
and configure Sentinel, jump directly to the Sentinel section in the
[Redis HA installation from source](redis_source.md#step-3-configuring-the-redis-sentinel-instances) documentation.
- [Omnibus GitLab **Enterprise Edition** (EE) package][ee]: Both Redis and Sentinel
are bundled in the package, so you can use the EE package to setup the whole
Redis HA infrastructure (master, slave and Sentinel) which is described in
this document.
- If you have installed GitLab using the Omnibus GitLab packages (CE or EE),
but you want to use your own external Redis server, follow steps 1-3 in the
[Redis HA installation from source](redis_source.md) documentation, then go
straight to step 4 in this guide to
[set up the GitLab application](#step-4-configuring-the-gitlab-application).
## Configuring Redis HA
This is the section where we install and setup the new Redis instances.
>**Notes:**
- We assume that you install GitLab and all HA components from scratch. If you
already have it installed and running, read how to
[switch from a single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha).
- Redis nodes (both master and slaves) will need the same password defined in
`redis['password']`. At any time during a failover the Sentinels can
reconfigure a node and change its status from master to slave and vice versa.
### Prerequisites
The prerequisites for a HA Redis setup are the following:
1. Provision the minimum required number of instances as specified in the
[recommended setup](#recommended-setup) section.
1. **Do NOT** install Redis or Redis Sentinel in the same machines your
GitLab application is running on. You can however opt in to install Redis
and Sentinel in the same machine (each in independent ones is recommended
though).
1. All Redis nodes must be able to talk to each other and accept incoming
connections over Redis (`6379`) and Sentinel (`26379`) ports (unless you
change the default ones).
1. The server that hosts the GitLab application must be able to access the
Redis nodes.
1. Protect the nodes from access from external networks ([Internet][it]), using
firewall.
### Step 1. Configuring the master Redis instance
1. SSH into the **master** Redis server and login as root:
**Slave Redis instance** ```
sudo -i
```
1. Follow same instructions from master with the extra change in `redis.conf`: 1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab
package you want using **steps 1 and 2** from the GitLab downloads page.
- Make sure you select the correct Omnibus package, with the same version
and type (Community, Enterprise editions) of your current install.
- Do not complete any other steps on the download page.
```conf 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
# IP and port of the master Redis server
slaveof 10.10.10.10 6379
```
1. Restart the Redis services for the changes to take effect. ```ruby
# Enable the master role and disable all other services in the machine
# (you can still enable Sentinel).
redis_master_role['enable'] = true
# IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces.
# If you really need to bind to an external accessible IP, make
# sure you add extra firewall rules to prevent unauthorized access.
redis['bind'] = '10.0.0.1'
# Define a port so Redis can listen for TCP requests which will allow other
# machines to connect to it.
redis['port'] = 6379
#### Omnibus Install # Set up password authentication for Redis (use the same password in all nodes).
redis['password'] = 'redis-password-goes-here'
```
You need to install the omnibus package in 3 different and independent machines. 1. To prevent database migrations from running on upgrade, run:
We will elect one as the initial `Master` and the other 2 as `Slaves`.
If you are migrating from a single machine install, you may want to setup the ```
machines as Slaves, pointing to the original machine as `Master`, to migrate sudo touch /etc/gitlab/skip-auto-migrations
the data first, and than switch to this setup. ```
To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`: Only the primary GitLab application server should handle migrations.
```ruby 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
redis['enable'] = false
```
**Master Redis instances** ### Step 2. Configuring the slave Redis instances
You need to make the following changes in `/etc/gitlab/gitlab.rb`: 1. SSH into the **slave** Redis server and login as root:
1. Define a `redis['bind']` address pointing to a local IP that your other machines ```
can reach you. If you really need to bind to an external acessible IP, make sudo -i
sure you add extra firewall rules to prevent unauthorized access. ```
1. Define a `redis['port']` to force redis to listin on TCP so other machines can
connect to it.
1. Set up password authentication with `redis['master_password']` (use the same
password in all nodes).
```ruby 1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab
## Redis TCP support (will disable UNIX socket transport) package you want using **steps 1 and 2** from the GitLab downloads page.
redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one - Make sure you select the correct Omnibus package, with the same version
redis['port'] = 6379 and type (Community, Enterprise editions) of your current install.
redis['requirepass'] = 'redis-password-goes-here' - Do not complete any other steps on the download page.
redis['master_password'] = 'redis-password-goes-here'
```
Reconfigure GitLab Omnibus for the changes to take effect: `sudo gitlab-ctl reconfigure` 1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
**Slave Redis instances** ```ruby
# Enable the slave role and disable all other services in the machine
# (you can still enable Sentinel). This will also set automatically
# `redis['master'] = false`.
redis_slave_role['enable'] = true
# IP address pointing to a local IP that the other machines can reach to.
# You can also set bind to '0.0.0.0' which listen in all interfaces.
# If you really need to bind to an external accessible IP, make
# sure you add extra firewall rules to prevent unauthorized access.
redis['bind'] = '10.0.0.2'
# Define a port so Redis can listen for TCP requests which will allow other
# machines to connect to it.
redis['port'] = 6379
You need to make the same changes listed for the `Master` instance, # The same password for Redeis authentication you set up for the master node.
with an additional `Slave` section as in the example below: redis['password'] = 'redis-password-goes-here'
```ruby # The IP of the master Redis node.
## Redis TCP support (will disable UNIX socket transport) redis['master_ip'] = '10.0.0.1'
redis['bind'] = '0.0.0.0' # or specify an IP to bind to a single one
redis['port'] = 6379
redis['requirepass'] = 'redis-password-goes-here'
redis['master_password'] = 'redis-password-goes-here'
## Slave redis instance # Port of master Redis server, uncomment to change to non default. Defaults
redis['master'] = false # to `6379`.
redis['master_ip'] = '10.10.10.10' # IP of master Redis server #redis['master_port'] = 6379
redis['master_port'] = 6379 # Port of master Redis server ```
```
Reconfigure GitLab Omnibus for the changes to take effect: `sudo gitlab-ctl reconfigure` 1. To prevent database migrations from running on upgrade, run:
#### Troubleshooting Replication ```
sudo touch /etc/gitlab/skip-auto-migrations
```
You can check if everything is correct by connecting to each server using Only the primary GitLab application server should handle migrations.
`redis-cli` application, and sending the `INFO` command.
If authentication was correctly defined, it should fail with: 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
`NOAUTH Authentication required` error. Try to authenticate with the 1. Go through the steps again for all the other slave nodes.
previous defined password with `AUTH redis-password-goes-here` and
try the `INFO` command again.
Look for the `# Replication` section where you should see some important ---
information like the `role` of the server.
When connected to a `master` redis, you will see the number of connected These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
`slaves`, and a list of each with connection details. a failover, as the nodes will be managed by the Sentinels, and even after a
`gitlab-ctl reconfigure`, they will get their configuration restored by
the same Sentinels.
When it's a `slave`, you will see details of the master connection and if ### Step 3. Configuring the Redis Sentinel instances
its `up` or `down`.
--- >**Note:**
Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. The
following section assumes you are using Omnibus GitLab Enterprise Edition.
For the Omnibus Community Edition and installations from source, follow the
[Redis HA source install](redis_source.md) guide.
Now that the Redis servers are all set up, let's configure the Sentinel Now that the Redis servers are all set up, let's configure the Sentinel
servers. servers.
If you are not sure if your Redis servers are working and replicating If you are not sure if your Redis servers are working and replicating
correctly, please read the [Troubleshooting Replication](#troubleshooting-replication) correctly, please read the [Troubleshooting Replication](#troubleshooting-replication)
and fix it before proceeding with Sentinel setup. and fix it before proceeding with Sentinel setup.
### Sentinel
You must have at least `3` Redis Sentinel servers, and they need to You must have at least `3` Redis Sentinel servers, and they need to
be each in a independent machine. You can install them in the same be each in an independent machine. You can configure them in the same
machines you installed the other `3` Redis servers. machines where you've configured the other Redis servers.
With GitLab Enterprise Edition, you can use the Omnibus package to setup
multiple machines with the Sentinel daemon.
---
1. SSH into the server that will host Redis Sentinel and login as root:
```
sudo -i
```
1. **You can omit this step if the Sentinels will be hosted in the same node as
the other Redis instances.**
[Download/install](https://about.gitlab.com/downloads-ee) the
Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the
GitLab downloads page.
- Make sure you select the correct Omnibus package, with the same version
the GitLab application is running.
- Do not complete any other steps on the download page.
1. Edit `/etc/gitlab/gitlab.rb` and add the contents (if you are installing the
Sentinels in the same node as the other Redis instances, some values might
be duplicate below):
```ruby
redis_sentinel_role['enable'] = true
# Must be the same in every sentinel node
redis['master_name'] = 'gitlab-redis'
# The same password for Redis authentication you set up for the master node.
redis['password'] = 'redis-password-goes-here'
# The IP of the master Redis node.
redis['master_ip'] = '10.0.0.1'
# Define a port so Redis can listen for TCP requests which will allow other
# machines to connect to it.
redis['port'] = 6379
# Port of master Redis server, uncomment to change to non default. Defaults
# to `6379`.
#redis['master_port'] = 6379
## Configure Sentinel
sentinel['bind'] = '10.0.0.1'
# Port that Sentinel listens on, uncomment to change to non default. Defaults
# to `26379`.
# sentinel['port'] = 26379
## Quorum must reflect the amount of voting sentinels it take to start a failover.
## Value must NOT be greater then the amount of sentinels.
##
## The quorum can be used to tune Sentinel in two ways:
## 1. If a the quorum is set to a value smaller than the majority of Sentinels
## we deploy, we are basically making Sentinel more sensible to master failures,
## triggering a failover as soon as even just a minority of Sentinels is no longer
## able to talk with the master.
## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
## making Sentinel able to failover only when there are a very large number (larger
## than majority) of well connected Sentinels which agree about the master being down.s
sentinel['quorum'] = 2
## Consider unresponsive server down after x amount of ms.
# sentinel['down_after_milliseconds'] = 10000
## Specifies the failover timeout in milliseconds. It is used in many ways:
##
## - The time needed to re-start a failover after a previous failover was
## already tried against the same master by a given Sentinel, is two
## times the failover timeout.
##
## - The time needed for a slave replicating to a wrong master according
## to a Sentinel current configuration, to be forced to replicate
## with the right master, is exactly the failover timeout (counting since
## the moment a Sentinel detected the misconfiguration).
##
## - The time needed to cancel a failover that is already in progress but
## did not produced any configuration change (SLAVEOF NO ONE yet not
## acknowledged by the promoted slave).
##
## - The maximum time a failover in progress waits for all the slaves to be
## reconfigured as slaves of the new master. However even after this time
## the slaves will be reconfigured by the Sentinels anyway, but not with
## the exact parallel-syncs progression as specified.
# sentinel['failover_timeout'] = 60000
```
1. To prevent database migrations from running on upgrade, run:
```
sudo touch /etc/gitlab/skip-auto-migrations
```
This number is required for the consensus algorithm to be effective Only the primary GitLab application server should handle migrations.
in the case of a failure. You should always have and `odd` number
of Sentinel nodes provisioned.
Here is a simple explanation on how Sentinel handles a failover: 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
1. Go through the steps again for all the other Sentinel nodes.
When a number of Sentinels (`quorum` value) agree the fact the `master` is ### Step 4. Configuring the GitLab application
not reachable, the **majority** of the sentinels must elect a temporary
Sentinel `leader`, that will be responsible to start the failover proceedings.
As an example, for a cluster of `3` Sentinels, at least `2` must agree on a The final part is to inform the main GitLab application server of the Redis
`leader`. If you have total of `5` at least `3` must agree on the leader. Sentinels servers and authentication credentials.
The `quorum` is only used to detect failure, not to elect the `leader`. You can enable or disable Sentinel support at any time in new or existing
installations. From the GitLab application perspective, all it requires is
the correct credentials for the Sentinel nodes.
Official [Sentinel documentation](http://redis.io/topics/sentinel#example-sentinel-deployments) While it doesn't require a list of all Sentinel nodes, in case of a failure,
also lists different network topologies and warns againts situations like it needs to access at least one of the listed.
network partition and how it can affect the state of the HA solution. Make
sure you read it carefully and understand the implications in your current
setup.
To make Sentinel setup easier, ee provide an [automated way to setup and run](#sentinel-setup-ee-only) >**Note:**
the Sentinel daemon with GitLab EE. The following steps should be performed in the [GitLab application server](gitlab.md)
which ideally should not have Redis or Sentinels on it for a HA setup.
#### Sentinel setup (Community Edition) 1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines:
For GitLab CE, you need to install, configure, execute and monitor Sentinel ```
by yourself. ## Must be the same in every sentinel node
redis['master_name'] = 'gitlab-redis'
## The same password for Redis authentication you set up for the master node.
redis['password'] = 'redis-password-goes-here'
## A list of sentinels with `host` and `port`
gitlab_rails['redis_sentinels'] = [
{'host' => '10.0.0.1', 'port' => 26379},
{'host' => '10.0.0.2', 'port' => 26379},
{'host' => '10.0.0.3', 'port' => 26379}
]
```
Here is an example configuration file (`sentinel.conf`) for a minimal Sentinel 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
node:
```conf ## Switching from an existing single-machine installation to Redis HA
bind 0.0.0.0 # bind to all interfaces or change to a specific IP
port 26379 # default sentinel port If you already have a single-machine GitLab install running, you will need to
sentinel auth-pass gitlab-redis redis-password-goes-here replicate from this machine first, before de-activating the Redis instance
sentinel monitor gitlab-redis 10.0.0.1 6379 2 inside it.
sentinel down-after-milliseconds gitlab-redis 10000
sentinel config-epoch gitlab-redis 0 Your single-machine install will be the initial **Master**, and the `3` others
sentinel leader-epoch gitlab-redis 0 should be configured as **Slave** pointing to this machine.
After replication catches up, you will need to stop services in the
single-machine install, to rotate the **Master** to one of the new nodes.
Make the required changes in configuration and restart the new nodes again.
To disable redis in the single install, edit `/etc/gitlab/gitlab.rb`:
```ruby
redis['enable'] = false
``` ```
#### Sentinel setup (EE Only) If you fail to replicate first, you may loose data (unprocessed background jobs).
## Example of a minimal configuration with 1 master, 2 slaves and 3 Sentinels
>**Note:**
Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For
different setups, read the
[available configuration setups](#available-configuration-setups) section.
In this example we consider that all servers have an internal network
interface with IPs in the `10.0.0.x` range, and that they can connect
to each other using these IPs.
In a real world usage, you would also setup firewall rules to prevent
unauthorized access from other machines and block traffic from the
outside (Internet).
We will use the same `3` nodes with **Redis** + **Sentinel** topology
discussed in [Redis setup overview](#redis-setup-overview) and
[Sentinel setup overview](#sentinel-setup-overview) documentation.
Here is a list and description of each **machine** and the assigned **IP**:
* `10.0.0.1`: Redis Master + Sentinel 1
* `10.0.0.2`: Redis Slave 1 + Sentinel 2
* `10.0.0.3`: Redis Slave 2 + Sentinel 3
* `10.0.0.4`: GitLab application
Please note that after the initial configuration, if a failover is initiated
by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master**
will change permanently (including in `redis.conf`) from one node to the other,
until a new failover is initiated again.
The same thing will happen with `sentinel.conf` that will be overridden after the
initial execution, after any new sentinel node starts watching the **Master**,
or a failover promotes a different **Master** node.
To setup sentinel, you must edit `/etc/gitlab/gitlab.rb` file. ### Example configuration for Redis master and Sentinel 1
This is a minimal configuration required to run the daemon:
In `/etc/gitlab/gitlab.rb`:
```ruby ```ruby
redis_master_role['enable'] = true
redis_sentinel_role['enable'] = true
redis['bind'] = '10.0.0.1'
redis['port'] = 6379
redis['password'] = 'redis-password-goes-here'
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance
redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance
redis['master_port'] = 6379 # port of the initial master redis instance #redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
redis['master_password'] = 'your-secure-password-here' # the same value defined in redis['password'] in the master instance sentinel['bind'] = '10.0.0.1'
# sentinel['port'] = 26379 # uncomment to change default port
sentinel['quorum'] = 2
# sentinel['down_after_milliseconds'] = 10000
# sentinel['failover_timeout'] = 60000
```
sentinel['enable'] = true [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
# sentinel['port'] = 26379
## Quorum must reflect the amount of voting sentinels it take to start a failover. ### Example configuration for Redis slave 1 and Sentinel 2
sentinel['quorum'] = 2
## Consider unresponsive server down after x amount of ms. In `/etc/gitlab/gitlab.rb`:
```ruby
redis_slave_role['enable'] = true
redis_sentinel_role['enable'] = true
redis['bind'] = '10.0.0.2'
redis['port'] = 6379
redis['password'] = 'redis-password-goes-here'
redis['master_password'] = 'redis-password-goes-here'
redis['master_ip'] = '10.0.0.1' # IP of master Redis server
#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
sentinel['bind'] = '10.0.0.2'
# sentinel['port'] = 26379 # uncomment to change default port
sentinel['quorum'] = 2
# sentinel['down_after_milliseconds'] = 10000 # sentinel['down_after_milliseconds'] = 10000
# sentinel['failover_timeout'] = 60000
```
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
### Example configuration for Redis slave 2 and Sentinel 3
In `/etc/gitlab/gitlab.rb`:
```ruby
redis_slave_role['enable'] = true
redis_sentinel_role['enable'] = true
redis['bind'] = '10.0.0.3'
redis['port'] = 6379
redis['password'] = 'redis-password-goes-here'
redis['master_password'] = 'redis-password-goes-here'
redis['master_ip'] = '10.0.0.1' # IP of master Redis server
#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
sentinel['bind'] = '10.0.0.3'
# sentinel['port'] = 26379 # uncomment to change default port
sentinel['quorum'] = 2
# sentinel['down_after_milliseconds'] = 10000
# sentinel['failover_timeout'] = 60000 # sentinel['failover_timeout'] = 60000
``` ```
When you install Sentinel in a separate machine, you need to control which [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
other services will be running in it. Take a look at the following variables
and enable or disable whenever it fits your strategy: ### Example configuration for the GitLab application
In `/etc/gitlab/gitlab.rb`:
```ruby ```ruby
# Enabled Redis and Sentinel services redis['master_name'] = 'gitlab-redis'
redis['enable'] = true redis['password'] = 'redis-password-goes-here'
gitlab_rails['redis_sentinels'] = [
{'host' => '10.0.0.1', 'port' => 26379},
{'host' => '10.0.0.2', 'port' => 26379},
{'host' => '10.0.0.3', 'port' => 26379}
]
```
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
## Advanced configuration
Omnibus GitLab configures some things behind the curtains to make the sysadmins'
lives easier. If you want to know what happens underneath keep reading.
### Control running services
In the previous example, we've used `redis_sentinel_role` and
`redis_master_role` which simplifies the amount of configuration changes.
If you want more control, here is what each one sets for you automatically
when enabled:
```ruby
## Redis Sentinel Role
redis_sentinel_role['enable'] = true
# When Sentinel Role is enabled, the following services are also enabled
sentinel['enable'] = true sentinel['enable'] = true
# Disabled all other services # The following services are disabled
redis['enable'] = false redis['enable'] = false
bootstrap['enable'] = false bootstrap['enable'] = false
nginx['enable'] = false nginx['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
postgresql['enable'] = false postgresql['enable'] = false
gitlab_workhorse['enable'] = false
gitlab_rails['enable'] = false gitlab_rails['enable'] = false
mailroom['enable'] = false mailroom['enable'] = false
```
Remember that enabling a new service may also require additional configuration -------
params (like `redis` for example).
--- ## Redis master/slave Role
redis_master_role['enable'] = true # enable only one of them
redis_slave_role['enable'] = true # enable only one of them
The final part is to inform the main GitLab application server of the Redis # When Redis Master or Slave role are enabled, the following services are
master and the new sentinels servers. # enabled/disabled. Note that if Redis and Sentinel roles are combined, both
# services will be enabled.
### GitLab setup # The following services are disabled
sentinel['enable'] = false
bootstrap['enable'] = false
nginx['enable'] = false
postgresql['enable'] = false
gitlab_rails['enable'] = false
mailroom['enable'] = false
You can enable or disable sentinel support at any time in new or existing # For Redis Slave role, also change this setting from default 'true' to 'false':
installations. From the GitLab application perspective, all it requires is redis['master'] = false
the correct credentials for the master Redis and for a few Sentinel nodes. ```
It doesn't require a list of all Sentinel nodes, as in case of a failure, You can find the relevant attributes defined in [gitlab_rails.rb][omnifile].
the application will need to query only one of them.
>**Note:** ## Troubleshooting
The following steps should be performed in the [GitLab application server](gitlab.md).
**For source based installations** There are a lot of moving parts that needs to be taken care carefully
in order for the HA setup to work as expected.
1. Edit `/home/git/gitlab/config/resque.yml` following the example in Before proceeding with the troubleshooting below, check your firewall rules:
`/home/git/gitlab/config/resque.yml.example`, and uncomment the sentinels
line, changing to the correct server credentials.
1. Restart GitLab for the changes to take effect.
**For Omnibus installations** - Redis machines
- Accept TCP connection in `6379`
- Connect to the other Redis machines via TCP in `6379`
- Sentinel machines
- Accept TCP connection in `26379`
- Connect to other Sentinel machines via TCP in `26379`
- Connect to the Redis machines via TCP in `6379`
1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines: ### Troubleshooting Redis replication
```ruby You can check if everything is correct by connecting to each server using
redis['master_name'] = "gitlab-redis" `redis-cli` application, and sending the `INFO` command.
redis['master_password'] = 'redis-password-goes-here'
gitlab_rails['redis_sentinels'] = [ If authentication was correctly defined, it should fail with:
{'host' => '10.10.10.1', 'port' => 26379}, `NOAUTH Authentication required` error. Try to authenticate with the
{'host' => '10.10.10.2', 'port' => 26379}, previous defined password with `AUTH redis-password-goes-here` and
{'host' => '10.10.10.3', 'port' => 26379} try the `INFO` command again.
]
``` Look for the `# Replication` section where you should see some important
information like the `role` of the server.
When connected to a `master` redis, you will see the number of connected
`slaves`, and a list of each with connection details:
```
# Replication
role:master
connected_slaves:1
slave0:ip=10.133.5.21,port=6379,state=online,offset=208037514,lag=1
master_repl_offset:208037658
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:206989083
repl_backlog_histlen:1048576
```
1. [Reconfigure] the GitLab for the changes to take effect. When it's a `slave`, you will see details of the master connection and if
its `up` or `down`:
### Sentinel troubleshooting ```
# Replication
role:slave
master_host:10.133.1.58
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:208096498
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
```
#### Omnibus install ### Troubleshooting Sentinel
If you get an error like: `Redis::CannotConnectError: No sentinels available.`, If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
there may be something wrong with your configuration files or it can be related there may be something wrong with your configuration files or it can be related
...@@ -424,42 +816,6 @@ The way the redis connector `redis-rb` works with sentinel is a bit ...@@ -424,42 +816,6 @@ The way the redis connector `redis-rb` works with sentinel is a bit
non-intuitive. We try to hide the complexity in omnibus, but it still requires non-intuitive. We try to hide the complexity in omnibus, but it still requires
a few extra configs. a few extra configs.
#### Source install
If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
there may be something wrong with your configuration files or it can be related
to [this issue][gh-531].
It's a bit non-intuitive the way you have to config `resque.yml` and
`sentinel.conf`, otherwise `redis-rb` will not work properly.
The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`)
**must** be used as the hostname in GitLab (`resque.yml` for source installations
or `gitlab-rails['redis_*']` in Omnibus):
```conf
# sentinel.conf:
sentinel monitor gitlab-redis 10.10.10.10 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel config-epoch gitlab-redis 0
sentinel leader-epoch gitlab-redis 0
```
```yaml
# resque.yaml
production:
url: redis://:myredispassword@gitlab-redis/
sentinels:
-
host: slave1.example.com # or use ip
port: 26380 # point to sentinel, not to redis port
-
host: slave2.exampl.com # or use ip
port: 26381 # point to sentinel, not to redis port
```
When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel)
--- ---
To make sure your configuration is correct: To make sure your configuration is correct:
...@@ -487,8 +843,8 @@ To make sure your configuration is correct: ...@@ -487,8 +843,8 @@ To make sure your configuration is correct:
1. To simulate a failover on master Redis, SSH into the Redis server and run: 1. To simulate a failover on master Redis, SSH into the Redis server and run:
```bash ```bash
# port must match your master redis port # port must match your master redis port, and the sleep time must be a few seconds bigger than defined one
redis-cli -h localhost -p 6379 DEBUG sleep 60 redis-cli -h localhost -p 6379 DEBUG sleep 20
``` ```
1. Then back in the Rails console from the first step, run: 1. Then back in the Rails console from the first step, run:
...@@ -500,10 +856,34 @@ To make sure your configuration is correct: ...@@ -500,10 +856,34 @@ To make sure your configuration is correct:
You should see a different port after a few seconds delay You should see a different port after a few seconds delay
(the failover/reconnect time). (the failover/reconnect time).
---
Read more on high-availability configuration: ## Changelog
Changes to Redis HA over time.
**8.14**
- Redis Sentinel support is production-ready and bundled in the Omnibus GitLab
Enterprise Edition package
- Documentation restructure for better readability
**8.11**
- Experimental Redis Sentinel support was added
### Experimental Redis Sentinel support
>
Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11.
Starting with 8.14, Redis Sentinel is no longer experimental.
If you used with versions `< 8.14` before, please check the updated
documentation here.
## Further reading
Read more on High Availability:
1. [High Availability Overview](README.md)
1. [Configure the database](database.md) 1. [Configure the database](database.md)
1. [Configure NFS](nfs.md) 1. [Configure NFS](nfs.md)
1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the GitLab application servers](gitlab.md)
...@@ -514,3 +894,10 @@ Read more on high-availability configuration: ...@@ -514,3 +894,10 @@ Read more on high-availability configuration:
[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure [reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure
[gh-531]: https://github.com/redis/redis-rb/issues/531 [gh-531]: https://github.com/redis/redis-rb/issues/531
[gh-534]: https://github.com/redis/redis-rb/issues/534 [gh-534]: https://github.com/redis/redis-rb/issues/534
[redis]: http://redis.io/
[sentinel]: http://redis.io/topics/sentinel
[omnifile]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb
[source]: ../../install/installation.md
[ce]: https://about.gitlab.com/downloads
[ee]: https://about.gitlab.com/downloads-ee
[it]: https://gitlab.com/gitlab-org/gitlab-ce/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png
doc/administration/high_availability/redis_source.md 0 → 100644
View file @ a9f7ce47
# Configuring non-Omnibus Redis for GitLab HA
This is the documentation for configuring a Highly Available Redis setup when
you have installed Redis all by yourself and not using the bundled one that
comes with the Omnibus packages.
We cannot stress enough the importance of reading the
[Overview section](redis.md#overview) of the Omnibus Redis HA as it provides
some invaluable information to the configuration of Redis. Please proceed to
read it before going forward with this guide.
We also highly recommend that you use the Omnibus GitLab packages, as we
optimize them specifically for GitLab, and we will take care of upgrading Redis
to the latest supported version.
If you're not sure whether this guide is for you, please refer to
[Available configuration setups](redis.md#available-configuration-setups) in
the Omnibus Redis HA documentation.
---
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents**
- [Configuring your own Redis server](#configuring-your-own-redis-server)
- [Prerequisites](#prerequisites)
- [Step 1. Configuring the master Redis instance](#step-1-configuring-the-master-redis-instance)
- [Step 2. Configuring the slave Redis instances](#step-2-configuring-the-slave-redis-instances)
- [Step 3. Configuring the Redis Sentinel instances](#step-3-configuring-the-redis-sentinel-instances)
- [Step 4. Configuring the GitLab application](#step-4-configuring-the-gitlab-application)
- [Example of minimal configuration with 1 master, 2 slaves and 3 Sentinels](#example-of-minimal-configuration-with-1-master-2-slaves-and-3-sentinels)
- [Example configuration for Redis master and Sentinel 1](#example-configuration-for-redis-master-and-sentinel-1)
- [Example configuration for Redis slave 1 and Sentinel 2](#example-configuration-for-redis-slave-1-and-sentinel-2)
- [Example configuration for Redis slave 2 and Sentinel 3](#example-configuration-for-redis-slave-2-and-sentinel-3)
- [Example configuration of the GitLab application](#example-configuration-of-the-gitlab-application)
- [Troubleshooting](#troubleshooting)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
## Configuring your own Redis server
This is the section where we install and setup the new Redis instances.
### Prerequisites
- All Redis servers in this guide must be configured to use a TCP connection
instead of a socket. To configure Redis to use TCP connections you need to
define both `bind` and `port` in the Redis config file. You can bind to all
interfaces (`0.0.0.0`) or specify the IP of the desired interface
(e.g., one from an internal network).
- Since Redis 3.2, you must define a password to receive external connections
(`requirepass`).
- If you are using Redis with Sentinel, you will also need to define the same
password for the slave password definition (`masterauth`) in the same instance.
In addition, read the prerequisites as described in the
[Omnibus Redis HA document](redis.md#prerequisites) since they provide some
valuable information for the general setup.
### Step 1. Configuring the master Redis instance
Assuming that the Redis master instance IP is `10.0.0.1`:
1. [Install Redis](../../install/installation.md#6-redis)
1. Edit `/etc/redis/redis.conf`:
```conf
## Define a `bind` address pointing to a local IP that your other machines
## can reach you. If you really need to bind to an external accessible IP, make
## sure you add extra firewall rules to prevent unauthorized access:
bind 10.0.0.1
## Define a `port` to force redis to listen on TCP so other machines can
## connect to it (default port is `6379`).
port 6379
## Set up password authentication (use the same password in all nodes).
## The password should be defined equal for both `requirepass` and `masterauth`
## when setting up Redis to use with Sentinel.
requirepass redis-password-goes-here
masterauth redis-password-goes-here
```
1. Restart the Redis service for the changes to take effect.
### Step 2. Configuring the slave Redis instances
Assuming that the Redis slave instance IP is `10.0.0.2`:
1. [Install Redis](../../install/installation.md#6-redis)
1. Edit `/etc/redis/redis.conf`:
```conf
## Define a `bind` address pointing to a local IP that your other machines
## can reach you. If you really need to bind to an external accessible IP, make
## sure you add extra firewall rules to prevent unauthorized access:
bind 10.0.0.2
## Define a `port` to force redis to listen on TCP so other machines can
## connect to it (default port is `6379`).
port 6379
## Set up password authentication (use the same password in all nodes).
## The password should be defined equal for both `requirepass` and `masterauth`
## when setting up Redis to use with Sentinel.
requirepass redis-password-goes-here
masterauth redis-password-goes-here
## Define `slaveof` pointing to the Redis master instance with IP and port.
slaveof 10.0.0.1 6379
```
1. Restart the Redis service for the changes to take effect.
1. Go through the steps again for all the other slave nodes.
### Step 3. Configuring the Redis Sentinel instances
Sentinel is a special type of Redis server. It inherits most of the basic
configuration options you can define in `redis.conf`, with specific ones
starting with `sentinel` prefix.
Assuming that the Redis Sentinel is installed on the same instance as Redis
master with IP `10.0.0.1` (some settings might overlap with the master):
1. [Install Redis Sentinel](http://redis.io/topics/sentinel)
1. Edit `/etc/redis/sentinel.conf`:
```conf
## Define a `bind` address pointing to a local IP that your other machines
## can reach you. If you really need to bind to an external accessible IP, make
## sure you add extra firewall rules to prevent unauthorized access:
bind 10.0.0.1
## Define a `port` to force Sentinel to listen on TCP so other machines can
## connect to it (default port is `6379`).
port 26379
## Set up password authentication (use the same password in all nodes).
## The password should be defined equal for both `requirepass` and `masterauth`
## when setting up Redis to use with Sentinel.
requirepass redis-password-goes-here
masterauth redis-password-goes-here
## Define with `sentinel auth-pass` the same shared password you have
## defined for both Redis master and slaves instances.
sentinel auth-pass gitlab-redis redis-password-goes-here
## Define with `sentinel monitor` the IP and port of the Redis
## master node, and the quorum required to start a failover.
sentinel monitor gitlab-redis 10.0.0.1 6379 2
## Define with `sentinel down-after-milliseconds` the time in `ms`
## that an unresponsive server will be considered down.
sentinel down-after-milliseconds gitlab-redis 10000
## Define a value for `sentinel failover_timeout` in `ms`. This has multiple
## meanings:
##
## * The time needed to re-start a failover after a previous failover was
## already tried against the same master by a given Sentinel, is two
## times the failover timeout.
##
## * The time needed for a slave replicating to a wrong master according
## to a Sentinel current configuration, to be forced to replicate
## with the right master, is exactly the failover timeout (counting since
## the moment a Sentinel detected the misconfiguration).
##
## * The time needed to cancel a failover that is already in progress but
## did not produced any configuration change (SLAVEOF NO ONE yet not
## acknowledged by the promoted slave).
##
## * The maximum time a failover in progress waits for all the slaves to be
## reconfigured as slaves of the new master. However even after this time
## the slaves will be reconfigured by the Sentinels anyway, but not with
## the exact parallel-syncs progression as specified.
sentinel failover_timeout 30000
```
1. Restart the Redis service for the changes to take effect.
1. Go through the steps again for all the other Sentinel nodes.
### Step 4. Configuring the GitLab application
You can enable or disable Sentinel support at any time in new or existing
installations. From the GitLab application perspective, all it requires is
the correct credentials for the Sentinel nodes.
While it doesn't require a list of all Sentinel nodes, in case of a failure,
it needs to access at least one of listed ones.
The following steps should be performed in the [GitLab application server](gitlab.md)
which ideally should not have Redis or Sentinels in the same machine for a HA
setup:
1. Edit `/home/git/gitlab/config/resque.yml` following the example in
[resque.yml.example][resque], and uncomment the Sentinel lines, pointing to
the correct server credentials:
```yaml
# resque.yaml
production:
url: redis://:redi-password-goes-here@gitlab-redis/
sentinels:
-
host: 10.0.0.1
port: 26379 # point to sentinel, not to redis port
-
host: 10.0.0.2
port: 26379 # point to sentinel, not to redis port
-
host: 10.0.0.3
port: 26379 # point to sentinel, not to redis port
```
1. [Restart GitLab][restart] for the changes to take effect.
## Example of minimal configuration with 1 master, 2 slaves and 3 Sentinels
In this example we consider that all servers have an internal network
interface with IPs in the `10.0.0.x` range, and that they can connect
to each other using these IPs.
In a real world usage, you would also setup firewall rules to prevent
unauthorized access from other machines, and block traffic from the
outside ([Internet][it]).
For this example, **Sentinel 1** will be configured in the same machine as the
**Redis Master**, **Sentinel 2** and **Sentinel 3** in the same machines as the
**Slave 1** and **Slave 2** respectively.
Here is a list and description of each **machine** and the assigned **IP**:
* `10.0.0.1`: Redis Master + Sentinel 1
* `10.0.0.2`: Redis Slave 1 + Sentinel 2
* `10.0.0.3`: Redis Slave 2 + Sentinel 3
* `10.0.0.4`: GitLab application
Please note that after the initial configuration, if a failover is initiated
by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master**
will change permanently (including in `redis.conf`) from one node to the other,
until a new failover is initiated again.
The same thing will happen with `sentinel.conf` that will be overridden after the
initial execution, after any new sentinel node starts watching the **Master**,
or a failover promotes a different **Master** node.
### Example configuration for Redis master and Sentinel 1
1. In `/etc/redis/redis.conf`:
```conf
bind 10.0.0.1
port 6379
requirepass redis-password-goes-here
masterauth redis-password-goes-here
```
1. In `/etc/redis/sentinel.conf`:
```conf
bind 10.0.0.1
port 26379
sentinel auth-pass gitlab-redis redis-password-goes-here
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel failover_timeout 30000
```
1. Restart the Redis service for the changes to take effect.
### Example configuration for Redis slave 1 and Sentinel 2
1. In `/etc/redis/redis.conf`:
```conf
bind 10.0.0.2
port 6379
requirepass redis-password-goes-here
masterauth redis-password-goes-here
slaveof 10.0.0.1 6379
```
1. In `/etc/redis/sentinel.conf`:
```conf
bind 10.0.0.2
port 26379
sentinel auth-pass gitlab-redis redis-password-goes-here
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel failover_timeout 30000
```
1. Restart the Redis service for the changes to take effect.
### Example configuration for Redis slave 2 and Sentinel 3
1. In `/etc/redis/redis.conf`:
```conf
bind 10.0.0.3
port 6379
requirepass redis-password-goes-here
masterauth redis-password-goes-here
slaveof 10.0.0.1 6379
```
1. In `/etc/redis/sentinel.conf`:
```conf
bind 10.0.0.3
port 26379
sentinel auth-pass gitlab-redis redis-password-goes-here
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel failover_timeout 30000
```
1. Restart the Redis service for the changes to take effect.
### Example configuration of the GitLab application
1. Edit `/home/git/gitlab/config/resque.yml`:
```yaml
production:
url: redis://:redi-password-goes-here@gitlab-redis/
sentinels:
-
host: 10.0.0.1
port: 26379 # point to sentinel, not to redis port
-
host: 10.0.0.2
port: 26379 # point to sentinel, not to redis port
-
host: 10.0.0.3
port: 26379 # point to sentinel, not to redis port
```
1. [Restart GitLab][restart] for the changes to take effect.
## Troubleshooting
We have a more detailed [Troubleshooting](redis.md#troubleshooting) explained
in the documentation for Omnibus GitLab installations. Here we will list only
the things that are specific to a source installation.
If you get an error in GitLab like: `Redis::CannotConnectError: No sentinels available.`,
there may be something wrong with your configuration files or it can be related
to [this upstream issue][gh-531].
It's a bit non-intuitive the way you have to config `resque.yml` and
`sentinel.conf`, otherwise `redis-rb` will not work properly.
The `master-group-name` ('gitlab-redis') defined in (`sentinel.conf`)
**must** be used as the hostname in GitLab (`resque.yml`):
```conf
# sentinel.conf:
sentinel monitor gitlab-redis 10.0.0.1 6379 2
sentinel down-after-milliseconds gitlab-redis 10000
sentinel config-epoch gitlab-redis 0
sentinel leader-epoch gitlab-redis 0
```
```yaml
# resque.yaml
production:
url: redis://:myredispassword@gitlab-redis/
sentinels:
-
host: 10.0.0.1
port: 26379 # point to sentinel, not to redis port
-
host: 10.0.0.2
port: 26379 # point to sentinel, not to redis port
-
host: 10.0.0.3
port: 26379 # point to sentinel, not to redis port
```
When in doubt, please read [Redis Sentinel documentation](http://redis.io/topics/sentinel)
[gh-531]: https://github.com/redis/redis-rb/issues/531
[downloads]: https://about.gitlab.com/downloads
[restart]: ../restart_gitlab.md#installations-from-source
[it]: https://gitlab.com/gitlab-org/gitlab-ce/uploads/c4cc8cd353604bd80315f9384035ff9e/The_Internet_IT_Crowd.png
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7