index.md 18.4 KB
Newer Older
Jacob Vosmaer's avatar
Jacob Vosmaer committed
1 2
# Gitaly

3
[Gitaly](https://gitlab.com/gitlab-org/gitaly) is the service that
4
provides high-level RPC access to Git repositories. Without it, no other
5 6 7
components can read or write Git data. GitLab components that access Git
repositories (gitlab-rails, gitlab-shell, gitlab-workhorse, etc.) act as clients
to Gitaly. End users do not have direct access to Gitaly.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
8

9 10 11
In the rest of this page, Gitaly server is referred to the standalone node that
only runs Gitaly, and Gitaly client to the GitLab Rails node that runs all other
processes except Gitaly.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
12 13 14

## Configuring Gitaly

15
The Gitaly service itself is configured via a TOML configuration file.
16
This file is documented [in the Gitaly
Jacob Vosmaer's avatar
Jacob Vosmaer committed
17 18
repository](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md).

19
In case you want to change some of its settings:
Jacob Vosmaer's avatar
Jacob Vosmaer committed
20

21
**For Omnibus GitLab**
Jacob Vosmaer's avatar
Jacob Vosmaer committed
22

23 24
1. Edit `/etc/gitlab/gitlab.rb` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676).
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
Jacob Vosmaer's avatar
Jacob Vosmaer committed
25

26
**For installations from source**
27

28 29
1. Edit `/home/git/gitaly/config.toml` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example).
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
30

31 32
## Running Gitaly on its own server

33
This is an optional way to deploy Gitaly which can benefit GitLab
34 35 36 37
installations that are larger than a single machine. Most
installations will be better served with the default configuration
used by Omnibus and the GitLab source installation guide.

38 39 40
Starting with GitLab 11.4, Gitaly is able to serve all Git requests without
needed a shared NFS mount for Git repository data.
Between 11.4 and 11.8 the exception was the
41
[Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer).
42 43 44
But since 11.8 the indexer uses Gitaly for data access as well. NFS can still
be leveraged for redudancy on block level of the Git data. But only has to
be mounted on the Gitaly server.
45

46 47
NOTE: **Note:** While Gitaly can be used as a replacement for NFS, it's not recommended
to use EFS as it may impact GitLab's performance. Review the [relevant documentation](../high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs)
48 49
for more details.

50 51
### Network architecture

52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73
The following list depicts what the network architecture of Gitaly is:

- GitLab Rails shards repositories into [repository storages](../repository_storage_paths.md).
- `/config/gitlab.yml` contains a map from storage names to
  `(Gitaly address, Gitaly token)` pairs.
- the `storage name` -\> `(Gitaly address, Gitaly token)` map in
  `/config/gitlab.yml` is the single source of truth for the Gitaly network
  topology.
- A `(Gitaly address, Gitaly token)` corresponds to a Gitaly server.
- A Gitaly server hosts one or more storages.
- Gitaly addresses must be specified in such a way that they resolve
  correctly for ALL Gitaly clients.
- Gitaly clients are: Unicorn, Sidekiq, gitlab-workhorse,
  gitlab-shell, Elasticsearch Indexer, and Gitaly itself.
- A Gitaly server must be able to make RPC calls **to itself** via its own
  `(Gitaly address, Gitaly token)` pair as specified in `/config/gitlab.yml`.
- Gitaly servers must not be exposed to the public internet as Gitaly's network
  traffic is unencrypted by default. The use of firewall is highly recommended
  to restrict access to the Gitaly server. Another option is to
  [use TLS](#tls-support).
- Authentication is done through a static token which is shared among the Gitaly
  and GitLab Rails nodes.
74 75

Below we describe how to configure a Gitaly server at address
76
`gitaly.internal:8075` with secret token `abc123secret`. We assume
77 78 79
your GitLab installation has two repository storages, `default` and
`storage1`.

80
### 1. Installation
81

82
First install Gitaly using either Omnibus GitLab or install it from source:
83

84 85 86 87
- For Omnibus GitLab: [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab
  package you want using **steps 1 and 2** from the GitLab downloads page but
  **_do not_** provide the `EXTERNAL_URL=` value.
- From source: [Install Gitaly](../../install/installation.md#install-gitaly).
88

89
### 2. Client side token configuration
90

91
Configure a token on the instance that runs the GitLab Rails application.
92

93
**For Omnibus GitLab**
94

95
1. On the client node(s), edit `/etc/gitlab/gitlab.rb`:
96

97 98 99
   ```ruby
   gitlab_rails['gitaly_token'] = 'abc123secret'
   ```
100

101
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
102

103 104 105 106 107 108 109 110 111
**For installations from source**

1. On the client node(s), edit `/home/git/gitlab/config/gitlab.yml`:

   ```yaml
   gitlab:
     gitaly:
       token: 'abc123secret'
   ```
112

113
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
114

115
### 3. Gitaly server configuration
116

117
Next, on the Gitaly server, you need to configure storage paths, enable
118 119
the network listener and configure the token.

120
NOTE: **Note:** if you want to reduce the risk of downtime when you enable
Jacob Vosmaer's avatar
Jacob Vosmaer committed
121 122 123
authentication you can temporarily disable enforcement, see [the
documentation on configuring Gitaly
authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication)
124 125
.

126
Gitaly must trigger some callbacks to GitLab via GitLab Shell. As a result,
127 128 129
the GitLab Shell secret must be the same between the other GitLab servers and
the Gitaly server. The easiest way to accomplish this is to copy `/etc/gitlab/gitlab-secrets.json`
from an existing GitLab server to the Gitaly server. Without this shared secret,
130
Git operations in GitLab will result in an API error.
131

132 133 134 135
NOTE: **Note:**
In most or all cases, the storage paths below end in `/repositories` which is
not that case with `path` in `git_data_dirs` of Omnibus GitLab installations.
Check the directory layout on your Gitaly server to be sure.
136

137
**For Omnibus GitLab**
138

139
1. Edit `/etc/gitlab/gitlab.rb`:
140

141 142 143 144
   <!--
   updates to following example must also be made at
   https://gitlab.com/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab
   -->
145

146 147
   ```ruby
   # /etc/gitlab/gitlab.rb
148

149 150 151 152 153 154 155 156
   # Avoid running unnecessary services on the Gitaly server
   postgresql['enable'] = false
   redis['enable'] = false
   nginx['enable'] = false
   prometheus['enable'] = false
   unicorn['enable'] = false
   sidekiq['enable'] = false
   gitlab_workhorse['enable'] = false
157

158 159 160 161 162 163 164 165 166
   # Prevent database connections during 'gitlab-ctl reconfigure'
   gitlab_rails['rake_cache_clear'] = false
   gitlab_rails['auto_migrate'] = false

   # Configure the gitlab-shell API callback URL. Without this, `git push` will
   # fail. This can be your 'front door' GitLab URL or an internal load
   # balancer.
   # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server.
   gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
167

168 169 170 171
   # Make Gitaly accept connections on all network interfaces. You must use
   # firewalls to restrict access to this address/port.
   gitaly['listen_addr'] = "0.0.0.0:8075"
   gitaly['auth_token'] = 'abc123secret'
172

173 174 175 176
   gitaly['storage'] = [
     { 'name' => 'default' },
     { 'name' => 'storage1' },
   ]
177

178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224
   # To use TLS for Gitaly you need to add
   gitaly['tls_listen_addr'] = "0.0.0.0:9999"
   gitaly['certificate_path'] = "path/to/cert.pem"
   gitaly['key_path'] = "path/to/key.pem"
   ```

   NOTE: **Note:**
   In some cases, you'll have to set `path` for `gitaly['storage']` in the
   format `'path' => '/mnt/gitlab/<storage name>/repositories'`.

1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).

**For installations from source**

1. On the client node(s), edit `/home/git/gitaly/config.toml`:

   ```toml
   listen_addr = '0.0.0.0:8075'
   tls_listen_addr = '0.0.0.0:9999'

   [tls]
   certificate_path = /path/to/cert.pem
   key_path = /path/to/key.pem

   [auth]
   token = 'abc123secret'

   [[storage]]
   name = 'default'

   [[storage]]
   name = 'storage1'
   ```

   NOTE: **Note:**
   In some cases, you'll have to set `path` for each `[[storage]]` in the
   format `path = '/mnt/gitlab/<storage name>/repositories'`.

1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).

### 4. Converting clients to use the Gitaly server

As the final step, you need to update the client machines to switch from using
their local Gitaly service to the new Gitaly server you just configured. This
is a risky step because if there is any sort of network, firewall, or name
resolution problem preventing your GitLab server from reaching the Gitaly server,
then all Gitaly requests will fail.
225

226
Additionally, you need to
227 228
[disable Rugged if previously manually enabled](../high_availability/nfs.md#improving-nfs-performance-with-gitlab).

229
We assume that your Gitaly server can be reached at
230 231
`gitaly.internal:8075` from your GitLab server, and that Gitaly can read and
write to `/mnt/gitlab/default` and `/mnt/gitlab/storage1` respectively.
232

233
**For Omnibus GitLab**
234

235
1. Edit `/etc/gitlab/gitlab.rb`:
236

237 238 239 240 241
   ```ruby
   git_data_dirs({
     'default' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' },
     'storage1' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' },
   })
242

243 244
   gitlab_rails['gitaly_token'] = 'abc123secret'
   ```
245

246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319
   NOTE: **Note:**
   In some cases, you'll have to set `path` for each `git_data_dirs` in the
   format `'path' => '/mnt/gitlab/<storage name>'`.

1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
1. Tail the logs to see the requests:

   ```sh
   sudo gitlab-ctl tail gitaly
   ```

**For installations from source**

1. Edit `/home/git/gitlab/config/gitlab.yml`:

   ```yaml
   gitlab:
     repositories:
       storages:
         default:
           gitaly_address: tcp://gitaly.internal:8075
         storage1:
           gitaly_address: tcp://gitaly.internal:8075

     gitaly:
       token: 'abc123secret'
   ```

   NOTE: **Note:**
   In some cases, you'll have to set `path` for each of the `storages` in the
   format `path: /mnt/gitlab/<storage name>/repositories`.

1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
1. Tail the logs to see the requests:

   ```sh
   tail -f /home/git/gitlab/log/gitaly.log
   ```

When you tail the Gitaly logs on your Gitaly server you should see requests
coming in. One sure way to trigger a Gitaly request is to clone a repository
from your GitLab server over HTTP.

### Disabling the Gitaly service in a cluster environment

If you are running Gitaly [as a remote
service](#running-gitaly-on-its-own-server) you may want to disable
the local Gitaly service that runs on your GitLab server by default.
Disabling Gitaly only makes sense when you run GitLab in a custom
cluster configuration, where different services run on different
machines. Disabling Gitaly on all machines in the cluster is not a
valid configuration.

To disable Gitaly on a client node:

**For Omnibus GitLab**

1. Edit `/etc/gitlab/gitlab.rb`:

   ```ruby
   gitaly['enable'] = false
   ```

1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).

**For installations from source**

1. Edit `/etc/default/gitlab`:

   ```shell
   gitaly_enabled=false
   ```

1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
320

321 322
## TLS support

323
> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22602) in GitLab 11.8.
324

325
Gitaly supports TLS encryption. To be able to communicate
326
with a Gitaly instance that listens for secure connections you will need to use `tls://` url
Ben Bodenmiller's avatar
Ben Bodenmiller committed
327
scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration.
328

329 330 331 332
You will need to bring your own certificates as this isn't provided automatically.
The certificate to be used needs to be installed on all Gitaly nodes and on all
client nodes that communicate with it following the procedure described in
[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates).
333

334 335
NOTE: **Note:**
It is possible to configure Gitaly servers with both an
336 337 338 339
unencrypted listening address `listen_addr` and an encrypted listening
address `tls_listen_addr` at the same time. This allows you to do a
gradual transition from unencrypted to encrypted traffic, if necessary.

340
To configure Gitaly with TLS:
Ahmad Hassan's avatar
Ahmad Hassan committed
341

342
**For Omnibus GitLab**
343

344
1. On the client nodes, edit `/etc/gitlab/gitlab.rb`:
Ahmad Hassan's avatar
Ahmad Hassan committed
345

346 347 348 349 350
   ```ruby
   git_data_dirs({
     'default' => { 'gitaly_address' => 'tls://gitaly.internal:9999' },
     'storage1' => { 'gitaly_address' => 'tls://gitaly.internal:9999' },
   })
351

352 353
   gitlab_rails['gitaly_token'] = 'abc123secret'
   ```
354

355 356
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
1. On the Gitaly server nodes, edit `/etc/gitlab/gitlab.rb`:
357

358 359 360 361
   ```ruby
   gitaly['tls_listen_addr'] = "0.0.0.0:9999"
   gitaly['certificate_path'] = "path/to/cert.pem"
   gitaly['key_path'] = "path/to/key.pem"
362
   ```
Ahmad Hassan's avatar
Ahmad Hassan committed
363

364
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
Ahmad Hassan's avatar
Ahmad Hassan committed
365

366
**For installations from source**
Ahmad Hassan's avatar
Ahmad Hassan committed
367

368
1. On the client nodes, edit `/home/git/gitlab/config/gitlab.yml`:
369

370 371 372 373 374 375 376 377
   ```yaml
   gitlab:
     repositories:
       storages:
         default:
           gitaly_address: tls://gitaly.internal:9999
         storage1:
           gitaly_address: tls://gitaly.internal:9999
378

379 380 381
     gitaly:
       token: 'abc123secret'
   ```
382

383 384 385
   NOTE: **Note:**
   In some cases, you'll have to set `path` for each of the `storages` in the
   format `path: /mnt/gitlab/<storage name>/repositories`.
Ahmad Hassan's avatar
Ahmad Hassan committed
386

387 388
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
1. On the Gitaly server nodes, edit `/home/git/gitaly/config.toml`:
Ahmad Hassan's avatar
Ahmad Hassan committed
389

390 391
   ```toml
   tls_listen_addr = '0.0.0.0:9999'
Ahmad Hassan's avatar
Ahmad Hassan committed
392

393 394 395 396
   [tls]
   certificate_path = '/path/to/cert.pem'
   key_path = '/path/to/key.pem'
   ```
397

398
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
399

400 401
To observe what type of connections are actually being used in a
production environment you can use the following Prometheus query:
402 403

```
404
sum(rate(gitaly_connections_total[5m])) by (type)
405 406
```

407
## `gitaly-ruby`
408

409 410 411 412 413 414 415 416
Gitaly was developed to replace the Ruby application code in GitLab.
In order to save time and/or avoid the risk of rewriting existing
application logic, in some cases we chose to copy some application code
from GitLab into Gitaly almost as-is. To be able to run that code,
`gitaly-ruby` was created, which is a "sidecar" process for the main Gitaly Go
process. Some examples of things that are implemented in `gitaly-ruby` are
RPCs that deal with wikis, and RPCs that create commits on behalf of
a user, such as merge commits.
417

418
### Number of `gitaly-ruby` workers
419

420 421 422 423 424
`gitaly-ruby` has much less capacity than Gitaly itself. If your Gitaly
server has to handle a lot of requests, the default setting of having
just one active `gitaly-ruby` sidecar might not be enough. If you see
`ResourceExhausted` errors from Gitaly, it's very likely that you have not
enough `gitaly-ruby` capacity.
425

426 427
You can increase the number of `gitaly-ruby` processes on your Gitaly
server with the following settings.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
428

429
**For Omnibus GitLab**
Jacob Vosmaer's avatar
Jacob Vosmaer committed
430

431
1. Edit `/etc/gitlab/gitlab.rb`:
432

433 434 435 436 437
   ```ruby
   # Default is 2 workers. The minimum is 2; 1 worker is always reserved as
   # a passive stand-by.
   gitaly['ruby_num_workers'] = 4
   ```
Jacob Vosmaer's avatar
Jacob Vosmaer committed
438

439
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
Jacob Vosmaer's avatar
Jacob Vosmaer committed
440

441
**For installations from source**
Jacob Vosmaer's avatar
Jacob Vosmaer committed
442

443
1. Edit `/home/git/gitaly/config.toml`:
Jacob Vosmaer's avatar
Jacob Vosmaer committed
444

445 446 447 448
   ```toml
   [gitaly-ruby]
   num_workers = 4
   ```
Jacob Vosmaer's avatar
Jacob Vosmaer committed
449

450
1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
451

452 453
## Eliminating NFS altogether

454 455
If you are planning to use Gitaly without NFS for your storage needs
and want to eliminate NFS from your environment altogether, there are
456 457
a few things that you need to do:

458 459 460 461 462 463 464 465 466 467 468
1. Make sure the [`git` user home directory](https://docs.gitlab.com/omnibus/settings/configuration.html#moving-the-home-directory-for-a-user) is on local disk.
1. Configure [database lookup of SSH keys](../operations/fast_ssh_key_lookup.md)
   to eliminate the need for a shared authorized_keys file.
1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage)
   including [live tracing](../job_traces.md#new-live-trace-architecture).
1. Configure [object storage for LFS objects](../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage).
1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only).

NOTE: **Note:**
One current feature of GitLab that still requires a shared directory (NFS) is
[GitLab Pages](../../user/project/pages/index.md).
469
There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196)
470
to eliminate the need for NFS to support GitLab Pages.
471

472 473 474
## Troubleshooting

### `gitaly-debug`
475 476 477 478 479 480 481 482 483

Since GitLab 11.6, Gitaly comes with a command-line tool called
`gitaly-debug` that can be run on a Gitaly server to aid in
troubleshooting. In GitLab 11.6 its only sub-command is
`simulate-http-clone` which allows you to measure the maximum possible
Git clone speed for a specific repository on the server.

For an up to date list of sub-commands see [the gitaly-debug
README](https://gitlab.com/gitlab-org/gitaly/blob/master/cmd/gitaly-debug/README.md).
484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512

### Client side GRPC logs

Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC
client has its own log file which may contain useful information when
you are seeing Gitaly errors. You can control the log level of the
gRPC client with the `GRPC_LOG_LEVEL` environment variable. The
default level is `WARN`.

### Observing `gitaly-ruby` traffic

[`gitaly-ruby`](#gitaly-ruby) is an internal implementation detail of Gitaly,
so, there's not that much visibility into what goes on inside
`gitaly-ruby` processes.

If you have Prometheus set up to scrape your Gitaly process, you can see
request rates and error codes for individual RPCs in `gitaly-ruby` by
querying `grpc_client_handled_total`. Strictly speaking, this metric does
not differentiate between `gitaly-ruby` and other RPCs, but in practice
(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal
calls from the main Gitaly process to one of its `gitaly-ruby` sidecars.

Assuming your `grpc_client_handled_total` counter only observes Gitaly,
the following query shows you RPCs are (most likely) internally
implemented as calls to `gitaly-ruby`:

```
sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0
```