architecture.md 43.7 KB
Newer Older
1 2 3 4
---
table_display_block: true
---

5
# GitLab Architecture Overview
6

7
## Software delivery
8

9
There are two software distributions of GitLab: the open source [Community Edition](https://gitlab.com/gitlab-org/gitlab-ce/) (CE), and the open core [Enterprise Edition](https://gitlab.com/gitlab-org/gitlab-ee/) (EE). GitLab is available under [different subscriptions](https://about.gitlab.com/pricing/).
10

11 12 13
New versions of GitLab are released in stable branches and the master branch is for bleeding edge development.

For information, see the [GitLab Release Process](https://gitlab.com/gitlab-org/release/docs/tree/master#gitlab-release-process).
14

15
Both EE and CE require some add-on components called gitlab-shell and Gitaly. These components are available from the [gitlab-shell](https://gitlab.com/gitlab-org/gitlab-shell/tree/master) and [gitaly](https://gitlab.com/gitlab-org/gitaly/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical.
16

17
## Components
18

19 20 21 22 23 24 25 26 27 28 29 30 31 32
A typical install of GitLab will be on GNU/Linux. It uses Nginx or Apache as a web front end to proxypass the Unicorn web server. By default, communication between Unicorn and the front end is via a Unix domain socket but forwarding requests via TCP is also supported. The web front end accesses `/home/git/gitlab/public` bypassing the Unicorn server to serve static pages, uploads (e.g. avatar images or attachments), and precompiled assets. GitLab serves web pages and a [GitLab API](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/api) using the Unicorn web server. It uses Sidekiq as a job queue which, in turn, uses redis as a non-persistent database backend for job information, meta data, and incoming jobs.

We also support deploying GitLab on Kubernetes using our [gitlab Helm chart](https://docs.gitlab.com/charts/).

The GitLab web app uses MySQL or PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository.

When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving git objects.

The add-on component gitlab-shell serves repositories over SSH. It manages the SSH keys within `/home/git/.ssh/authorized_keys` which should not be manually edited. gitlab-shell accesses the bare repositories through Gitaly to serve git objects and communicates with redis to submit jobs to Sidekiq for GitLab to process. gitlab-shell queries the GitLab API to determine authorization and access.

Gitaly executes git operations from gitlab-shell and the GitLab web app, and provides an API to the GitLab web app to get attributes from git (e.g. title, branches, tags, other meta data), and to get blobs (e.g. diffs, commits, files).

You may also be interested in the [production architecture of GitLab.com](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/).

Rachel Nienaber's avatar
Rachel Nienaber committed
33 34 35 36 37 38 39 40 41 42
### Simplified Component Overview

This is a simplified architecture diagram that can be used to
understand GitLab's architecture.

A complete architecture diagram is available in our
[component diagram](#component-diagram) below.

![Simplified Component Overview](img/architecture_simplified.png)

Evan Read's avatar
Evan Read committed
43
<!--
Rachel Nienaber's avatar
Rachel Nienaber committed
44 45 46 47
To update this diagram, GitLab team members can edit this source file:
https://docs.google.com/drawings/d/1fBzAyklyveF-i-2q-OHUIqDkYfjjxC4mq5shwKSZHLs/edit.
 -->

48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80
### Component diagram

```mermaid
graph TB

  HTTP[HTTP/HTTPS] -- TCP 80, 443 --> NGINX[NGINX]
  SSH -- TCP 22 --> GitLabShell[GitLab Shell]
  SMTP[SMTP Gateway]
  Geo[GitLab Geo Node] -- TCP 22, 80, 443 --> NGINX

  GitLabShell --TCP 8080 -->Unicorn["Unicorn (GitLab Rails)"]
  GitLabShell --> Gitaly
  GitLabShell --> Redis
  Unicorn --> PgBouncer[PgBouncer]
  Unicorn --> Redis
  Unicorn --> Gitaly
  Redis --> Sidekiq
  Sidekiq["Sidekiq (GitLab Rails, ES Indexer)"] --> PgBouncer
  GitLabWorkhorse[GitLab Workhorse] --> Unicorn
  GitLabWorkhorse --> Redis
  GitLabWorkhorse --> Gitaly
  Gitaly --> Redis
  NGINX --> GitLabWorkhorse
  NGINX -- TCP 8090 --> GitLabPages[GitLab Pages]
  NGINX --> Grafana[Grafana]
  Grafana -- TCP 9090 --> Prometheus[Prometheus]
  Prometheus -- TCP 80, 443 --> Unicorn
  RedisExporter[Redis Exporter] --> Redis
  Prometheus -- TCP 9121 --> RedisExporter
  PostgreSQLExporter[PostgreSQL Exporter] --> PostgreSQL
  PgBouncerExporter[PgBouncer Exporter] --> PgBouncer
  Prometheus -- TCP 9187 --> PostgreSQLExporter
  Prometheus -- TCP 9100 --> NodeExporter[Node Exporter]
81
  Prometheus -- TCP 9168 --> GitLabMonitor[GitLab Monitor]
82 83 84 85 86 87 88 89 90 91 92 93 94 95
  Prometheus -- TCP 9127 --> PgBouncerExporter
  GitLabMonitor --> PostgreSQL
  GitLabMonitor --> GitLabShell
  GitLabMonitor --> Sidekiq
  PgBouncer --> Consul
  PostgreSQL --> Consul
  PgBouncer --> PostgreSQL
  NGINX --> Registry
  Unicorn --> Registry
  NGINX --> Mattermost
  Mattermost --- Unicorn
  Prometheus --> Alertmanager
  Migrations --> PostgreSQL
  Runner -- TCP 443 --> NGINX
96 97
  Unicorn -- TCP 9200 --> Elasticsearch
  Sidekiq -- TCP 9200 --> Elasticsearch
98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118
  Sidekiq -- TCP 80, 443 --> Sentry
  Unicorn -- TCP 80, 443 --> Sentry
  Sidekiq -- UDP 6831 --> Jaeger
  Unicorn -- UDP 6831 --> Jaeger
  Gitaly -- UDP 6831 --> Jaeger
  GitLabShell -- UDP 6831 --> Jaeger
  GitLabWorkhorse -- UDP 6831 --> Jaeger
  Alertmanager -- TCP 25 --> SMTP
  Sidekiq -- TCP 25 --> SMTP
  Unicorn -- TCP 25 --> SMTP
  Unicorn -- TCP 369 --> LDAP
  Sidekiq -- TCP 369 --> LDAP
  Unicorn -- TCP 443 --> ObjectStorage["Object Storage"]
  Sidekiq -- TCP 443 --> ObjectStorage
  GitLabWorkhorse -- TCP 443 --> ObjectStorage
  Registry -- TCP 443 --> ObjectStorage
  Geo -- TCP 5432 --> PostgreSQL
```

### Component legend

119 120 121 122
- ✅ - Installed by default
- ⚙ - Requires additional configuration, or GitLab Managed Apps
- ⤓ - Manual installation required
- ❌ - Not supported or no instructions available
123 124 125 126 127

Component statuses are linked to configuration documentation for each component.

### Component list

128
| Component | Description | [Omnibus GitLab](https://docs.gitlab.com/omnibus/) | [GitLab chart](https://docs.gitlab.com/charts/) | [Minikube Minimal](https://docs.gitlab.com/charts/development/minikube/#deploying-gitlab-with-minimal-settings) | [GitLab.com](https://gitlab.com) | [Source](../install/installation.md) | [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) | CE/EE |
129 130
| --------- | ----------- |:--------------------:|:------------------:|:-----:|:--------:|:--------:|:-------:|:-------:|
| [NGINX](#nginx) | Routes requests to appropriate components, terminates SSL | [][nginx-omnibus] | [][nginx-charts] | [][nginx-charts] | [](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [][nginx-source] | ❌ | CE & EE |
131 132
| [Unicorn (GitLab Rails)](#unicorn) | Handles requests for the web interface and API | [][unicorn-omnibus] | [][unicorn-charts] | [][unicorn-charts] | [](../user/gitlab_com/index.md#unicorn) | [][unicorn-source] | [][gitlab-yml] | CE & EE |
| [Sidekiq](#sidekiq) | Background jobs processor | [][sidekiq-omnibus] | [][sidekiq-charts] | [](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/index.html) | [](../user/gitlab_com/index.md#sidekiq) | [][gitlab-yml] | [][gitlab-yml] | CE & EE |
133 134 135
| [Gitaly](#gitaly) | Git RPC service for handling all git calls made by GitLab | [][gitaly-omnibus] | [][gitaly-charts] | [][gitaly-charts] | [](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [][gitaly-source] | ✅ | CE & EE |
| [GitLab Workhorse](#gitlab-workhorse) | Smart reverse proxy, handles large HTTP requests | [][workhorse-omnibus] | [][workhorse-charts] | [][workhorse-charts] | [](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [][workhorse-source] | ✅ | CE & EE |
| [GitLab Shell](#gitlab-shell) | Handles `git` over SSH sessions | [][shell-omnibus] | [][shell-charts] | [][shell-charts] | [](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [][shell-source] | [][gitlab-yml] | CE & EE |
136 137
| [GitLab Pages](#gitlab-pages) | Hosts static websites | [][pages-omnibus] | [][pages-charts] | [][pages-charts] | [](../user/gitlab_com/index.md#gitlab-pages) | [][pages-source] | [][pages-gdk] | CE & EE  |
| [Registry](#registry) | Container registry, allows pushing and pulling of images | [][registry-omnibus] | [][registry-charts] | [][registry-charts] | [](../user/project/container_registry.md#build-and-push-images) | [][registry-source] | [][registry-gdk] | CE & EE  |
138
| [Redis](#redis) | Caching service | [][redis-omnibus] | [][redis-omnibus] | [][redis-charts] | [](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [][redis-source] | ✅ | CE & EE |
139
| [PostgreSQL](#postgresql) | Database | [][postgres-omnibus] | [][postgres-charts] | [][postgres-charts] | [](../user/gitlab_com/index.md#postgresql) | [][postgres-source] | ✅ | CE & EE |
140
| [PgBouncer](#pgbouncer) | Database connection pooling, failover | [][pgbouncer-omnibus] | [][pgbouncer-charts] | [][pgbouncer-charts] | [](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#database-architecture) | ❌ | ❌ | EE Only |
141 142
| [Consul](#consul) | Database node discovery, failover | [][consul-omnibus] | [][consul-charts] | [][consul-charts] | [](../user/gitlab_com/index.md#consul) | ❌ | ❌ | EE Only |
| [GitLab self-monitoring: Prometheus](#prometheus) | Time-series database, metrics collection, and query service | [][prometheus-omnibus] | [][prometheus-charts] | [][prometheus-charts] | [](../user/gitlab_com/index.md#prometheus) | ❌ | ❌ | CE & EE |
143
| [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | [][alertmanager-omnibus] | [][alertmanager-charts] |  [][alertmanager-charts] | [](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE |
144
| [GitLab self-monitoring: Grafana](#grafana) | Metrics dashboard | [][grafana-omnibus] | [][grafana-charts] |  [][grafana-charts] | [](https://dashboards.gitlab.com/d/RZmbBr7mk/gitlab-triage?refresh=30s) | ❌ | ❌ | CE & EE |
145 146 147 148 149
| [GitLab self-monitoring: Sentry](#sentry) | Track errors generated by the GitLab instance | [][sentry-omnibus] | [][sentry-charts] | [][sentry-charts] | [](https://about.gitlab.com/handbook/support/workflows/services/gitlab_com/500_errors.html#searching-sentry) | [][gitlab-yml] | [][gitlab-yml] | CE & EE |
| [GitLab self-monitoring: Jaeger](#jaeger) | View traces generated by the GitLab instance | [][jaeger-omnibus] | [][jaeger-charts] | [][jaeger-charts] | [](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4104) | [][jaeger-source] | [][jaeger-gdk] | CE & EE |
| [Redis Exporter](#redis-exporter) | Prometheus endpoint with Redis metrics | [][redis-exporter-omnibus] | [][redis-exporter-charts] | [][redis-exporter-charts] | [](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE |
| [Postgres Exporter](#postgres-exporter) | Prometheus endpoint with PostgreSQL metrics | [][postgres-exporter-omnibus] | [][postgres-exporter-charts] | [][postgres-exporter-charts] | [](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE |
| [PgBouncer Exporter](#pgbouncer-exporter) | Prometheus endpoint with PgBouncer metrics | [][pgbouncer-exporter-omnibus] | [][pgbouncer-exporter-charts] | [][pgbouncer-exporter-charts] | [](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE |
150
| [GitLab Monitor](#gitlab-monitor) | Generates a variety of GitLab metrics | [][gitlab-monitor-omnibus] | [][gitlab-monitor-charts] | [][gitlab-monitor-charts] | [](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE |
151
| [Node Exporter](#node-exporter) | Prometheus endpoint with system metrics | [][node-exporter-omnibus] | [][node-exporter-charts] | [][node-exporter-charts] | [](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE |
152
| [Mattermost](#mattermost) | Open-source Slack alternative | [][mattermost-omnibus] | [][mattermost-charts] | [][mattermost-charts] | [](../user/project/integrations/mattermost.md) | ❌ | ❌ | CE & EE |
153
| [MinIO](#minio) | Object storage service | [][minio-omnibus] | [][minio-charts] | [][minio-charts] | [](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#storage-architecture) | ❌ | [][minio-gdk] | CE & EE |
154
| [Runner](#gitlab-runner) | Executes GitLab CI jobs | [][runner-omnibus] | [][runner-charts] | [][runner-charts] | [](../user/gitlab_com/index.md#shared-runners) | [][runner-source] | [][runner-gdk] | CE & EE |
155 156
| [Database Migrations](#database-migrations) | Database migrations | [][database-migrations-omnibus] | [][database-migrations-charts] | [][database-migrations-charts] | ✅ | [][database-migrations-source] | ✅ | CE & EE |
| [Certificate Management](#certificate-management) | TLS Settings, Let's Encrypt | [][certificate-management-omnibus] | [][certificate-management-charts] | [][certificate-management-charts] | [](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#secrets-management) | [][certificate-management-source] | [][certificate-management-gdk] | CE & EE |
157
| [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | [][geo-omnibus] | [][geo-charts] | [][geo-charts] | ✅ | ❌ | [][geo-gdk] | EE Only |
158
| [LDAP Authentication](#ldap-authentication) | Authenticate users against centralized LDAP directory | [][ldap-omnibus] | [][ldap-charts] | [][ldap-charts] | [](https://about.gitlab.com/pricing/#gitlab-com) | [][gitlab-yml] | [][ldap-gdk] | CE & EE |
159 160
| [Outbound email (SMTP)](#outbound-email) | Send email messages to users | [][outbound-email-omnibus] | [][outbound-email-charts] | [][outbound-email-charts] | [](../user/gitlab_com/index.md#mail-configuration) | [][gitlab-yml] | [][gitlab-yml] | CE & EE |
| [Inbound email (SMTP)](#inbound-email) | Receive messages to update issues | [][inbound-email-omnibus] | [][inbound-email-charts] | [][inbound-email-charts] | [](../user/gitlab_com/index.md#mail-configuration) | [][gitlab-yml] | [][gitlab-yml] |  CE & EE |
161
| [Elasticsearch](#elasticsearch) | Improved search within GitLab | [][elasticsearch-omnibus] | [][elasticsearch-charts] | [][elasticsearch-charts] | [](https://gitlab.com/groups/gitlab-org/-/epics/153) | [][elasticsearch-source] | [][elasticsearch-gdk] | EE Only |
162 163
| [Sentry integration](#sentry) | Error tracking for deployed apps | [][sentry-integration] | [][sentry-integration] | [][sentry-integration] | [][sentry-integration] | [][sentry-integration] | [][sentry-integration] | CE & EE |
| [Jaeger integration](#jaeger) | Distributed tracing for deployed apps | [][jaeger-integration] | [][jaeger-integration] | [][jaeger-integration] | [][jaeger-integration] | [][jaeger-integration] | [][jaeger-integration] | EE Only |
164
| [GitLab Managed Apps](#gitlab-managed-apps) | Deploy [Helm](https://docs.helm.sh/), [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/), [Cert-Manager](https://docs.cert-manager.io/en/latest/), [Prometheus](https://prometheus.io/docs/introduction/overview/), a [Runner](https://docs.gitlab.com/runner/), [JupyterHub](http://jupyter.org/), [Knative](https://cloud.google.com/knative) to a cluster | [][managed-k8s-apps] | [][managed-k8s-apps] | [][managed-k8s-apps] | [][managed-k8s-apps] | [][managed-k8s-apps] | [][managed-k8s-apps] | CE & EE |
165 166 167 168

### Component details

This document is designed to be consumed by systems administrators and GitLab Support Engineers who want to understand more about the internals of GitLab and how they work together.
169

170
When deployed, GitLab should be considered the amalgamation of the below processes. When troubleshooting or debugging, be as specific as possible as to which component you are referencing. That should increase clarity and reduce confusion.
171

172
**Layers**
173 174 175 176 177 178 179 180

GitLab can be considered to have two layers from a process perspective:

- **Monitoring**: Anything from this layer is not required to deliver GitLab the application, but will allow administrators more insight into their infrastructure and what the service as a whole is doing.
- **Core**: Any process that is vital for the delivery of GitLab as a platform. If any of these processes halt there will be a GitLab outage. For the Core layer, you can further divide into:
  - **Processors**: These processes are responsible for actually performing operations and presenting the service.
  - **Data**: These services store/expose structured data for the GitLab service.

181
#### Alertmanager
182

183 184 185
- [Project page](https://github.com/prometheus/alertmanager/blob/master/README.md)
- Configuration: [Omnibus][alertmanager-omnibus], [Charts][alertmanager-charts]
- Layer: Monitoring
186
- Process: `alertmanager`
187

188
[Alert manager](https://prometheus.io/docs/alerting/alertmanager/) is a tool provided by Prometheus that _"handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts."_ You can read more in [issue gitlab-ce#45740](https://gitlab.com/gitlab-org/gitlab-ce/issues/45740) about what we will be alerting on.
189

190
#### Certificate management
191

192
- Project page: [Omnibus](https://github.com/certbot/certbot/blob/master/README.rst), [Charts](https://github.com/jetstack/cert-manager/blob/master/README.md)
193
- Configuration: [Omnibus][certificate-management-omnibus], [Charts][certificate-management-charts], [Source][certificate-management-source], [GDK][certificate-management-gdk]
194
- Layer: Core Service (Processor)
195

196 197 198 199 200 201 202
#### Consul

- [Project page](https://github.com/hashicorp/consul/blob/master/README.md)
- Configuration: [Omnibus][consul-omnibus], [Charts][consul-charts]
- Layer: Core Service (Data)

Consul is a tool for service discovery and configuration. Consul is distributed, highly available, and extremely scalable.
203

204
#### Database migrations
205

206
- Configuration: [Omnibus][registry-omnibus], [Charts][registry-charts], [Source][database-migrations-source]
207 208 209 210 211
- Layer: Core Service (Data)

#### Elasticsearch

- [Project page](https://github.com/elastic/elasticsearch/blob/master/README.textile)
212
- Configuration: [Omnibus][elasticsearch-omnibus], [Charts][elasticsearch-charts], [Source][elasticsearch-source], [GDK][elasticsearch-gdk]
213 214 215 216 217 218 219
- Layer: Core Service (Data)

Elasticsearch is a distributed RESTful search engine built for the cloud.

#### Gitaly

- [Project page](https://gitlab.com/gitlab-org/gitaly/blob/master/README.md)
220
- Configuration: [Omnibus][gitaly-omnibus], [Charts][gitaly-charts], [Source][gitaly-source]
221
- Layer: Core Service (Data)
222
- Process: `gitaly`
223

224
Gitaly is a service designed by GitLab to remove our need for NFS for Git storage in distributed deployments of GitLab (think GitLab.com or High Availability Deployments). As of 11.3.0, this service handles all Git level access in GitLab. You can read more about the project [in the project's readme](https://gitlab.com/gitlab-org/gitaly).
225

226 227
#### Gitlab Geo

228
- Configuration: [Omnibus][geo-omnibus], [Charts][geo-charts], [GDK][geo-gdk]
229
- Layer: Core Service (Processor)
230

231 232 233 234
#### Gitlab Monitor

- [Project page](https://gitlab.com/gitlab-org/gitlab-monitor)
- Configuration: [Omnibus][gitlab-monitor-omnibus], [Charts][gitlab-monitor-charts]
235
- Layer: Monitoring
236
- Process: `gitlab-monitor`
237

238
GitLab Monitor is a process designed in house that allows us to export metrics about GitLab application internals to Prometheus. You can read more [in the project's readme](https://gitlab.com/gitlab-org/gitlab-monitor).
239

240
#### Gitlab Pages
241

242
- Configuration: [Omnibus][pages-omnibus], [Charts][pages-charts], [Source][pages-source], [GDK][pages-gdk]
243 244 245 246 247 248 249 250 251
- Layer: Core Service (Processor)

GitLab Pages is a feature that allows you to publish static websites directly from a repository in GitLab.

You can use it either for personal or business websites, such as portfolios, documentation, manifestos, and business presentations. You can also attribute any license to your content.

#### Gitlab Runner

- [Project page](https://gitlab.com/gitlab-org/gitlab-runner/blob/master/README.md)
252
- Configuration: [Omnibus][runner-omnibus], [Charts][runner-charts], [Source][runner-source], [GDK][runner-gdk]
253 254 255 256 257 258 259 260 261
- Layer: Core Service (Processor)

GitLab Runner runs tests and sends the results to GitLab.

GitLab CI is the open-source continuous integration service included with GitLab that coordinates the testing. The old name of this project was GitLab CI Multi Runner but please use "GitLab Runner" (without CI) from now on.

#### Gitlab Shell

- [Project page](https://gitlab.com/gitlab-org/gitlab-shell/blob/master/README.md)
262
- Configuration: [Omnibus][shell-omnibus], [Charts][shell-charts], [Source][shell-source], [GDK][gitlab-yml]
263 264 265 266 267 268 269
- Layer: Core Service (Processor)

[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) is a program designed at GitLab to handle ssh-based `git` sessions, and modifies the list of authorized keys. GitLab Shell is not a Unix shell nor a replacement for Bash or Zsh.

#### Gitlab Workhorse

- [Project page](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/README.md)
270
- Configuration: [Omnibus][workhorse-omnibus], [Charts][workhorse-charts], [Source][workhorse-source]
271
- Layer: Core Service (Processor)
272
- Process: `gitlab-workhorse`
273

274
[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alleviate pressure from Unicorn. You can read more about the [historical reasons for developing](https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole.
275

276 277 278 279 280 281 282 283 284 285 286
#### Grafana

- [Project page](https://github.com/grafana/grafana/blob/master/README.md)
- Configuration: [Omnibus][grafana-omnibus], [Charts][grafana-charts]
- Layer: Monitoring

Grafana is an open source, feature rich metrics dashboard and graph editor for Graphite, Elasticsearch, OpenTSDB, Prometheus and InfluxDB.

#### Jaeger

- [Project page](https://github.com/jaegertracing/jaeger/blob/master/README.md)
287
- Configuration: [Omnibus][jaeger-omnibus], [Charts][jaeger-charts], [Source][jaeger-source], [GDK][jaeger-gdk]
288 289 290
- Layer: Monitoring

Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing system. It can be used for monitoring microservices-based distributed systems.
291

292 293 294 295
#### Logrotate

- [Project page](https://github.com/logrotate/logrotate/blob/master/README.md)
- Configuration: [Omnibus](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate)
296
- Layer: Core Service
297
- Process: `logrotate`
298

299
GitLab is comprised of a large number of services that all log. We started bundling our own logrotate as of 7.4 to make sure we were logging responsibly. This is just a packaged version of the common open source offering.
300

301 302 303 304 305 306
#### Mattermost

- [Project page](https://github.com/mattermost/mattermost-server/blob/master/README.md)
- Configuration: [Omnibus][mattermost-omnibus], [Charts][mattermost-charts]
- Layer: Core Service (Processor)

Evan Read's avatar
Evan Read committed
307
Mattermost is an open source, private cloud, Slack-alternative from <https://mattermost.com>.
308 309 310 311

#### MinIO

- [Project page](https://github.com/minio/minio/blob/master/README.md)
312
- Configuration: [Omnibus][minio-omnibus], [Charts][minio-charts], [GDK][minio-gdk]
313 314 315 316 317
- Layer: Core Service (Data)

MinIO is an object storage server released under Apache License v2.0. It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB.

#### NGINX
318

319
- Project page: [Omnibus](https://github.com/nginx/nginx), [Charts](https://github.com/kubernetes/ingress-nginx/blob/master/README.md)
320
- Configuration: [Omnibus][nginx-omnibus], [Charts][nginx-charts], [Source][nginx-source]
321
- Layer: Core Service (Processor)
322
- Process: `nginx`
323

Evan Read's avatar
Evan Read committed
324
Nginx as an ingress port for all HTTP requests and routes them to the appropriate sub-systems within GitLab. We are bundling an unmodified version of the popular open source webserver.
325

326
#### Node Exporter
327

328 329
- [Project page](https://github.com/prometheus/node_exporter/blob/master/README.md)
- Configuration: [Omnibus][node-exporter-omnibus], [Charts][node-exporter-charts]
330
- Layer: Monitoring
331
- Process: `node-exporter`
332

333
[Node Exporter](https://github.com/prometheus/node_exporter) is a Prometheus tool that gives us metrics on the underlying machine (think CPU/Disk/Load). It's just a packaged version of the common open source offering from the Prometheus project.
334

335 336 337 338 339
#### PgBouncer

- [Project page](https://github.com/pgbouncer/pgbouncer/blob/master/README.md)
- Configuration: [Omnibus][pgbouncer-omnibus], [Charts][pgbouncer-charts]
- Layer: Core Service (Data)
340

341 342 343 344 345 346
Lightweight connection pooler for PostgreSQL.

#### PgBouncer Exporter

- [Project page](https://github.com/stanhu/pgbouncer_exporter/blob/master/README.md)
- Configuration: [Omnibus][pgbouncer-exporter-omnibus], [Charts][pgbouncer-exporter-charts]
347 348
- Layer: Monitoring

349
Prometheus exporter for PgBouncer. Exports metrics at 9127/metrics.
350

351
#### Postgresql
352

353
- [Project page](https://github.com/postgres/postgres/blob/master/README)
354
- Configuration: [Omnibus][postgres-omnibus], [Charts][postgres-charts], [Source][postgres-source]
355
- Layer: Core Service (Data)
356
- Process: `postgresql`
357

358
GitLab packages the popular Database to provide storage for Application meta data and user information.
359

360
#### Postgres Exporter
361

362 363 364
- [Project page](https://github.com/wrouesnel/postgres_exporter/blob/master/README.md)
- Configuration: [Omnibus][postgres-exporter-omnibus], [Charts][postgres-exporter-charts]
- Layer: Monitoring
365
- Process: `postgres-exporter`
366 367 368 369 370 371 372

[Postgres-exporter](https://github.com/wrouesnel/postgres_exporter) is the community provided Prometheus exporter that will deliver data about Postgres to Prometheus for use in Grafana Dashboards.

#### Prometheus

- [Project page](https://github.com/prometheus/prometheus/blob/master/README.md)
- Configuration: [Omnibus][prometheus-omnibus], [Charts][prometheus-charts]
373
- Layer: Monitoring
374
- Process: `prometheus`
375 376 377

Prometheus is a time-series tool that helps GitLab administrators expose metrics about the individual processes used to provide GitLab the service.

378
#### Redis
379

380
- [Project page](https://github.com/antirez/redis/blob/unstable/README.md)
381
- Configuration: [Omnibus][redis-omnibus], [Charts][redis-charts], [Source][redis-source]
382
- Layer: Core Service (Data)
383
- Process: `redis`
384

385
Redis is packaged to provide a place to store:
386 387 388

- session data
- temporary cache information
389
- background job queues
390

391
#### Redis Exporter
392

393 394
- [Project page](https://github.com/oliver006/redis_exporter/blob/master/README.md)
- Configuration: [Omnibus][redis-exporter-omnibus], [Charts][redis-exporter-charts]
395
- Layer: Monitoring
396
- Process: `redis-exporter`
397

398
[Redis Exporter](https://github.com/oliver006/redis_exporter) is designed to give specific metrics about the Redis process to Prometheus so that we can graph these metrics in Grafana.
399

400 401 402
#### Registry

- [Project page](https://github.com/docker/distribution/blob/master/README.md)
403
- Configuration: [Omnibus][registry-omnibus], [Charts][registry-charts], [Source][registry-source], [GDK][registry-gdk]
404 405 406 407 408 409 410 411 412 413 414 415 416
- Layer: Core Service (Processor)

The registry is what users use to store their own Docker images. The bundled
registry uses nginx as a load balancer and GitLab as an authentication manager.
Whenever a client requests to pull or push an image from the registry, it will
return a `401` response along with a header detailing where to get an
authentication token, in this case the GitLab instance. The client will then
request a pull or push auth token from GitLab and retry the original request
to the registry. Learn more about [token authentication](https://docs.docker.com/registry/spec/auth/token/).

An external registry can also be configured to use GitLab as an auth endpoint.

#### Sentry
417

418
- [Project page](https://github.com/getsentry/sentry/blob/master/README.rst)
419
- Configuration: [Omnibus][sentry-omnibus], [Charts][sentry-charts], [Source][gitlab-yml], [GDK][gitlab-yml]
420 421 422 423 424 425 426
- Layer: Monitoring

Sentry fundamentally is a service that helps you monitor and fix crashes in realtime. The server is in Python, but it contains a full API for sending events from any language, in any application.

#### Sidekiq

- [Project page](https://github.com/mperham/sidekiq/blob/master/README.md)
427
- Configuration: [Omnibus][sidekiq-omnibus], [Charts][sidekiq-charts], [Source][gitlab-yml], [GDK][gitlab-yml]
428
- Layer: Core Service (Processor)
429
- Process: `sidekiq`
430 431 432

Sidekiq is a Ruby background job processor that pulls jobs from the redis queue and processes them. Background jobs allow GitLab to provide a faster request/response cycle by moving work into the background.

433
#### Unicorn
434

435
- [Project page](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/README.md)
436
- Configuration: [Omnibus][unicorn-omnibus], [Charts][unicorn-charts], [Source][unicorn-source], [GDK][gitlab-yml]
437
- Layer: Core Service (Processor)
438
- Process: `unicorn`
439

440
[Unicorn](https://bogomips.org/unicorn/) is a Ruby application server that is used to run the core Rails Application that provides the user facing features in GitLab. Often process output you will see this as `bundle` or `config.ru` depending on the GitLab version.
441

442
#### LDAP Authentication
443

444
- Configuration: [Omnibus][ldap-omnibus], [Charts][ldap-charts], [Source][gitlab-yml], [GDK][ldap-gdk]
445
- Layer: Core Service (Processor)
446

447
#### Outbound Email
448

449
- Configuration: [Omnibus][outbound-email-omnibus], [Charts][outbound-email-charts], [Source][gitlab-yml], [GDK][gitlab-yml]
450
- Layer: Core Service (Processor)
451

452
#### Inbound Email
453

454
- Configuration: [Omnibus][inbound-email-omnibus], [Charts][inbound-email-charts], [Source][gitlab-yml], [GDK][gitlab-yml]
455
- Layer: Core Service (Processor)
456

457
#### GitLab Managed Apps
458

459
- Configuration: [Omnibus][managed-k8s-apps], [Charts][managed-k8s-apps], [Source][managed-k8s-apps], [GDK][managed-k8s-apps]
460
- Layer: Core Service (Processor)
461

462
GitLab provides [GitLab Managed Apps](../user/project/clusters/index.md#installing-applications), a one-click install for various applications which can be added directly to your configured cluster. These applications are needed for Review Apps and deployments when using Auto DevOps. You can install them after you create a cluster.
463

464 465 466 467 468 469 470 471 472 473 474
## GitLab by Request Type

GitLab provides two "interfaces" for end users to access the service:

- Web HTTP Requests (Viewing the UI/API)
- Git HTTP/SSH Requests (Pushing/Pulling Git Data)

It's important to understand the distinction as some processes are used in both and others are exclusive to a specific request type.

### GitLab Web HTTP Request Cycle

475
When making a request to an HTTP Endpoint (think `/users/sign_in`) the request will take the following path through the GitLab Service:
476

477 478
- nginx - Acts as our first line reverse proxy.
- gitlab-workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Unicorn.
479
- unicorn - Since this is a web request, and it needs to access the application it will go to Unicorn.
Takuya Noguchi's avatar
Takuya Noguchi committed
480
- Postgres/Gitaly/Redis - Depending on the type of request, it may hit these services to store or retrieve data.
481 482 483 484 485 486

### GitLab Git Request Cycle

Below we describe the different pathing that HTTP vs. SSH Git requests will take. There is some overlap with the Web Request Cycle but also some differences.

### Web Request (80/443)
487

488 489 490
TODO

### SSH Request (22)
491

492
TODO
493

494
## System Layout
495

496
When referring to `~git` in the pictures it means the home directory of the git user which is typically `/home/git`.
497

498 499 500 501
GitLab is primarily installed within the `/home/git` user home directory as `git` user. Within the home directory is where the gitlabhq server software resides as well as the repositories (though the repository location is configurable).

The bare repositories are located in `/home/git/repositories`. GitLab is a ruby on rails application so the particulars of the inner workings can be learned by studying how a ruby on rails application works.

502
To serve repositories over SSH there's an add-on application called gitlab-shell which is installed in `/home/git/gitlab-shell`.
503

504 505 506
### Installation Folder Summary

To summarize here's the [directory structure of the `git` user home directory](../install/structure.md).
507

508
### Processes
509

510 511 512
```sh
ps aux | grep '^git'
```
513

514
GitLab has several components to operate. As a system user (i.e. any user that is not the `git` user) it requires a persistent database (MySQL/PostreSQL) and redis database. It also uses Apache httpd or Nginx to proxypass Unicorn. As the `git` user it starts Sidekiq and Unicorn (a simple ruby HTTP server running on port `8080` by default). Under the GitLab user there are normally 4 processes: `unicorn_rails master` (1 process), `unicorn_rails worker` (2 processes), `sidekiq` (1 process).
515

516
### Repository access
517

518
Repositories get accessed via HTTP or SSH. HTTP cloning/push/pull utilizes the GitLab API and SSH cloning is handled by gitlab-shell (previously explained).
519

520
## Troubleshooting
521

522
See the README for more information.
523

524
### Init scripts of the services
525

526
The GitLab init script starts and stops Unicorn and Sidekiq.
527 528

```
529
/etc/init.d/gitlab
530
Usage: service gitlab {start|stop|restart|reload|status}
531
```
532

533 534 535
Redis (key-value store/non-persistent database)

```
536
/etc/init.d/redis
537
Usage: /etc/init.d/redis {start|stop|status|restart|condrestart|try-restart}
538 539 540
```

SSH daemon
541

542
```
543
/etc/init.d/sshd
544 545 546
Usage: /etc/init.d/sshd {start|stop|restart|reload|force-reload|condrestart|try-restart|status}
```

547
Web server (one of the following)
548 549

```
550
/etc/init.d/httpd
551 552 553 554 555 556 557 558 559
Usage: httpd {start|stop|restart|condrestart|try-restart|force-reload|reload|status|fullstatus|graceful|help|configtest}

$ /etc/init.d/nginx
Usage: nginx {start|stop|restart|reload|force-reload|status|configtest}
```

Persistent database (one of the following)

```
560
/etc/init.d/mysqld
561 562 563 564 565 566
Usage: /etc/init.d/mysqld {start|stop|status|restart|condrestart|try-restart|reload|force-reload}

$ /etc/init.d/postgresql
Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [version ..]
```

567
### Log locations of the services
568 569 570

gitlabhq (includes Unicorn and Sidekiq logs)

571
- `/home/git/gitlab/log/` contains `application.log`, `production.log`, `sidekiq.log`, `unicorn.stdout.log`, `git_json.log` and `unicorn.stderr.log` normally.
572 573 574

gitlab-shell

575
- `/home/git/gitlab-shell/gitlab-shell.log`
576 577 578

ssh

579 580
- `/var/log/auth.log` auth log (on Ubuntu).
- `/var/log/secure` auth log (on RHEL).
581 582 583

nginx

584
- `/var/log/nginx/` contains error and access logs.
585 586 587

Apache httpd

588
- [Explanation of Apache logs](https://httpd.apache.org/docs/2.2/logs.html).
589 590
- `/var/log/apache2/` contains error and output logs (on Ubuntu).
- `/var/log/httpd/` contains error and output logs (on RHEL).
591 592 593

redis

594
- `/var/log/redis/redis.log` there are also log-rotated logs there.
595 596 597

PostgreSQL

598
- `/var/log/postgresql/*`
599 600 601

MySQL

602 603
- `/var/log/mysql/*`
- `/var/log/mysql.*`
604

605
### GitLab specific config files
606

607
GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced config files include:
608

609 610 611
- `gitlab.yml` - GitLab configuration.
- `unicorn.rb` - Unicorn web server settings.
- `database.yml` - Database connection settings.
612

613
gitlab-shell has a configuration file at `/home/git/gitlab-shell/config.yml`.
614

615
### Maintenance Tasks
616

617
[GitLab](https://gitlab.com/gitlab-org/gitlab-ce/tree/master) provides rake tasks with which you see version information and run a quick check on your configuration to ensure it is configured properly within the application. See [maintenance rake tasks](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/raketasks/maintenance.md).
618
In a nutshell, do the following:
619 620 621 622 623 624 625 626

```
sudo -i -u git
cd gitlab
bundle exec rake gitlab:env:info RAILS_ENV=production
bundle exec rake gitlab:check RAILS_ENV=production
```

627
Note: It is recommended to log into the `git` user using `sudo -i -u git` or `sudo su - git`. While the sudo commands provided by gitlabhq work in Ubuntu they do not always work in RHEL.
Sid Sijbrandij's avatar
Sid Sijbrandij committed
628 629 630

## GitLab.com

631
We've also detailed [our architecture of GitLab.com](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/) but this is probably over the top unless you have millions of users.
632 633 634

[alertmanager-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template
[alertmanager-charts]: https://github.com/helm/charts/tree/master/stable/prometheus
635 636 637
[nginx-omnibus]: https://docs.gitlab.com/omnibus/settings/
[nginx-charts]: https://docs.gitlab.com/charts/charts/nginx/
[nginx-source]: ../install/installation.md#9-nginx
638
[unicorn-omnibus]: https://docs.gitlab.com/omnibus/settings/unicorn.html
639 640
[unicorn-charts]: https://docs.gitlab.com/charts/charts/gitlab/unicorn/
[unicorn-source]: ../install/installation.md#configure-it
641
[gitlab-yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example
642
[sidekiq-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template
643 644 645 646
[sidekiq-charts]: https://docs.gitlab.com/charts/charts/gitlab/sidekiq/
[gitaly-omnibus]: ../administration/gitaly/index.md
[gitaly-charts]: https://docs.gitlab.com/charts/charts/gitlab/gitaly/
[gitaly-source]: ../install/installation.md#install-gitaly
647
[workhorse-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template
648 649
[workhorse-charts]: https://docs.gitlab.com/charts/charts/gitlab/unicorn/
[workhorse-source]: ../install/installation.md#install-gitlab-workhorse
650
[shell-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template
651 652 653
[shell-charts]: https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/
[shell-source]: ../install/installation.md#install-gitlab-shell
[pages-omnibus]: ../administration/pages/index.md
654
[pages-charts]: https://gitlab.com/charts/gitlab/issues/37
655
[pages-source]: ../install/installation.md#install-gitlab-pages
656
[pages-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/pages.md
657 658 659
[registry-omnibus]: ../administration/container_registry.md#container-registry-domain-configuration
[registry-charts]: https://docs.gitlab.com/charts/charts/registry/
[registry-source]: ../administration/container_registry.md#enable-the-container-registry
660
[registry-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/registry.md
661
[redis-omnibus]: https://docs.gitlab.com/omnibus/settings/redis.html
662 663
[redis-charts]: https://docs.gitlab.com/charts/charts/redis/
[redis-source]: ../install/installation.md#7-redis
664 665
[postgres-omnibus]: https://docs.gitlab.com/omnibus/settings/database.html
[postgres-charts]: https://github.com/helm/charts/tree/master/stable/postgresql
666 667
[postgres-source]: ../install/installation.md#6-database
[pgbouncer-omnibus]: ../administration/high_availability/pgbouncer.md
668
[pgbouncer-charts]: https://docs.gitlab.com/charts/installation/deployment.html#postgresql
669
[consul-omnibus]: ../administration/high_availability/consul.md
670
[consul-charts]: https://docs.gitlab.com/charts/installation/deployment.html#postgresql
671
[prometheus-omnibus]: ../administration/monitoring/prometheus/index.md
672
[prometheus-charts]: https://github.com/helm/charts/tree/master/stable/prometheus
673
[grafana-omnibus]: ../administration/monitoring/performance/grafana_configuration.md
674 675 676 677 678
[grafana-charts]: https://github.com/helm/charts/tree/master/stable/grafana
[sentry-omnibus]: https://docs.gitlab.com/omnibus/settings/configuration.html#error-reporting-and-logging-with-sentry
[sentry-charts]: https://gitlab.com/charts/gitlab/issues/1319
[jaeger-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4104
[jaeger-charts]: https://gitlab.com/charts/gitlab/issues/1320
679 680 681 682 683
[jaeger-source]: ../development/distributed_tracing.md#enabling-distributed-tracing
[jaeger-gdk]: ../development/distributed_tracing.html#using-jaeger-in-the-gitlab-development-kit
[redis-exporter-omnibus]: ../administration/monitoring/prometheus/redis_exporter.md
[redis-exporter-charts]: https://docs.gitlab.com/charts/charts/redis/
[postgres-exporter-omnibus]: ../administration/monitoring/prometheus/postgres_exporter.md
684
[postgres-exporter-charts]: https://github.com/helm/charts/tree/master/stable/postgresql
685
[pgbouncer-exporter-omnibus]: ../administration/monitoring/prometheus/pgbouncer_exporter.md
686
[pgbouncer-exporter-charts]: https://docs.gitlab.com/charts/installation/deployment.html#postgresql
687
[gitlab-monitor-omnibus]: ../administration/monitoring/prometheus/gitlab_monitor_exporter.md
688
[gitlab-monitor-charts]: https://docs.gitlab.com/charts/charts/gitlab/gitlab-monitor/index.html
689
[node-exporter-omnibus]: ../administration/monitoring/prometheus/node_exporter.md
690 691 692 693
[node-exporter-charts]: https://gitlab.com/charts/gitlab/issues/1332
[mattermost-omnibus]: https://docs.gitlab.com/omnibus/gitlab-mattermost/
[mattermost-charts]: https://docs.mattermost.com/install/install-mmte-helm-gitlab-helm.html
[minio-omnibus]: https://min.io/download
694
[minio-charts]: https://docs.gitlab.com/charts/charts/minio/
695
[minio-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/object_storage.md
696 697
[runner-omnibus]: https://docs.gitlab.com/runner/
[runner-charts]: https://docs.gitlab.com/runner/install/kubernetes.html
698 699
[runner-source]: https://docs.gitlab.com/runner/
[runner-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/runner.md
700
[database-migrations-omnibus]: https://docs.gitlab.com/omnibus/settings/database.html#disabling-automatic-database-migration
701
[database-migrations-charts]: https://docs.gitlab.com/charts/charts/gitlab/migrations/
702
[database-migrations-source]: ../update/upgrading_from_source.md#13-install-libs-migrations-etc
703 704
[certificate-management-omnibus]: https://docs.gitlab.com/omnibus/settings/ssl.html
[certificate-management-charts]: https://docs.gitlab.com/charts/installation/tls.html
705
[certificate-management-source]: ../install/installation.md#using-https
706
[certificate-management-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/https.md
707
[geo-omnibus]: ../administration/geo/replication/index.md#setup-instructions
708
[geo-charts]: https://gitlab.com/charts/gitlab/issues/8
709
[geo-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/geo.md
710
[ldap-omnibus]: ../administration/auth/ldap.md
711
[ldap-charts]: https://docs.gitlab.com/charts/charts/globals.html#ldap
712
[ldap-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/ldap.md
713 714
[outbound-email-omnibus]: https://docs.gitlab.com/omnibus/settings/smtp.html
[outbound-email-charts]: https://docs.gitlab.com/charts/installation/command-line-options.html#outgoing-email-configuration
715
[inbound-email-omnibus]: ../administration/incoming_email.md
716
[inbound-email-charts]: https://docs.gitlab.com/charts/installation/command-line-options.html#incoming-email-configuration
717 718 719
[elasticsearch-omnibus]: ../integration/elasticsearch.md
[elasticsearch-charts]: ../integration/elasticsearch.md
[elasticsearch-source]: ../integration/elasticsearch.md
720
[elasticsearch-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/elasticsearch.md
721 722 723
[sentry-integration]: ../user/project/operations/error_tracking.md
[jaeger-integration]: ../user/project/operations/tracing.md
[managed-k8s-apps]: ../user/project/clusters/index.md#installing-applications