Skip to content

G
gitlab-ce
Commit 7d62ea23 authored by Nick Gaskill's avatar Nick Gaskill Committed by Achilleas Pipinellis
Browse files
Options

Put reference architecture instructions for 2k in one page

parent 72728242
Show whitespace changes
Inline Side-by-side
Showing with 1188 additions and 74 deletions
doc/administration/reference_architectures/2k_users.md
View file @ 7d62ea23
---
reading_time: true
---
# Reference architecture: up to 2,000 users # Reference architecture: up to 2,000 users
This page describes GitLab reference architecture for up to 2,000 users. This page describes GitLab reference architecture for up to 2,000 users.
...@@ -8,80 +12,861 @@ For a full list of reference architectures, see ...@@ -8,80 +12,861 @@ For a full list of reference architectures, see
> - **High Availability:** False > - **High Availability:** False
> - **Test RPS rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS > - **Test RPS rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS
| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS | Azure | | Service | Nodes | Configuration | GCP | AWS | Azure |
|--------------------------------------------------------------|-------|---------------------------------|---------------|-----------------------|----------------| |--------------------------------------------------------------|-----------|---------------------------------|---------------|-----------------------|----------------|
| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | | Load balancer | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 |
| Object Storage ([4](#footnotes)) | - | - | - | - | - | | Object Storage | n/a | n/a | n/a | n/a | n/a |
| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | | NFS Server (optional, not recommended) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 |
| PostgreSQL | 1 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | D2s v3 | | PostgreSQL | 1 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 |
| Redis ([3](#footnotes)) | 1 | 1 vCPU, 3.75GB Memory | `n1-standard-1` | `m5.large` | D2s v3 | | Redis | 1 | 1 vCPU, 3.75GB Memory | n1-standard-1 | m5.large | D2s v3 |
| Gitaly ([5](#footnotes)) ([7](#footnotes)) | X ([2](#footnotes)) | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | | Gitaly | 1 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 |
| GitLab Rails ([1](#footnotes)), Sidekiq | 2 | 8 vCPU, 7.2GB Memory | `n1-highcpu-8` | `c5.2xlarge` | F8s v2 | | GitLab Rails | 2 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | F8s v2 |
| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | | Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 |
The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms)
CPU platform on GCP. On different hardware you may find that adjustments, either lower
or higher, are required for your CPU or Node counts accordingly. For more information, a
[Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found
[here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks).
AWS-equivalent and Azure-equivalent configurations are rough suggestions
and may change in the future. They have not been tested and validated.
## Setup instructions For data objects such as LFS, Uploads, Artifacts, etc, an [object storage service](#configure-the-object-storage)
is recommended over NFS where possible, due to better performance and availability.
Since this doesn't require a node to be set up, it's marked as not applicable (n/a)
in the table above.
1. [Configure the external load balancing node](../high_availability/load_balancer.md) ## Setup components
To set up GitLab and its components to accommodate up to 2,000 users:
1. [Configure the external load balancing node](#configure-the-load-balancer)
that will handle the load balancing of the two GitLab application services nodes. that will handle the load balancing of the two GitLab application services nodes.
1. [Configure the Object Storage](../object_storage.md) ([4](#footnotes)) used for shared data objects. 1. [Configure the Object Storage](#configure-the-object-storage)
1. (Optional) [Configure NFS](../high_availability/nfs.md) to have used for shared data objects.
shared disk storage service as an alternative to Gitaly and/or 1. [Configure NFS (Optional)](#configure-nfs-optional)
[Object Storage](../object_storage.md) (although not recommended). to have shared disk storage service as an alternative to Gitaly and/or Object Storage (although
NFS is required for GitLab Pages, you can skip this step if you're not using that feature. not recommended). NFS is required for GitLab Pages, you can skip this step if you're not using
1. [Configure PostgreSQL](../high_availability/load_balancer.md), the database for GitLab. that feature.
1. [Configure Redis](../high_availability/redis.md). 1. [Configure PostgreSQL](#configure-postgresql),
1. [Configure Gitaly](../gitaly/index.md#run-gitaly-on-its-own-server), the database for GitLab.
which is used to provide access to the Git repositories. 1. [Configure Redis](#configure-redis).
1. [Configure the main GitLab Rails application](../high_availability/gitlab.md) 1. [Configure Gitaly](#configure-gitaly),
to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all which provides access to the Git repositories.
frontend requests (UI, API, Git over HTTP/SSH). 1. [Configure the main GitLab Rails application](#configure-gitlab-rails)
1. [Configure Prometheus](../high_availability/monitoring_node.md) to monitor your GitLab environment. to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend requests (UI, API, Git
over HTTP/SSH).
## Footnotes 1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab environment.
1. In our architectures we run each GitLab Rails node using the Puma webserver ## Configure the load balancer
and have its number of workers set to 90% of available CPUs along with four threads. For
nodes that are running Rails with other components the worker value should be reduced NOTE: **Note:**
accordingly where we've found 50% achieves a good balance but this is dependent This architecture has been tested and validated with [HAProxy](https://www.haproxy.org/)
on workload. as the load balancer. Although other load balancers with similar feature sets
could also be used, those load balancers have not been validated.
1. Gitaly node requirements are dependent on customer data, specifically the number of
projects and their sizes. We recommend two nodes as an absolute minimum for HA environments In an active/active GitLab configuration, you will need a load balancer to route
and at least four nodes should be used when supporting 50,000 or more users. traffic to the application servers. The specifics on which load balancer to use
We also recommend that each Gitaly node should store no more than 5TB of data or the exact configuration is beyond the scope of GitLab documentation. We hope
and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) that if you're managing multi-node systems like GitLab you have a load balancer of
set to 20% of available CPUs. Additional nodes should be considered in conjunction choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM,
with a review of expected data size and spread based on the recommendations above. and Citrix Net Scaler. This documentation will outline what ports and protocols
you need to use with GitLab.
1. Recommended Redis setup differs depending on the size of the architecture.
For smaller architectures (less than 3,000 users) a single instance should suffice. The next question is how you will handle SSL in your environment.
For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all There are several different options:
classes and that Redis Sentinel is hosted alongside Consul.
For larger architectures (10,000 users or more) we suggest running a separate - [The application node terminates SSL](#application-node-terminates-ssl).
[Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class - [The load balancer terminates SSL without backend SSL](#load-balancer-terminates-ssl-without-backend-ssl)
and another for the Queues and Shared State classes respectively. We also recommend and communication is not secure between the load balancer and the application node.
that you run the Redis Sentinel clusters separately for each Redis Cluster. - [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl)
and communication is *secure* between the load balancer and the application node.
1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md)
over NFS where possible, due to better performance and availability. ### Application node terminates SSL
1. NFS can be used as an alternative for both repository data (replacing Gitaly) and Configure your load balancer to pass connections on port 443 as `TCP` rather
object storage but this isn't typically recommended for performance reasons. Note however it is required for than `HTTP(S)` protocol. This will pass the connection to the application node's
[GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). NGINX service untouched. NGINX will have the SSL certificate and listen on port 443.
1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) See the [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https)
as the load balancer. Although other load balancers with similar feature sets for details on managing SSL certificates and configuring NGINX.
could also be used, those load balancers have not been validated.
### Load balancer terminates SSL without backend SSL
1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over
HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`.
as these components have heavy I/O. These IOPS values are recommended only as a starter The load balancer will then be responsible for managing SSL certificates and
as with time they may be adjusted higher or lower depending on the scale of your terminating SSL.
environment's workload. If you're running the environment on a Cloud provider
you may need to refer to their documentation on how configure IOPS correctly. Since communication between the load balancer and GitLab will not be secure,
there is some additional configuration needed. See the
1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) [NGINX proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl)
CPU platform on GCP. On different hardware you may find that adjustments, either lower for details.
or higher, are required for your CPU or Node counts accordingly. For more information, a
[Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found ### Load balancer terminates SSL with backend SSL
[here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks).
Configure your load balancer(s) to use the 'HTTP(S)' protocol rather than 'TCP'.
The load balancer(s) will be responsible for managing SSL certificates that
end users will see.
Traffic will also be secure between the load balancer(s) and NGINX in this
scenario. There is no need to add configuration for proxied SSL since the
connection will be secure all the way. However, configuration will need to be
added to GitLab to configure SSL certificates. See
[NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https)
for details on managing SSL certificates and configuring NGINX.
### Ports
The basic ports to be used are shown in the table below.
| LB Port | Backend Port | Protocol |
| ------- | ------------ | ------------------------ |
| 80 | 80 | HTTP (*1*) |
| 443 | 443 | TCP or HTTPS (*1*) (*2*) |
| 22 | 22 | TCP |
- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals) support requires
your load balancer to correctly handle WebSocket connections. When using
HTTP or HTTPS proxying, this means your load balancer must be configured
to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the
[web terminal](../integration/terminal.md) integration guide for
more details.
- (*2*): When using HTTPS protocol for port 443, you will need to add an SSL
certificate to the load balancers. If you wish to terminate SSL at the
GitLab application server instead, use TCP protocol.
If you're using GitLab Pages with custom domain support you will need some
additional port configurations.
GitLab Pages requires a separate virtual IP address. Configure DNS to point the
`pages_external_url` from `/etc/gitlab/gitlab.rb` at the new virtual IP address. See the
[GitLab Pages documentation](../pages/index.md) for more information.
| LB Port | Backend Port | Protocol |
| ------- | ------------- | --------- |
| 80 | Varies (*1*) | HTTP |
| 443 | Varies (*1*) | TCP (*2*) |
- (*1*): The backend port for GitLab Pages depends on the
`gitlab_pages['external_http']` and `gitlab_pages['external_https']`
setting. See [GitLab Pages documentation](../pages/index.md) for more details.
- (*2*): Port 443 for GitLab Pages should always use the TCP protocol. Users can
configure custom domains with custom SSL, which would not be possible
if SSL was terminated at the load balancer.
#### Alternate SSH Port
Some organizations have policies against opening SSH port 22. In this case,
it may be helpful to configure an alternate SSH hostname that allows users
to use SSH on port 443. An alternate SSH hostname will require a new virtual IP address
compared to the other GitLab HTTP configuration above.
Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`.
| LB Port | Backend Port | Protocol |
| ------- | ------------ | -------- |
| 443 | 22 | TCP |
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
## Configure the object storage
GitLab supports using an object storage service for holding numerous types of data.
It's recommended over [NFS](#configure-nfs-optional) and in general it's better
in larger setups as object storage is typically much more performant, reliable,
and scalable.
Object storage options that GitLab has tested, or is aware of customers using include:
- SaaS/Cloud solutions such as [Amazon S3](https://aws.amazon.com/s3/), [Google cloud storage](https://cloud.google.com/storage).
- On-premises hardware and appliances from various storage vendors.
- MinIO. There is [a guide to deploying this](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html) within our Helm Chart documentation.
For configuring GitLab to use Object Storage refer to the following guides
based on what features you intend to use:
1. Configure [object storage for backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage).
1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage)
including [incremental logging](../job_logs.md#new-incremental-logging-architecture).
1. Configure [object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage).
1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only).
1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage).
1. Configure [object storage for Container Registry](../packages/container_registry.md#container-registry-storage-driver) (optional feature).
1. Configure [object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature).
1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)**
1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)**
1. Configure [object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)**
1. Configure [object storage for autoscale Runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional - for improved performance).
1. Configure [object storage for Terraform state files](../terraform_state.md#using-object-storage-core-only).
Using separate buckets for each data type is the recommended approach for GitLab.
A limitation of our configuration is that each use of object storage is separately configured.
[We have an issue for improving this](https://gitlab.com/gitlab-org/gitlab/-/issues/23345)
and easily using one bucket with separate folders is one improvement that this might bring.
There is at least one specific issue with using the same bucket:
when GitLab is deployed with the Helm chart restore from backup
[will not properly function](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer)
unless separate buckets are used.
One risk of using a single bucket would be if your organization decided to
migrate GitLab to the Helm deployment in the future. GitLab would run, but the situation with
backups might not be realized until the organization had a critical requirement for the backups to
work.
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
## Configure NFS (optional)
[Object storage](#configure-the-object-storage), along with [Gitaly](#configure-gitaly)
are recommended over NFS wherever possible for improved performance. If you intend
to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pages-requires-nfs).
See how to [configure NFS](../high_availability/nfs.md).
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
## Configure PostgreSQL
In this section, you'll be guided through configuring an external PostgreSQL database
to be used with GitLab.
### Provide your own PostgreSQL instance
If you're hosting GitLab on a cloud provider, you can optionally use a
managed service for PostgreSQL. For example, AWS offers a managed Relational
Database Service (RDS) that runs PostgreSQL.
If you use a cloud-managed service, or provide your own PostgreSQL:
1. Set up PostgreSQL according to the
[database requirements document](../../install/requirements.md#database).
1. Set up a `gitlab` username with a password of your choice. The `gitlab` user
needs privileges to create the `gitlabhq_production` database.
1. Configure the GitLab application servers with the appropriate details.
This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails).
### Standalone PostgreSQL using Omnibus GitLab
1. SSH into the PostgreSQL server.
1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab
package you want using **steps 1 and 2** from the GitLab downloads page.
- Do not complete any other steps on the download page.
1. Generate a password hash for PostgreSQL. This assumes you will use the default
username of `gitlab` (recommended). The command will request a password
and confirmation. Use the value that is output by this command in the next
step as the value of `POSTGRESQL_PASSWORD_HASH`.
```shell
sudo gitlab-ctl pg-password-md5 gitlab
```
1. Edit `/etc/gitlab/gitlab.rb` and add the contents below, updating placeholder
values appropriately.
- `POSTGRESQL_PASSWORD_HASH` - The value output from the previous step
- `APPLICATION_SERVER_IP_BLOCKS` - A space delimited list of IP subnets or IP
addresses of the GitLab application servers that will connect to the
database. Example: `%w(123.123.123.123/32 123.123.123.234/32)`
```ruby
# Disable all components except PostgreSQL
roles ['postgres_role']
repmgr['enable'] = false
consul['enable'] = false
prometheus['enable'] = false
alertmanager['enable'] = false
pgbouncer_exporter['enable'] = false
redis_exporter['enable'] = false
gitlab_exporter['enable'] = false
# Set the network addresses that the exporters used for monitoring will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
postgres_exporter['listen_address'] = '0.0.0.0:9187'
postgres_exporter['dbname'] = 'gitlabhq_production'
postgres_exporter['password'] = 'POSTGRESQL_PASSWORD_HASH'
# Set the PostgreSQL address and port
postgresql['listen_address'] = '0.0.0.0'
postgresql['port'] = 5432
# Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value
postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH'
# Replace APPLICATION_SERVER_IP_BLOCK with the CIDR address of the application node
postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 APPLICATION_SERVER_IP_BLOCK)
# Disable automatic database migrations
gitlab_rails['auto_migrate'] = false
```
1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
1. Note the PostgreSQL node's IP address or hostname, port, and
plain text password. These will be necessary when configuring the [GitLab
application server](#configure-gitlab-rails) later.
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html)
are supported and can be added if needed.
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
## Configure Redis
In this section, you'll be guided through configuring an external Redis instance
to be used with GitLab.
### Provide your own Redis instance
Redis version 5.0 or higher is required, as this is what ships with
Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions
do not support an optional count argument to SPOP which is now required for
[Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md).
In addition, GitLab makes use of certain commands like `UNLINK` and `USAGE` which
were introduced only in Redis 4.
Managed Redis from cloud providers such as AWS ElastiCache will work. If these
services support high availability, be sure it is not the Redis Cluster type.
Note the Redis node's IP address or hostname, port, and password (if required).
These will be necessary when configuring the
[GitLab application servers](#configure-gitlab-rails) later.
### Standalone Redis using Omnibus GitLab
The Omnibus GitLab package can be used to configure a standalone Redis server.
The steps below are the minimum necessary to configure a Redis server with
Omnibus:
1. SSH into the Redis server.
1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab
package you want using **steps 1 and 2** from the GitLab downloads page.
- Do not complete any other steps on the download page.
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby
## Enable Redis
redis['enable'] = true
## Disable all other services
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
puma['enable'] = false
unicorn['enable'] = false
postgresql['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
alertmanager['enable'] = false
pgbouncer_exporter['enable'] = false
gitlab_exporter['enable'] = false
gitaly['enable'] = false
grafana['enable'] = false
redis['bind'] = '0.0.0.0'
redis['port'] = 6379
redis['password'] = 'SECRET_PASSWORD_HERE'
gitlab_rails['enable'] = false
# Set the network addresses that the exporters used for monitoring will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
redis_exporter['listen_address'] = '0.0.0.0:9121'
redis_exporter['flags'] = {
'redis.addr' => 'redis://0.0.0.0:6379',
'redis.password' => 'SECRET_PASSWORD_HERE',
}
```
1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
1. Note the Redis node's IP address or hostname, port, and
Redis password. These will be necessary when [configuring the GitLab
application servers](#configure-gitlab-rails) later.
Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html)
are supported and can be added if needed.
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
## Configure Gitaly
Deploying Gitaly in its own server can benefit GitLab installations that are
larger than a single machine. Gitaly node requirements are dependent on data,
specifically the number of projects and their sizes. It's recommended that each
Gitaly node store no more than 5TB of data. Your 2K setup may require one or more
nodes depending on your repository storage requirements.
We strongly recommend that all Gitaly nodes should be set up with SSD disks with a throughput of at least
8,000 IOPS for read operations and 2,000 IOPS for write, as Gitaly has heavy I/O.
These IOPS values are recommended only as a starter as with time they may be
adjusted higher or lower depending on the scale of your environment's workload.
If you're running the environment on a Cloud provider
you may need to refer to their documentation on how configure IOPS correctly.
Some things to note:
- The GitLab Rails application shards repositories into [repository storages](../repository_storage_paths.md).
- A Gitaly server can host one or more storages.
- A GitLab server can use one or more Gitaly servers.
- Gitaly addresses must be specified in such a way that they resolve
correctly for ALL Gitaly clients.
- Gitaly servers must not be exposed to the public internet, as Gitaly's network
traffic is unencrypted by default. The use of a firewall is highly recommended
to restrict access to the Gitaly server. Another option is to
[use TLS](#gitaly-tls-support).
TIP: **Tip:**
For more information about Gitaly's history and network architecture see the
[standalone Gitaly documentation](../gitaly/index.md).
Note: **Note:** The token referred to throughout the Gitaly documentation is
just an arbitrary password selected by the administrator. It is unrelated to
tokens created for the GitLab API or other similar web API tokens.
Below we describe how to configure one Gitaly server `gitaly1.internal` with
secret token `gitalysecret`. We assume your GitLab installation has two
repository storages: `default` and `storage1`.
To configure the Gitaly server:
1. [Download/Install](https://about.gitlab.com/install/) the Omnibus GitLab
package you want using **steps 1 and 2** from the GitLab downloads page but
**without** providing the `EXTERNAL_URL` value.
1. Edit `/etc/gitlab/gitlab.rb` to configure storage paths, enable
the network listener and configure the token:
<!--
updates to following example must also be made at
https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab
-->
```ruby
# /etc/gitlab/gitlab.rb
# Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests
# to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API.
# The following two values must be the same as their respective values
# of the GitLab Rails application setup
gitaly['auth_token'] = 'gitlaysecret'
gitlab_shell['secret_token'] = 'shellsecret'
# Avoid running unnecessary services on the Gitaly server
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
puma['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
grafana['enable'] = false
# If you run a seperate monitoring node you can disable these services
alertmanager['enable'] = false
prometheus['enable'] = false
# Prevent database connections during 'gitlab-ctl reconfigure'
gitlab_rails['rake_cache_clear'] = false
gitlab_rails['auto_migrate'] = false
# Configure the gitlab-shell API callback URL. Without this, `git push` will
# fail. This can be your 'front door' GitLab URL or an internal load
# balancer.
# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server.
gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
# Make Gitaly accept connections on all network interfaces. You must use
# firewalls to restrict access to this address/port.
# Comment out following line if you only want to support TLS connections
gitaly['listen_addr'] = "0.0.0.0:8075"
gitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
# Set the network addresses that the exporters used for monitoring will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
```
1. Append the following to `/etc/gitlab/gitlab.rb` on `gitaly1.internal`:
```ruby
git_data_dirs({
'default' => {
'path' => '/var/opt/gitlab/git-data'
},
'storage1' => {
'path' => '/mnt/gitlab/git-data'
},
})
```
<!--
updates to following example must also be made at
https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab
-->
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
1. Confirm that Gitaly can perform callbacks to the internal API:
```shell
sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml
```
### Gitaly TLS support
Gitaly supports TLS encryption. To be able to communicate
with a Gitaly instance that listens for secure connections you will need to use `tls://` URL
scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration.
You will need to bring your own certificates as this isn't provided automatically.
The certificate, or its certificate authority, must be installed on all Gitaly
nodes (including the Gitaly node using the certificate) and on all client nodes
that communicate with it following the procedure described in
[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates).
NOTE: **Note**
The self-signed certificate must specify the address you use to access the
Gitaly server. If you are addressing the Gitaly server by a hostname, you can
either use the Common Name field for this, or add it as a Subject Alternative
Name. If you are addressing the Gitaly server by its IP address, you must add it
as a Subject Alternative Name to the certificate.
[gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691).
NOTE: **Note:**
It is possible to configure Gitaly servers with both an
unencrypted listening address `listen_addr` and an encrypted listening
address `tls_listen_addr` at the same time. This allows you to do a
gradual transition from unencrypted to encrypted traffic, if necessary.
To configure Gitaly with TLS:
1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there:
```shell
sudo mkdir -p /etc/gitlab/ssl
sudo chmod 755 /etc/gitlab/ssl
sudo cp key.pem cert.pem /etc/gitlab/ssl/
sudo chmod 644 key.pem cert.pem
```
1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when
calling into itself:
```shell
sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/
```
1. Edit `/etc/gitlab/gitlab.rb` and add:
<!--
updates to following example must also be made at
https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab
-->
```ruby
gitaly['tls_listen_addr'] = "0.0.0.0:9999"
gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
gitaly['key_path'] = "/etc/gitlab/ssl/key.pem"
```
1. Delete `gitaly['listen_addr']` to allow only encrypted connections.
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
## Configure GitLab Rails
NOTE: **Note:**
In our architectures we run each GitLab Rails node using the Puma webserver
and have its number of workers set to 90% of available CPUs along with four threads. For
nodes that are running Rails with other components the worker value should be reduced
accordingly where we've found 50% achieves a good balance but this is dependent
on workload.
This section describes how to configure the GitLab application (Rails) component.
On each node perform the following:
1. If you're [using NFS](#configure-nfs-optional):
1. If necessary, install the NFS client utility packages using the following
commands:
```shell
# Ubuntu/Debian
apt-get install nfs-common
# CentOS/Red Hat
yum install nfs-utils nfs-utils-lib
```
1. Specify the necessary NFS mounts in `/etc/fstab`.
The exact contents of `/etc/fstab` will depend on how you chose
to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md)
for examples and the various options.
1. Create the shared directories. These may be different depending on your NFS
mount locations.
```shell
mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data
```
1. Download/install Omnibus GitLab using **steps 1 and 2** from
[GitLab downloads](https://about.gitlab.com/install/). Do not complete other
steps on the download page.
1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration.
To maintain uniformity of links across nodes, the `external_url`
on the application server should point to the external URL that users will use
to access GitLab. This would be the URL of the [load balancer](#configure-the-load-balancer)
which will route traffic to the GitLab application server:
```ruby
external_url 'https://gitlab.example.com'
# Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests
# to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API.
# The following two values must be the same as their respective values
# of the Gitaly setup
gitlab_rails['gitaly_token'] = 'gitalyecret'
gitlab_shell['secret_token'] = 'shellsecret'
git_data_dirs({
'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' },
'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' },
'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' },
})
## Disable components that will not be on the GitLab application server
roles ['application_role']
gitaly['enable'] = false
nginx['enable'] = true
## PostgreSQL connection details
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server
gitlab_rails['db_password'] = 'DB password'
## Redis connection details
gitlab_rails['redis_port'] = '6379'
gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server
gitlab_rails['redis_password'] = 'Redis Password'
# Set the network addresses that the exporters used for monitoring will listen on
node_exporter['listen_address'] = '0.0.0.0:9100'
gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229'
sidekiq['listen_address'] = "0.0.0.0"
puma['listen'] = '0.0.0.0'
# Add the monitoring node's IP address to the monitoring whitelist and allow it to
# scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with
# the address and/or subnets gathered from the monitoring node
gitlab_rails['monitoring_whitelist'] = ['<MONITOR NODE IP>/32', '127.0.0.0/8']
nginx['status']['options']['allow'] = ['<MONITOR NODE IP>/32', '127.0.0.0/8']
## Uncomment and edit the following options if you have set up NFS
##
## Prevent GitLab from starting if NFS data mounts are not available
##
#high_availability['mountpoint'] = '/var/opt/gitlab/git-data'
##
## Ensure UIDs and GIDs match between servers for permissions via NFS
##
#user['uid'] = 9000
#user['gid'] = 9000
#web_server['uid'] = 9001
#web_server['gid'] = 9001
#registry['uid'] = 9002
#registry['gid'] = 9002
```
1. If you're using [Gitaly with TLS support](#gitaly-tls-support), make sure the
`git_data_dirs` entry is configured with `tls` instead of `tcp`:
```ruby
git_data_dirs({
'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' },
'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' },
'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' },
})
```
1. Copy the cert into `/etc/gitlab/trusted-certs`:
```shell
sudo cp cert.pem /etc/gitlab/trusted-certs/
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly.
1. Tail the logs to see the requests:
```shell
sudo gitlab-ctl tail gitaly
```
NOTE: **Note:** When you specify `https` in the `external_url`, as in the example
above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If
certificates are not present, NGINX will fail to start. See the
[NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https)
for more information.
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
## Configure Prometheus
The Omnibus GitLab package can be used to configure a standalone Monitoring node
running [Prometheus](../monitoring/prometheus/index.md) and
[Grafana](../monitoring/performance/grafana_configuration.md):
1. SSH into the Monitoring node.
1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab
package you want using **steps 1 and 2** from the GitLab downloads page.
Do not complete any other steps on the download page.
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
```ruby
external_url 'http://gitlab.example.com'
# Enable Prometheus
prometheus['enable'] = true
prometheus['listen_address'] = '0.0.0.0:9090'
prometheus['monitor_kubernetes'] = false
# Enable Login form
grafana['disable_login_form'] = false
# Enable Grafana
grafana['enable'] = true
grafana['admin_password'] = 'toomanysecrets'
# Disable all other services
gitlab_rails['auto_migrate'] = false
alertmanager['enable'] = false
gitaly['enable'] = false
gitlab_exporter['enable'] = false
gitlab_workhorse['enable'] = false
nginx['enable'] = true
postgres_exporter['enable'] = false
postgresql['enable'] = false
redis['enable'] = false
redis_exporter['enable'] = false
sidekiq['enable'] = false
puma['enable'] = false
unicorn['enable'] = false
node_exporter['enable'] = false
gitlab_exporter['enable'] = false
```
1. Prometheus also needs some scrape configs to pull all the data from the various
nodes where we configured exporters. Assuming that your nodes' IPs are:
```plaintext
1.1.1.1: postgres
1.1.1.2: redis
1.1.1.3: gitaly1
1.1.1.4: rails1
1.1.1.5: rails2
```
Add the following to `/etc/gitlab/gitlab.rb`:
```ruby
prometheus['scrape_configs'] = [
{
'job_name': 'postgres',
'static_configs' => [
'targets' => ['1.1.1.1:9187'],
],
},
{
'job_name': 'redis',
'static_configs' => [
'targets' => ['1.1.1.2:9121'],
],
},
{
'job_name': 'gitaly',
'static_configs' => [
'targets' => ['1.1.1.3:9236'],
],
},
{
'job_name': 'gitlab-nginx',
'static_configs' => [
'targets' => ['1.1.1.4:8060', '1.1.1.5:8060'],
],
},
{
'job_name': 'gitlab-workhorse',
'static_configs' => [
'targets' => ['1.1.1.4:9229', '1.1.1.5:9229'],
],
},
{
'job_name': 'gitlab-rails',
'metrics_path': '/-/metrics',
'static_configs' => [
'targets' => ['1.1.1.4:8080', '1.1.1.5:8080'],
],
},
{
'job_name': 'gitlab-sidekiq',
'static_configs' => [
'targets' => ['1.1.1.4:8082', '1.1.1.5:8082'],
],
},
{
'job_name': 'node',
'static_configs' => [
'targets' => ['1.1.1.1:9100', '1.1.1.2:9100', '1.1.1.3:9100', '1.1.1.4:9100', '1.1.1.5:9100'],
],
},
]
```
1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to
`http[s]://<MONITOR NODE>/-/grafana`
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
## Troubleshooting
See the [troubleshooting documentation](troubleshooting.md).
<div align="right">
<a type="button" class="btn btn-default" href="#setup-components">
Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i>
</a>
</div>
doc/administration/reference_architectures/troubleshooting.md 0 → 100644
View file @ 7d62ea23
# Troubleshooting a reference architecture set up
This page serves as the troubleshooting documentation if you followed one of
the [reference architectures](index.md#reference-architectures).
## Troubleshooting object storage
### S3 API compatibility issues
Not all S3 providers [are fully compatible](../../raketasks/backup_restore.md#other-s3-providers)
with the Fog library that GitLab uses. Symptoms include:
```plaintext
411 Length Required
```
### GitLab Pages requires NFS
If you intend to use [GitLab Pages](../../user/project/pages/index.md), this currently requires
[NFS](../high_availability/nfs.md). There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196)
to remove this dependency. In the future, GitLab Pages may use
[object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/208135).
The dependency on disk storage also prevents Pages being deployed using the
[GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/37).
### Incremental logging is required for CI to use object storage
If you configure GitLab to use object storage for CI logs and artifacts,
[you must also enable incremental logging](../job_logs.md#new-incremental-logging-architecture).
### Proxy Download
A number of the use cases for object storage allow client traffic to be redirected to the
object storage back end, like when Git clients request large files via LFS or when
downloading CI artifacts and logs.
When the files are stored on local block storage or NFS, GitLab has to act as a proxy.
With object storage, the default behavior is for GitLab to redirect to the object
storage device rather than proxy the request.
The `proxy_download` setting controls this behavior: the default is generally `false`.
Verify this in the documentation for each use case. Set it to `true` to make
GitLab proxy the files rather than redirect.
When not proxying files, GitLab returns an
[HTTP 302 redirect with a pre-signed, time-limited object storage URL](https://gitlab.com/gitlab-org/gitlab/-/issues/32117#note_218532298).
This can result in some of the following problems:
- If GitLab is using non-secure HTTP to access the object storage, clients may generate
`https->http` downgrade errors and refuse to process the redirect. The solution to this
is for GitLab to use HTTPS. LFS, for example, will generate this error:
```plaintext
LFS: lfsapi/client: refusing insecure redirect, https->http
```
- Clients will need to trust the certificate authority that issued the object storage
certificate, or may return common TLS errors such as:
```plaintext
x509: certificate signed by unknown authority
```
- Clients will need network access to the object storage. Errors that might result
if this access is not in place include:
```plaintext
Received status code 403 from server: Forbidden
```
### ETag mismatch
Using the default GitLab settings, some object storage back-ends such as
[MinIO](https://gitlab.com/gitlab-org/gitlab/-/issues/23188)
and [Alibaba](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1564)
might generate `ETag mismatch` errors.
When using GitLab direct upload, the
[workaround for MinIO](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1564#note_244497658)
is to use the `--compat` parameter on the server.
We are working on a fix to GitLab component Workhorse, and also
a workaround, in the mean time, to
[allow ETag verification to be disabled](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18175).
## Troubleshooting Redis
If the application node cannot connect to the Redis node, check your firewall rules and
make sure Redis can accept TCP connections under port `6379`.
## Troubleshooting Gitaly
### Checking versions when using standalone Gitaly nodes
When using standalone Gitaly nodes, you must make sure they are the same version
as GitLab to ensure full compatibility. Check **Admin Area > Gitaly Servers** on
your GitLab instance and confirm all Gitaly Servers are `Up to date`.
![Gitaly standalone software versions diagram](../gitaly/img/gitlab_gitaly_version_mismatch_v12_4.png)
### `gitaly-debug`
The `gitaly-debug` command provides "production debugging" tools for Gitaly and Git
performance. It is intended to help production engineers and support
engineers investigate Gitaly performance problems.
If you're using GitLab 11.6 or newer, this tool should be installed on
your GitLab / Gitaly server already at `/opt/gitlab/embedded/bin/gitaly-debug`.
If you're investigating an older GitLab version you can compile this
tool offline and copy the executable to your server:
```shell
git clone https://gitlab.com/gitlab-org/gitaly.git
cd cmd/gitaly-debug
GOOS=linux GOARCH=amd64 go build -o gitaly-debug
```
To see the help page of `gitaly-debug` for a list of supported sub-commands, run:
```shell
gitaly-debug -h
```
### Commits, pushes, and clones return a 401
```plaintext
remote: GitLab: 401 Unauthorized
```
You will need to sync your `gitlab-secrets.json` file with your GitLab
app nodes.
### Client side gRPC logs
Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC
client has its own log file which may contain useful information when
you are seeing Gitaly errors. You can control the log level of the
gRPC client with the `GRPC_LOG_LEVEL` environment variable. The
default level is `WARN`.
You can run a gRPC trace with:
```shell
sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check
```
### Observing `gitaly-ruby` traffic
[`gitaly-ruby`](../gitaly/index.md#gitaly-ruby) is an internal implementation detail of Gitaly,
so, there's not that much visibility into what goes on inside
`gitaly-ruby` processes.
If you have Prometheus set up to scrape your Gitaly process, you can see
request rates and error codes for individual RPCs in `gitaly-ruby` by
querying `grpc_client_handled_total`. Strictly speaking, this metric does
not differentiate between `gitaly-ruby` and other RPCs, but in practice
(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal
calls from the main Gitaly process to one of its `gitaly-ruby` sidecars.
Assuming your `grpc_client_handled_total` counter only observes Gitaly,
the following query shows you RPCs are (most likely) internally
implemented as calls to `gitaly-ruby`:
```prometheus
sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0
```
### Repository changes fail with a `401 Unauthorized` error
If you're running Gitaly on its own server and notice that users can
successfully clone and fetch repositories (via both SSH and HTTPS), but can't
push to them or make changes to the repository in the web UI without getting a
`401 Unauthorized` message, then it's possible Gitaly is failing to authenticate
with the other nodes due to having the wrong secrets file.
Confirm the following are all true:
- When any user performs a `git push` to any repository on this Gitaly node, it
fails with the following error (note the `401 Unauthorized`):
```shell
remote: GitLab: 401 Unauthorized
To <REMOTE_URL>
! [remote rejected] branch-name -> branch-name (pre-receive hook declined)
error: failed to push some refs to '<REMOTE_URL>'
```
- When any user adds or modifies a file from the repository using the GitLab
UI, it immediately fails with a red `401 Unauthorized` banner.
- Creating a new project and [initializing it with a README](../../gitlab-basics/create-project.md#blank-projects)
successfully creates the project but doesn't create the README.
- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors
when reaching the `/api/v4/internal/allowed` endpoint:
```shell
# api_json.log
{
"time": "2019-07-18T00:30:14.967Z",
"severity": "INFO",
"duration": 0.57,
"db": 0,
"view": 0.57,
"status": 401,
"method": "POST",
"path": "\/api\/v4\/internal\/allowed",
"params": [
{
"key": "action",
"value": "git-receive-pack"
},
{
"key": "changes",
"value": "REDACTED"
},
{
"key": "gl_repository",
"value": "REDACTED"
},
{
"key": "project",
"value": "\/path\/to\/project.git"
},
{
"key": "protocol",
"value": "web"
},
{
"key": "env",
"value": "{\"GIT_ALTERNATE_OBJECT_DIRECTORIES\":[],\"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE\":[],\"GIT_OBJECT_DIRECTORY\":null,\"GIT_OBJECT_DIRECTORY_RELATIVE\":null}"
},
{
"key": "user_id",
"value": "2"
},
{
"key": "secret_token",
"value": "[FILTERED]"
}
],
"host": "gitlab.example.com",
"ip": "REDACTED",
"ua": "Ruby",
"route": "\/api\/:version\/internal\/allowed",
"queue_duration": 4.24,
"gitaly_calls": 0,
"gitaly_duration": 0,
"correlation_id": "XPUZqTukaP3"
}
# nginx_access.log
[IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby"
```
To fix this problem, confirm that your `gitlab-secrets.json` file
on the Gitaly node matches the one on all other nodes. If it doesn't match,
update the secrets file on the Gitaly node to match the others, then
[reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure).
### Command line tools cannot connect to Gitaly
If you are having trouble connecting to a Gitaly node with command line (CLI) tools, and certain actions result in a `14: Connect Failed` error message, it means that gRPC cannot reach your Gitaly node.
Verify that you can reach Gitaly via TCP:
```shell
sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT]
```
If the TCP connection fails, check your network settings and your firewall rules. If the TCP connection succeeds, your networking and firewall rules are correct.
If you use proxy servers in your command line environment, such as Bash, these can interfere with your gRPC traffic.
If you use Bash or a compatible command line environment, run the following commands to determine whether you have proxy servers configured:
```shell
echo $http_proxy
echo $https_proxy
```
If either of these variables have a value, your Gitaly CLI connections may be getting routed through a proxy which cannot connect to Gitaly.
To remove the proxy setting, run the following commands (depending on which variables had values):
```shell
unset http_proxy
unset https_proxy
```
### Gitaly not listening on new address after reconfiguring
When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` values, Gitaly may continue to listen on the old address after a `sudo gitlab-ctl reconfigure`.
When this occurs, performing a `sudo gitlab-ctl restart` will resolve the issue. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/issues/2521) is resolved.
### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly node
If this error occurs even though file permissions are correct, it's likely that
the Gitaly node is experiencing
[clock drift](https://en.wikipedia.org/wiki/Clock_drift).
Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time
server to keep them synchronized if possible.
## Troubleshooting the GitLab Rails application
- `mount: wrong fs type, bad option, bad superblock on`
You have not installed the necessary NFS client utilities. See step 1 above.
- `mount: mount point /var/opt/gitlab/... does not exist`
This particular directory does not exist on the NFS server. Ensure
the share is exported and exists on the NFS server and try to remount.
## Troubleshooting Monitoring
If the monitoring node is not receiving any data, check that the exporters are
capturing data.
```shell
curl http[s]://localhost:<EXPORTER LISTENING PORT>/metric
```
or
```shell
curl http[s]://localhost:<EXPORTER LISTENING PORT>/-/metric
```
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7