Commit df8396be authored by John Cai's avatar John Cai Committed by Evan Read

Document praefect's separate_database_metrics setting

praefect['separate_database_metrics'] will put praefect metrics that run
queries on the database on a separate /db_metrics endpoint so that
customers can choose to configure a slower scrape interval. Also, with a
multiple Praefect setup, customers can configure only one Praefect node
to query the database.

Also add /db_metrics prometheus metrics to the gitaly/index.md.
parent a38ffa2a
......@@ -490,7 +490,15 @@ The following are useful queries for monitoring Gitaly:
### Monitor Gitaly Cluster
To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics:
To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics. There are two separate metrics
endpoints from which metrics can be scraped:
- The default `/metrics` endpoint.
- `/db_metrics`, which contains metrics that require database queries.
#### Default Prometheus `/metrics` endpoint
The following metrics are available from the `/metrics` endpoint:
- `gitaly_praefect_read_distribution`, a counter to track [distribution of reads](#distributed-reads).
It has two labels:
......@@ -523,6 +531,16 @@ To monitor [strong consistency](#strong-consistency), you can use the following
You can also monitor the [Praefect logs](../logs.md#praefect-logs).
#### Database metrics `/db_metrics` endpoint
The following metrics are available from the `/db_metrics` endpoint:
- `gitaly_praefect_unavailable_repositories`, the number of repositories that have no healthy, up to date replicas.
- `gitaly_praefect_read_only_repositories`, the number of repositories in read-only mode within a virtual storage.
This is an older metric that is still available for backwards compatibility reasons. `gitaly_praefect_unavailable_repositories`
is a more accurate.
- `gitaly_praefect_replication_queue_depth`, the number of jobs in the replication queue.
## Recover from failure
Gitaly Cluster can [recover from certain types of failure](recovery.md).
......
......@@ -418,10 +418,21 @@ On the **Praefect** node:
```ruby
praefect['listen_addr'] = '0.0.0.0:2305'
```
1. Configure Prometheus metrics by editing
`/etc/gitlab/gitlab.rb`:
```ruby
# Enable Prometheus metrics access to Praefect. You must use firewalls
# to restrict access to this address/port.
# The default metrics endpoint is /metrics
praefect['prometheus_listen_addr'] = '0.0.0.0:9652'
# Some metrics run queries against the database. Enabling separate database metrics allows
# these metrics to be collected when the metrics are
# scraped on a separate /db_metrics endpoint.
praefect['separate_database_metrics'] = true
```
1. Configure a strong `auth_token` for **Praefect** by editing
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment