Commit bb605763 authored by Fiona Neill's avatar Fiona Neill

Merge branch 'eread/split-out-recovery-information' into 'master'

Separate recovery information from configuration information

See merge request gitlab-org/gitlab!75591
parents 45f883b4 480d53ff
......@@ -283,8 +283,7 @@ Gitaly Cluster provides the following features:
- [Replication factor](#replication-factor) of repositories for increased redundancy.
- [Automatic failover](praefect.md#automatic-failover-and-primary-election-strategies) from the
primary Gitaly node to secondary Gitaly nodes.
- Reporting of possible [data loss](praefect.md#check-for-data-loss) if replication queue is
non-empty.
- Reporting of possible [data loss](recovery.md#check-for-data-loss) if replication queue isn't empty.
Follow the [Gitaly Cluster epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) for improvements
including [horizontally distributing reads](https://gitlab.com/groups/gitlab-org/-/epics/2013).
......@@ -524,6 +523,10 @@ To monitor [strong consistency](#strong-consistency), you can use the following
You can also monitor the [Praefect logs](../logs.md#praefect-logs).
## Recover from failure
Gitaly Cluster can [recover from certain types of failure](recovery.md).
## Do not bypass Gitaly
GitLab doesn't advise directly accessing Gitaly repositories stored on disk with a Git client,
......
This diff is collapsed.
This diff is collapsed.
......@@ -376,7 +376,7 @@ Here are common errors and potential causes:
To determine the primary node of a repository:
- In GitLab 14.6 and later, use the [`praefect metadata`](praefect.md#retrieve-repository-metadata) subcommand.
- In GitLab 14.6 and later, use the [`praefect metadata`](#view-repository-metadata) subcommand.
- In GitLab 13.12 to GitLab 14.5 with [repository-specific primaries](praefect.md#repository-specific-primary-nodes),
use the [`gitlab:praefect:replicas` Rake task](../raketasks/praefect.md#replica-checksums).
- With legacy election strategies in GitLab 13.12 and earlier, the primary was the same for all repositories in a virtual storage.
......@@ -392,13 +392,97 @@ To determine the primary node of a repository:
curl localhost:9652/metrics | grep gitaly_praefect_primaries`
```
### View repository metadata
> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3481) in GitLab 14.6.
Gitaly Cluster maintains a [metadata database](index.md#components) about the repositories stored on the cluster. Use the `praefect metadata` subcommand
to inspect the metadata for troubleshooting.
You can retrieve a repository's metadata by its Praefect-assigned repository ID:
```shell
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -repository-id <repository-id>
```
You can also retrieve a repository's metadata by its virtual storage and relative path:
```shell
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -virtual-storage <virtual-storage> -relative-path <relative-path>
```
#### Examples
To retrieve the metadata for a repository with a Praefect-assigned repository ID of 1:
```shell
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -repository-id 1
```
To retrieve the metadata for a repository with virtual storage `default` and relative path `@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git`:
```shell
sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml metadata -virtual-storage default -relative-path @hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git
```
Either of these examples retrieve the following metadata for an example repository:
```plaintext
Repository ID: 54771
Virtual Storage: "default"
Relative Path: "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git"
Replica Path: "@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git"
Primary: "gitaly-1"
Generation: 1
Replicas:
- Storage: "gitaly-1"
Assigned: true
Generation: 1, fully up to date
Healthy: true
Valid Primary: true
- Storage: "gitaly-2"
Assigned: true
Generation: 0, behind by 1 changes
Healthy: true
Valid Primary: false
- Storage: "gitaly-3"
Assigned: true
Generation: replica not yet created
Healthy: false
Valid Primary: false
```
#### Available metadata
The metadata retrieved by `praefect metadata` includes the fields in the following tables.
| Field | Description |
|:------------------|:-------------------------------------------------------------------------------------------------------------------|
| `Repository ID` | Permanent unique ID assigned to the repository by Praefect. Different to the ID GitLab uses for repositories. |
| `Virtual Storage` | Name of the virtual storage the repository is stored in. |
| `Relative Path` | Repository's path in the virtual storage. |
| `Replica Path` | Where on the Gitaly node's disk the repository's replicas are stored. |
| `Primary` | Current primary of the repository. |
| `Generation` | Used by Praefect to track repository changes. Each write in the repository increments the repository's generation. |
| `Replicas` | A list of replicas that exist or are expected to exist. |
For each replica, the following metadata is available:
| `Replicas` Field | Description |
|:-----------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `Storage` | Name of the Gitaly storage that contains the replica. |
| `Assigned` | Indicates whether the replica is expected to exist in the storage. Can be `false` if a Gitaly node is removed from the cluster or if the storage contains an extra copy after the repository's replication factor was decreased. |
| `Generation` | Latest confirmed generation of the replica. It indicates:<br><br>- The replica is fully up to date if the generation matches the repository's generation.<br>- The replica is outdated if the replica's generation is less than the repository's generation.<br>- `replica not yet created` if the replica does not yet exist at all on the storage. |
| `Healthy` | Indicates whether the Gitaly node that is hosting this replica is considered healthy by the consensus of Praefect nodes. |
| `Valid Primary` | Indicates whether the replica is fit to serve as the primary node. If the repository's primary is not a valid primary, a failover occurs on the next write to the repository if there is another replica that is a valid primary. A replica is a valid primary if:<br><br>- It is stored on a healthy Gitaly node.<br>- It is fully up to date.<br>- It is not targeted by a pending deletion job from decreasing replication factor.<br>- It is assigned. |
### Check that repositories are in sync
Is [some cases](index.md#known-issues) the Praefect database can get out of sync with the underlying Gitaly nodes. To check that
a given repository is fully synced on all nodes, run the [`gitlab:praefect:replicas` Rake task](../raketasks/praefect.md#replica-checksums)
that checksums the repository on all Gitaly nodes.
The [Praefect dataloss](praefect.md#check-for-data-loss) command only checks the state of the repo in the Praefect database, and cannot
The [Praefect dataloss](recovery.md#check-for-data-loss) command only checks the state of the repo in the Praefect database, and cannot
be relied to detect sync problems in this scenario.
### Relation does not exist errors
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment