Remove support for using Geo with an installation from source

Using Geo with source-based installs was deprecated in GitLab 11.5.

This commit removes this support now that the deprecation notice has
been in place for quite a few releases.
parent 773de162
---
title: Remove support for using Geo with an installation from source
merge_request: 28737
author:
type: other
...@@ -29,12 +29,7 @@ the node more time before scheduling a planned failover. ...@@ -29,12 +29,7 @@ the node more time before scheduling a planned failover.
Run the following commands in a Rails console on the **primary** node: Run the following commands in a Rails console on the **primary** node:
```sh ```sh
# Omnibus GitLab
gitlab-rails console gitlab-rails console
# Installation from source
cd /home/git/gitlab
sudo -u git -H bin/rails console RAILS_ENV=production
``` ```
To check if automatic background verification is enabled: To check if automatic background verification is enabled:
...@@ -102,12 +97,7 @@ disable if you need. Run the following commands in a Rails console on the ...@@ -102,12 +97,7 @@ disable if you need. Run the following commands in a Rails console on the
**primary** node: **primary** node:
```sh ```sh
# Omnibus GitLab
gitlab-rails console gitlab-rails console
# Installation from source
cd /home/git/gitlab
sudo -u git -H bin/rails console RAILS_ENV=production
``` ```
To disable automatic background re-verification: To disable automatic background re-verification:
...@@ -131,31 +121,15 @@ to be resynced without the backoff period: ...@@ -131,31 +121,15 @@ to be resynced without the backoff period:
For repositories: For repositories:
- Omnibus Installation ```sh
sudo gitlab-rake geo:verification:repository:reset
```sh ```
sudo gitlab-rake geo:verification:repository:reset
```
- Source Installation
```sh
sudo -u git -H bundle exec rake geo:verification:repository:reset RAILS_ENV=production
```
For wikis: For wikis:
- Omnibus Installation ```sh
sudo gitlab-rake geo:verification:wiki:reset
```sh ```
sudo gitlab-rake geo:verification:wiki:reset
```
- Source Installation
```sh
sudo -u git -H bundle exec rake geo:verification:wiki:reset RAILS_ENV=production
```
## Reconcile differences with checksum mismatches ## Reconcile differences with checksum mismatches
...@@ -169,7 +143,7 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch ...@@ -169,7 +143,7 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch
1. On the project admin page get the **Gitaly storage name**, and **Gitaly relative path**: 1. On the project admin page get the **Gitaly storage name**, and **Gitaly relative path**:
![Project admin page](img/checksum-differences-admin-project-page.png) ![Project admin page](img/checksum-differences-admin-project-page.png)
1. Navigate to the project's repository directory on both **primary** and **secondary** nodes. For an installation from source, the path is usually `/home/git/repositories`. For Omnibus installs, the path is usually `/var/opt/gitlab/git-data/repositories`. Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure. 1. Navigate to the project's repository directory on both **primary** and **secondary** nodes (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure.
```sh ```sh
cd /var/opt/gitlab/git-data/repositories cd /var/opt/gitlab/git-data/repositories
......
# Geo configuration (GitLab Omnibus) **[PREMIUM ONLY]** # Geo configuration **[PREMIUM ONLY]**
NOTE: **Note:**
This is the documentation for the Omnibus GitLab packages. For installations
from source, follow the [**Geo nodes configuration for installations
from source**][configuration-source] guide.
## Configuring a new **secondary** node ## Configuring a new **secondary** node
...@@ -303,7 +298,6 @@ See the [updating the Geo nodes document](updating_the_geo_nodes.md). ...@@ -303,7 +298,6 @@ See the [updating the Geo nodes document](updating_the_geo_nodes.md).
See the [troubleshooting document](troubleshooting.md). See the [troubleshooting document](troubleshooting.md).
[configuration-source]: configuration_source.md
[setup-geo-omnibus]: index.md#using-omnibus-gitlab [setup-geo-omnibus]: index.md#using-omnibus-gitlab
[Hashed Storage]: ../../repository_storage_types.md [Hashed Storage]: ../../repository_storage_types.md
[Disaster Recovery]: ../disaster_recovery/index.md [Disaster Recovery]: ../disaster_recovery/index.md
......
# Geo configuration (source) **[PREMIUM ONLY]**
NOTE: **Note:**
This documentation applies to GitLab source installations. In GitLab 11.5, this documentation was deprecated and will be removed in a future release.
Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html). For installations
using the Omnibus GitLab packages, follow the
[**Omnibus Geo nodes configuration**][configuration] guide.
## Configuring a new **secondary** node
NOTE: **Note:**
This is the final step in setting up a **secondary** node. Stages of the setup
process must be completed in the documented order. Before attempting the steps
in this stage, [complete all prior stages](index.md#using-gitlab-installed-from-source-deprecated).
The basic steps of configuring a **secondary** node are to:
- Replicate required configurations between the **primary** and **secondary** nodes.
- Configure a tracking database on each **secondary** node.
- Start GitLab on the **secondary** node.
You are encouraged to first read through all the steps before executing them
in your testing/production environment.
NOTE: **Note:**
**Do not** set up any custom authentication on **secondary** nodes, this will be handled by the **primary** node.
NOTE: **Note:**
**Do not** add anything in the **secondary** node's admin area (**Admin Area > Geo**). This is handled solely by the **primary** node.
### Step 1. Manually replicate secret GitLab values
GitLab stores a number of secret values in the `/home/git/gitlab/config/secrets.yml`
file which *must* match between the **primary** and **secondary** nodes. Until there is
a means of automatically replicating these between nodes (see [gitlab-org/gitlab-ee#3789]), they must
be manually replicated to **secondary** nodes.
1. SSH into the **primary** node, and execute the command below:
```sh
sudo cat /home/git/gitlab/config/secrets.yml
```
This will display the secrets that need to be replicated, in YAML format.
1. SSH into the **secondary** node and login as the `git` user:
```sh
sudo -i -u git
```
1. Make a backup of any existing secrets:
```sh
mv /home/git/gitlab/config/secrets.yml /home/git/gitlab/config/secrets.yml.`date +%F`
```
1. Copy `/home/git/gitlab/config/secrets.yml` from the **primary** node to the **secondary** node, or
copy-and-paste the file contents between nodes:
```sh
sudo editor /home/git/gitlab/config/secrets.yml
# paste the output of the `cat` command you ran on the primary
# save and exit
```
1. Ensure the file permissions are correct:
```sh
chown git:git /home/git/gitlab/config/secrets.yml
chmod 0600 /home/git/gitlab/config/secrets.yml
```
1. Restart GitLab
```sh
service gitlab restart
```
Once restarted, the **secondary** node will automatically start replicating missing data
from the **primary** node in a process known as backfill. Meanwhile, the **primary** node
will start to notify the **secondary** node of any changes, so that the **secondary** node can
act on those notifications immediately.
Make sure the **secondary** node is running and accessible. You can login to
the **secondary** node with the same credentials as used for the **primary** node.
### Step 2. Manually replicate the **primary** node's SSH host keys
Read [Manually replicate the **primary** node's SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys)
### Step 3. Add the **secondary** GitLab node
1. Navigate to the **primary** node's **Admin Area > Geo**
(`/admin/geo/nodes`) in your browser.
1. Add the **secondary** node by providing its full URL. **Do NOT** check the
**This is a primary node** checkbox.
1. Optionally, choose which namespaces should be replicated by the
**secondary** node. Leave blank to replicate all. Read more in
[selective synchronization](#selective-synchronization).
1. Click the **Add node** button.
1. SSH into your GitLab **secondary** server and restart the services:
```sh
service gitlab restart
```
Check if there are any common issue with your Geo setup by running:
```sh
bundle exec rake gitlab:geo:check
```
1. SSH into your GitLab **primary** server and login as root to verify the
**secondary** node is reachable or there are any common issue with your Geo setup:
```sh
bundle exec rake gitlab:geo:check
```
Once reconfigured, the **secondary** node will automatically start
replicating missing data from the **primary** node in a process known as backfill.
Meanwhile, the **primary** node will start to notify the **secondary** node of any changes, so
that the **secondary** node can act on those notifications immediately.
Make sure the **secondary** node is running and accessible.
You can log in to the **secondary** node with the same credentials as used for the **primary** node.
### Step 4. Enabling Hashed Storage
Read [Enabling Hashed Storage](configuration.md#step-4-enabling-hashed-storage).
### Step 5. (Optional) Configuring the secondary to trust the primary
You can safely skip this step if your **primary** node uses a CA-issued HTTPS certificate.
If your **primary** node is using a self-signed certificate for *HTTPS* support, you will
need to add that certificate to the **secondary** node's trust store. Retrieve the
certificate from the **primary** node and follow your distribution's instructions for
adding it to the **secondary** node's trust store. In Debian/Ubuntu, you would follow these steps:
```sh
sudo -i
cp <primary_node_certification_file> /usr/local/share/ca-certificates
update-ca-certificates
```
### Step 6. Enable Git access over HTTP/HTTPS
Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
method to be enabled. Navigate to **Admin Area > Settings**
(`/admin/application_settings`) on the **primary** node, and set
`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`.
### Step 7. Verify proper functioning of the secondary node
Read [Verify proper functioning of the secondary node][configuration-verify-node].
## Selective synchronization
Read [Selective synchronization][configuration-selective-replication].
## Troubleshooting
Read the [troubleshooting document][troubleshooting].
[gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789
[configuration]: configuration.md
[configuration-selective-replication]: configuration.md#selective-synchronization
[configuration-verify-node]: configuration.md#step-7-verify-proper-functioning-of-the-secondary-node
[troubleshooting]: troubleshooting.md
# Geo database replication (GitLab Omnibus) **[PREMIUM ONLY]** # Geo database replication **[PREMIUM ONLY]**
NOTE: **Note:** NOTE: **Note:**
This is the documentation for the Omnibus GitLab packages. For installations The following steps are for Omnibus installs only. Using Geo with source-based installs was **deprecated** in GitLab 11.5.
from source, follow the
[Geo database replication (source)](database_source.md) guide.
NOTE: **Note:** NOTE: **Note:**
If your GitLab installation uses external (not managed by Omnibus) PostgreSQL If your GitLab installation uses external (not managed by Omnibus) PostgreSQL
...@@ -102,10 +100,15 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -102,10 +100,15 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
else. else.
If you are using an external database not managed by Omnibus GitLab, you need If you are using an external database not managed by Omnibus GitLab, you need
to create the replicator user and define a password to it manually. to create the replicator user and define a password to it manually:
For information on how to create a replication user, refer to the
[appropriate step](database_source.md#step-1-configure-the-primary-server) ```sql
in [Geo database replication (source)](database_source.md). --- Create a new user 'replicator'
CREATE USER gitlab_replicator;
--- Set/change a password and grants replication privilege
ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>';
```
1. Configure PostgreSQL to listen on network interfaces: 1. Configure PostgreSQL to listen on network interfaces:
...@@ -340,7 +343,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ...@@ -340,7 +343,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
## ##
## Secondary address ## Secondary address
## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node
## ##
postgresql['listen_address'] = '<secondary_node_ip>' postgresql['listen_address'] = '<secondary_node_ip>'
postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']
...@@ -383,8 +386,7 @@ the database on the **primary** node, replicates the database, and creates the ...@@ -383,8 +386,7 @@ the database on the **primary** node, replicates the database, and creates the
needed files for streaming replication. needed files for streaming replication.
The directories used are the defaults that are set up in Omnibus. If you have The directories used are the defaults that are set up in Omnibus. If you have
changed any defaults or are using a source installation, configure it as you changed any defaults, configure it as you see fit replacing the directories and paths.
see fit replacing the directories and paths.
CAUTION: **Warning:** CAUTION: **Warning:**
Make sure to run this on the **secondary** server as it removes all PostgreSQL's Make sure to run this on the **secondary** server as it removes all PostgreSQL's
......
This diff is collapsed.
...@@ -186,30 +186,13 @@ If you installed GitLab using the Omnibus packages (highly recommended): ...@@ -186,30 +186,13 @@ If you installed GitLab using the Omnibus packages (highly recommended):
1. Optional: [Configure a secondary LDAP server](../../auth/ldap.md) for the **secondary** node. See [notes on LDAP](#ldap). 1. Optional: [Configure a secondary LDAP server](../../auth/ldap.md) for the **secondary** node. See [notes on LDAP](#ldap).
1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md). 1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md).
### Using GitLab installed from source (Deprecated)
NOTE: **Note:**
In GitLab 11.5, support for using Geo in GitLab source installations was deprecated and will be removed in a future release. Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html).
If you installed GitLab from source:
1. [Install GitLab Enterprise Edition](../../../install/installation.md) on the server that will serve as the **secondary** node. Do not create an account or log in to the new **secondary** node.
1. [Upload the GitLab License](https://docs.gitlab.com/ee/user/admin_area/license.html) on the **primary** node to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher.
1. [Set up the database replication](database_source.md) (`primary (read-write) <-> secondary (read-only)` topology).
1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). Do this step for **both** **primary** and **secondary** nodes.
1. [Configure GitLab](configuration_source.md) to set the **primary** and **secondary** nodes.
1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md).
## Post-installation documentation ## Post-installation documentation
After installing GitLab on the **secondary** nodes and performing the initial configuration, see the following documentation for post-installation information. After installing GitLab on the **secondary** nodes and performing the initial configuration, see the following documentation for post-installation information.
### Configuring Geo ### Configuring Geo
For information on configuring Geo, see: For information on configuring Geo, see [Geo configuration](configuration.md).
- [Geo configuration (GitLab Omnibus)](configuration.md).
- [Geo configuration (source)](configuration_source.md). Configuring Geo in GitLab source installations was **deprecated** in GitLab 11.5.
### Updating Geo ### Updating Geo
......
...@@ -120,9 +120,7 @@ questions from [owasp.org](https://www.owasp.org). ...@@ -120,9 +120,7 @@ questions from [owasp.org](https://www.owasp.org).
### What details regarding required OS components and lock‐down needs have been defined? ### What details regarding required OS components and lock‐down needs have been defined?
- The recommended installation method (Omnibus) packages most components itself. - The supported installation method (Omnibus) packages most components itself.
A from-source installation method exists. Both are documented at
<https://docs.gitlab.com/ee/administration/geo/replication/index.html>
- There are significant dependencies on the system-installed OpenSSH daemon (Geo - There are significant dependencies on the system-installed OpenSSH daemon (Geo
requires users to set up custom authentication methods) and the omnibus or requires users to set up custom authentication methods) and the omnibus or
system-provided PostgreSQL daemon (it must be configured to listen on TCP, system-provided PostgreSQL daemon (it must be configured to listen on TCP,
......
...@@ -310,7 +310,7 @@ same host on different ports. That is, 5432 and 5431 respectively. ...@@ -310,7 +310,7 @@ same host on different ports. That is, 5432 and 5431 respectively.
### Checking configuration ### Checking configuration
NOTE: **Note:** NOTE: **Note:**
The following steps are for Omnibus installs only. Using Geo with source-based installs [is deprecated](index.md#using-gitlab-installed-from-source-deprecated). The following steps are for Omnibus installs only. Using Geo with source-based installs was **deprecated** in GitLab 11.5.
To check the configuration: To check the configuration:
......
...@@ -337,6 +337,53 @@ is prepended with the relevant node for better clarity: ...@@ -337,6 +337,53 @@ is prepended with the relevant node for better clarity:
1. **[secondary]** Create the `replica.sh` script as described in the 1. **[secondary]** Create the `replica.sh` script as described in the
[database configuration document][database-source-replication]. [database configuration document][database-source-replication].
1. 1. **[secondary]** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the
embedded paths if necessary:
```
#!/bin/bash
PORT="5432"
USER="gitlab_replicator"
echo ---------------------------------------------------------------
echo WARNING: Make sure this script is run from the secondary server
echo ---------------------------------------------------------------
echo
echo Enter the IP or FQDN of the primary PostgreSQL server
read HOST
echo Enter the password for $USER@$HOST
read -s PASSWORD
echo Enter the required sslmode
read SSLMODE
echo Stopping PostgreSQL and all GitLab services
sudo service gitlab stop
sudo service postgresql stop
echo Backing up postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/
echo Cleaning up old cluster directory
sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data
echo Starting base backup as the replicator user
echo Enter the password for $USER@$HOST
sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P
echo Writing recovery.conf file
sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_
standby_mode = 'on'
primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE'
_EOF1_
"
echo Restoring postgresql.conf
sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/
echo Starting PostgreSQL
sudo service postgresql start
```
1. **[secondary]** Run the recovery script using the credentials from the 1. **[secondary]** Run the recovery script using the credentials from the
previous step: previous step:
...@@ -396,8 +443,6 @@ and it is required since 10.0. ...@@ -396,8 +443,6 @@ and it is required since 10.0.
[update]: ../../../update/README.md [update]: ../../../update/README.md
[database]: database.md [database]: database.md
[database-replication]: database.md#step-3-initiate-the-replication-process
[database-source-replication]: database_source.md#step-3-initiate-the-replication-process
[Hashed Storage]: ../../repository_storage_types.md [Hashed Storage]: ../../repository_storage_types.md
[hashed-migration]: ../../raketasks/storage.md [hashed-migration]: ../../raketasks/storage.md
[ssh-fast-lookup]: ../../operations/fast_ssh_key_lookup.md [ssh-fast-lookup]: ../../operations/fast_ssh_key_lookup.md
...@@ -135,7 +135,7 @@ Component statuses are linked to configuration documentation for each component. ...@@ -135,7 +135,7 @@ Component statuses are linked to configuration documentation for each component.
| [Runner](#gitlab-runner) | Executes GitLab CI jobs | [][runner-omnibus] | [][runner-charts] | [][runner-charts] | [](https://docs.gitlab.com/ee/user/gitlab_com/#shared-runners) | [][runner-source] | [][runner-gdk] | CE & EE | | [Runner](#gitlab-runner) | Executes GitLab CI jobs | [][runner-omnibus] | [][runner-charts] | [][runner-charts] | [](https://docs.gitlab.com/ee/user/gitlab_com/#shared-runners) | [][runner-source] | [][runner-gdk] | CE & EE |
| [Database Migrations](#database-migrations) | Database migrations | [][database-migrations-omnibus] | [][database-migrations-charts] | [][database-migrations-charts] | ✅ | [][database-migrations-source] | ✅ | CE & EE | | [Database Migrations](#database-migrations) | Database migrations | [][database-migrations-omnibus] | [][database-migrations-charts] | [][database-migrations-charts] | ✅ | [][database-migrations-source] | ✅ | CE & EE |
| [Certificate Management](#certificate-management) | TLS Settings, Let's Encrypt | [][certificate-management-omnibus] | [][certificate-management-charts] | [][certificate-management-charts] | [](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#secrets-management) | [][certificate-management-source] | [][certificate-management-gdk] | CE & EE | | [Certificate Management](#certificate-management) | TLS Settings, Let's Encrypt | [][certificate-management-omnibus] | [][certificate-management-charts] | [][certificate-management-charts] | [](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#secrets-management) | [][certificate-management-source] | [][certificate-management-gdk] | CE & EE |
| [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | [][geo-omnibus] | [][geo-charts] | [][geo-charts] | ✅ | [](../administration/geo/replication/configuration_source.md) | [][geo-gdk] | EE Only | | [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | [][geo-omnibus] | [][geo-charts] | [][geo-charts] | ✅ | | [][geo-gdk] | EE Only |
| [LDAP Authentication](#ldap-authentication) | Authenticate users against centralized LDAP directory | [][ldap-omnibus] | [][ldap-charts] | [][ldap-charts] | [](https://about.gitlab.com/pricing/#gitlab-com) | [][gitlab-yml] | [][ldap-gdk] | CE & EE | | [LDAP Authentication](#ldap-authentication) | Authenticate users against centralized LDAP directory | [][ldap-omnibus] | [][ldap-charts] | [][ldap-charts] | [](https://about.gitlab.com/pricing/#gitlab-com) | [][gitlab-yml] | [][ldap-gdk] | CE & EE |
| [Outbound email (SMTP)](#outbound-email) | Send email messages to users | [][outbound-email-omnibus] | [][outbound-email-charts] | [][outbound-email-charts] | [](https://docs.gitlab.com/ee/user/gitlab_com/#mail-configuration) | [][gitlab-yml] | [][gitlab-yml] | CE & EE | | [Outbound email (SMTP)](#outbound-email) | Send email messages to users | [][outbound-email-omnibus] | [][outbound-email-charts] | [][outbound-email-charts] | [](https://docs.gitlab.com/ee/user/gitlab_com/#mail-configuration) | [][gitlab-yml] | [][gitlab-yml] | CE & EE |
| [Inbound email (SMTP)](#inbound-email) | Receive messages to update issues | [][inbound-email-omnibus] | [][inbound-email-charts] | [][inbound-email-charts] | [](https://docs.gitlab.com/ee/user/gitlab_com/#mail-configuration) | [][gitlab-yml] | [][gitlab-yml] | CE & EE | | [Inbound email (SMTP)](#inbound-email) | Receive messages to update issues | [][inbound-email-omnibus] | [][inbound-email-charts] | [][inbound-email-charts] | [](https://docs.gitlab.com/ee/user/gitlab_com/#mail-configuration) | [][gitlab-yml] | [][gitlab-yml] | CE & EE |
......
--- ---
redirect_to: '../administration/geo/replication/configuration_source.md' redirect_to: '../administration/geo/replication/configuration.md'
--- ---
This document was moved to [another location](../administration/geo/replication/configuration_source.md). This document was moved to [another location](../administration/geo/replication/configuration.md).
--- ---
redirect_to: '../administration/geo/replication/database_source.md' redirect_to: '../administration/geo/replication/database.md'
--- ---
This document was moved to [another location](../administration/geo/replication/database_source.md). This document was moved to [another location](../administration/geo/replication/database.md).
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment