Automatic merge of gitlab-org/gitlab-ce master

changelogs/unreleased/11609-geo-remove-support-for-using-geo-with-an-installation-from-source-docs.yml

+---
+title: Remove support for using Geo with an installation from source
+merge_request: 28737
+author:
+type: other

doc/administration/geo/disaster_recovery/background_verification.md

@@ -29,12 +29,7 @@ the node more time before scheduling a planned failover.
 Run the following commands in a Rails console on the **primary** node:
 
 ```sh
-# Omnibus GitLab
 gitlab-rails console
-
-# Installation from source
-cd /home/git/gitlab
-sudo -u git -H bin/rails console RAILS_ENV=production
 ```
 
 To check if automatic background verification is enabled:
@@ -102,12 +97,7 @@ disable if you need. Run the following commands in a Rails console on the
 **primary** node:
 
 ```sh
-# Omnibus GitLab
 gitlab-rails console
-
-# Installation from source
-cd /home/git/gitlab
-sudo -u git -H bin/rails console RAILS_ENV=production
 ```
 
 To disable automatic background re-verification:
@@ -131,31 +121,15 @@ to be resynced without the backoff period:
 
 For repositories:
 
-- Omnibus Installation
-
-    ```sh
-    sudo gitlab-rake geo:verification:repository:reset
-    ```
-
-- Source Installation
-
-    ```sh
-    sudo -u git -H bundle exec rake geo:verification:repository:reset RAILS_ENV=production
-    ```
+```sh
+sudo gitlab-rake geo:verification:repository:reset
+```
 
 For wikis:
 
-- Omnibus Installation
-
-    ```sh
-    sudo gitlab-rake geo:verification:wiki:reset
-    ```
-
-- Source Installation
-
-    ```sh
-    sudo -u git -H bundle exec rake geo:verification:wiki:reset RAILS_ENV=production
-    ```
+```sh
+sudo gitlab-rake geo:verification:wiki:reset
+```
 
 ## Reconcile differences with checksum mismatches
 
@@ -169,7 +143,7 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch
 1. On the project admin page get the **Gitaly storage name**, and **Gitaly relative path**:
    ![Project admin page](img/checksum-differences-admin-project-page.png)
 
-1. Navigate to the project's repository directory on both **primary** and **secondary** nodes. For an installation from source, the path is usually `/home/git/repositories`. For Omnibus installs, the path is usually `/var/opt/gitlab/git-data/repositories`. Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure.
+1. Navigate to the project's repository directory on both **primary** and **secondary** nodes (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure.
 
     ```sh
     cd /var/opt/gitlab/git-data/repositories

doc/administration/geo/replication/configuration.md

-# Geo configuration (GitLab Omnibus) **[PREMIUM ONLY]**
-
-NOTE: **Note:**
-This is the documentation for the Omnibus GitLab packages. For installations
-from source, follow the [**Geo nodes configuration for installations
-from source**][configuration-source] guide.
+# Geo configuration **[PREMIUM ONLY]**
 
 ## Configuring a new **secondary** node
 
@@ -303,7 +298,6 @@ See the [updating the Geo nodes document](updating_the_geo_nodes.md).
 
 See the [troubleshooting document](troubleshooting.md).
 
-[configuration-source]: configuration_source.md
 [setup-geo-omnibus]: index.md#using-omnibus-gitlab
 [Hashed Storage]: ../../repository_storage_types.md
 [Disaster Recovery]: ../disaster_recovery/index.md

doc/administration/geo/replication/configuration_source.md

-# Geo configuration (source) **[PREMIUM ONLY]**
-
-NOTE: **Note:**
-This documentation applies to GitLab source installations. In GitLab 11.5, this documentation was deprecated and will be removed in a future release.
-Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html). For installations
-using the Omnibus GitLab packages, follow the
-[**Omnibus Geo nodes configuration**][configuration] guide.
-
-## Configuring a new **secondary** node
-
-NOTE: **Note:**
-This is the final step in setting up a **secondary** node. Stages of the setup
-process must be completed in the documented order. Before attempting the steps
-in this stage, [complete all prior stages](index.md#using-gitlab-installed-from-source-deprecated).
-
-The basic steps of configuring a **secondary** node are to:
-
-- Replicate required configurations between the **primary** and **secondary** nodes.
-- Configure a tracking database on each **secondary** node.
-- Start GitLab on the **secondary** node.
-
-You are encouraged to first read through all the steps before executing them
-in your testing/production environment.
-
-NOTE: **Note:**
-**Do not** set up any custom authentication on **secondary** nodes, this will be handled by the **primary** node.
-
-NOTE: **Note:**
-**Do not** add anything in the **secondary** node's admin area (**Admin Area > Geo**). This is handled solely by the **primary** node.
-
-### Step 1. Manually replicate secret GitLab values
-
-GitLab stores a number of secret values in the `/home/git/gitlab/config/secrets.yml`
-file which *must* match between the **primary** and **secondary** nodes. Until there is
-a means of automatically replicating these between nodes (see [gitlab-org/gitlab-ee#3789]), they must
-be manually replicated to **secondary** nodes.
-
-1. SSH into the **primary** node, and execute the command below:
-
-    ```sh
-    sudo cat /home/git/gitlab/config/secrets.yml
-    ```
-
-    This will display the secrets that need to be replicated, in YAML format.
-
-1. SSH into the **secondary** node and login as the `git` user:
-
-    ```sh
-    sudo -i -u git
-    ```
-
-1. Make a backup of any existing secrets:
-
-    ```sh
-    mv /home/git/gitlab/config/secrets.yml /home/git/gitlab/config/secrets.yml.`date +%F`
-    ```
-
-1. Copy `/home/git/gitlab/config/secrets.yml` from the **primary** node to the **secondary** node, or
-   copy-and-paste the file contents between nodes:
-
-    ```sh
-    sudo editor /home/git/gitlab/config/secrets.yml
-
-    # paste the output of the `cat` command you ran on the primary
-    # save and exit
-    ```
-
-1. Ensure the file permissions are correct:
-
-    ```sh
-    chown git:git /home/git/gitlab/config/secrets.yml
-    chmod 0600 /home/git/gitlab/config/secrets.yml
-    ```
-
-1. Restart GitLab
-
-    ```sh
-    service gitlab restart
-    ```
-
-Once restarted, the **secondary** node will automatically start replicating missing data
-from the **primary** node in a process known as backfill. Meanwhile, the **primary** node
-will start to notify the **secondary** node of any changes, so that the **secondary** node can
-act on those notifications immediately.
-
-Make sure the **secondary** node is running and accessible. You can login to
-the **secondary** node with the same credentials as used for the **primary** node.
-
-### Step 2. Manually replicate the **primary** node's SSH host keys
-
-Read [Manually replicate the **primary** node's SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys)
-
-### Step 3. Add the **secondary** GitLab node
-
-1. Navigate to the **primary** node's **Admin Area > Geo**
-   (`/admin/geo/nodes`) in your browser.
-1. Add the **secondary** node by providing its full URL. **Do NOT** check the
-   **This is a primary node** checkbox.
-1. Optionally, choose which namespaces should be replicated by the
-   **secondary** node. Leave blank to replicate all. Read more in
-   [selective synchronization](#selective-synchronization).
-1. Click the **Add node** button.
-1. SSH into your GitLab **secondary** server and restart the services:
-
-    ```sh
-    service gitlab restart
-    ```
-
-    Check if there are any common issue with your Geo setup by running:
-
-    ```sh
-    bundle exec rake gitlab:geo:check
-    ```
-
-1. SSH into your GitLab **primary** server and login as root to verify the
-   **secondary** node is reachable or there are any common issue with your Geo setup:
-
-    ```sh
-    bundle exec rake gitlab:geo:check
-    ```
-
-Once reconfigured, the **secondary** node will automatically start
-replicating missing data from the **primary** node in a process known as backfill.
-Meanwhile, the **primary** node will start to notify the **secondary** node of any changes, so
-that the **secondary** node can act on those notifications immediately.
-
-Make sure the **secondary** node is running and accessible.
-You can log in to the **secondary** node with the same credentials as used for the **primary** node.
-
-### Step 4. Enabling Hashed Storage
-
-Read [Enabling Hashed Storage](configuration.md#step-4-enabling-hashed-storage).
-
-### Step 5. (Optional) Configuring the secondary to trust the primary
-
-You can safely skip this step if your **primary** node uses a CA-issued HTTPS certificate.
-
-If your **primary** node is using a self-signed certificate for *HTTPS* support, you will
-need to add that certificate to the **secondary** node's trust store. Retrieve the
-certificate from the **primary** node and follow your distribution's instructions for
-adding it to the **secondary** node's trust store. In Debian/Ubuntu, you would follow these steps:
-
-```sh
-sudo -i
-cp <primary_node_certification_file> /usr/local/share/ca-certificates
-update-ca-certificates
-```
-
-### Step 6. Enable Git access over HTTP/HTTPS
-
-Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone
-method to be enabled. Navigate to **Admin Area > Settings**
-(`/admin/application_settings`) on the **primary** node, and set
-`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`.
-
-### Step 7. Verify proper functioning of the secondary node
-
-Read [Verify proper functioning of the secondary node][configuration-verify-node].
-
-## Selective synchronization
-
-Read [Selective synchronization][configuration-selective-replication].
-
-## Troubleshooting
-
-Read the [troubleshooting document][troubleshooting].
-
-[gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789
-[configuration]: configuration.md
-[configuration-selective-replication]: configuration.md#selective-synchronization
-[configuration-verify-node]: configuration.md#step-7-verify-proper-functioning-of-the-secondary-node
-[troubleshooting]: troubleshooting.md

doc/administration/geo/replication/database.md

-# Geo database replication (GitLab Omnibus) **[PREMIUM ONLY]**
+# Geo database replication **[PREMIUM ONLY]**
 
 NOTE: **Note:**
-This is the documentation for the Omnibus GitLab packages. For installations
-from source, follow the
-[Geo database replication (source)](database_source.md) guide.
+The following steps are for Omnibus installs only. Using Geo with source-based installs was **deprecated** in GitLab 11.5.
 
 NOTE: **Note:**
 If your GitLab installation uses external (not managed by Omnibus) PostgreSQL
@@ -102,10 +100,15 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
     else.
 
     If you are using an external database not managed by Omnibus GitLab, you need
-    to create the replicator user and define a password to it manually.
-    For information on how to create a replication user, refer to the
-    [appropriate step](database_source.md#step-1-configure-the-primary-server)
-    in [Geo database replication (source)](database_source.md).
+    to create the replicator user and define a password to it manually:
+
+    ```sql
+    --- Create a new user 'replicator'
+    CREATE USER gitlab_replicator;
+
+    --- Set/change a password and grants replication privilege
+    ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>';
+    ```
 
 1. Configure PostgreSQL to listen on network interfaces:
 
@@ -340,7 +343,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o
 
     ##
     ## Secondary address
-    ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node 
+    ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node
     ##
     postgresql['listen_address'] = '<secondary_node_ip>'
     postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32']
@@ -383,8 +386,7 @@ the database on the **primary** node, replicates the database, and creates the
 needed files for streaming replication.
 
 The directories used are the defaults that are set up in Omnibus. If you have
-changed any defaults or are using a source installation, configure it as you
-see fit replacing the directories and paths.
+changed any defaults, configure it as you see fit replacing the directories and paths.
 
 CAUTION: **Warning:**
 Make sure to run this on the **secondary** server as it removes all PostgreSQL's

doc/administration/geo/replication/index.md

@@ -186,30 +186,13 @@ If you installed GitLab using the Omnibus packages (highly recommended):
 1. Optional: [Configure a secondary LDAP server](../../auth/ldap.md) for the **secondary** node. See [notes on LDAP](#ldap).
 1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md).
 
-### Using GitLab installed from source (Deprecated)
-
-NOTE: **Note:**
-In GitLab 11.5, support for using Geo in GitLab source installations was deprecated and will be removed in a future release. Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html).
-
-If you installed GitLab from source:
-
-1. [Install GitLab Enterprise Edition](../../../install/installation.md) on the server that will serve as the **secondary** node. Do not create an account or log in to the new **secondary** node.
-1. [Upload the GitLab License](https://docs.gitlab.com/ee/user/admin_area/license.html) on the **primary** node to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher.
-1. [Set up the database replication](database_source.md) (`primary (read-write) <-> secondary (read-only)` topology).
-1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). Do this step for **both** **primary** and **secondary** nodes.
-1. [Configure GitLab](configuration_source.md) to set the **primary** and **secondary** nodes.
-1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md).
-
 ## Post-installation documentation
 
 After installing GitLab on the **secondary** nodes and performing the initial configuration, see the following documentation for post-installation information.
 
 ### Configuring Geo
 
-For information on configuring Geo, see:
-
-- [Geo configuration (GitLab Omnibus)](configuration.md).
-- [Geo configuration (source)](configuration_source.md). Configuring Geo in GitLab source installations was **deprecated** in GitLab 11.5.
+For information on configuring Geo, see [Geo configuration](configuration.md).
 
 ### Updating Geo
 

doc/administration/geo/replication/security_review.md

@@ -120,9 +120,7 @@ questions from [owasp.org](https://www.owasp.org).
 
 ### What details regarding required OS components and lock‐down needs have been defined?
 
-- The recommended installation method (Omnibus) packages most components itself.
-  A from-source installation method exists. Both are documented at
-  <https://docs.gitlab.com/ee/administration/geo/replication/index.html>
+- The supported installation method (Omnibus) packages most components itself.
 - There are significant dependencies on the system-installed OpenSSH daemon (Geo
   requires users to set up custom authentication methods) and the omnibus or
   system-provided PostgreSQL daemon (it must be configured to listen on TCP,

doc/administration/geo/replication/troubleshooting.md

@@ -310,7 +310,7 @@ same host on different ports. That is, 5432 and 5431 respectively.
 ### Checking configuration
 
 NOTE: **Note:**
-The following steps are for Omnibus installs only. Using Geo with source-based installs [is deprecated](index.md#using-gitlab-installed-from-source-deprecated).
+The following steps are for Omnibus installs only. Using Geo with source-based installs was **deprecated** in GitLab 11.5.
 
 To check the configuration:
 

doc/administration/geo/replication/updating_the_geo_nodes.md

@@ -337,6 +337,53 @@ is prepended with the relevant node for better clarity:
     1. **[secondary]** Create the `replica.sh` script as described in the
        [database configuration document][database-source-replication].
 
+    1. 1. **[secondary]** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the
+       embedded paths if necessary:
+
+        ```
+        #!/bin/bash
+
+        PORT="5432"
+        USER="gitlab_replicator"
+        echo ---------------------------------------------------------------
+        echo WARNING: Make sure this script is run from the secondary server
+        echo ---------------------------------------------------------------
+        echo
+        echo Enter the IP or FQDN of the primary PostgreSQL server
+        read HOST
+        echo Enter the password for $USER@$HOST
+        read -s PASSWORD
+        echo Enter the required sslmode
+        read SSLMODE
+
+        echo Stopping PostgreSQL and all GitLab services
+        sudo service gitlab stop
+        sudo service postgresql stop
+
+        echo Backing up postgresql.conf
+        sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/
+
+        echo Cleaning up old cluster directory
+        sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data
+
+        echo Starting base backup as the replicator user
+        echo Enter the password for $USER@$HOST
+        sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P
+
+        echo Writing recovery.conf file
+        sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_
+          standby_mode = 'on'
+          primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE'
+        _EOF1_
+        "
+
+        echo Restoring postgresql.conf
+        sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/
+
+        echo Starting PostgreSQL
+        sudo service postgresql start
+        ```
+
     1. **[secondary]** Run the recovery script using the credentials from the
        previous step:
 
@@ -396,8 +443,6 @@ and it is required since 10.0.
 
 [update]: ../../../update/README.md
 [database]: database.md
-[database-replication]: database.md#step-3-initiate-the-replication-process
-[database-source-replication]: database_source.md#step-3-initiate-the-replication-process
 [Hashed Storage]: ../../repository_storage_types.md
 [hashed-migration]: ../../raketasks/storage.md
 [ssh-fast-lookup]: ../../operations/fast_ssh_key_lookup.md

doc/development/architecture.md

@@ -135,7 +135,7 @@ Component statuses are linked to configuration documentation for each component.
 | [Runner](#gitlab-runner) | Executes GitLab CI jobs | [⤓][runner-omnibus] | [✅][runner-charts] | [⚙][runner-charts] | [✅](../user/gitlab_com/index.md#shared-runners) | [⚙][runner-source] | [⚙][runner-gdk] | CE & EE |
 | [Database Migrations](#database-migrations) | Database migrations | [✅][database-migrations-omnibus] | [✅][database-migrations-charts] | [✅][database-migrations-charts] | ✅ | [⚙][database-migrations-source] | ✅ | CE & EE |
 | [Certificate Management](#certificate-management) | TLS Settings, Let's Encrypt | [✅][certificate-management-omnibus] | [✅][certificate-management-charts] | [⚙][certificate-management-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#secrets-management) | [⚙][certificate-management-source] | [⚙][certificate-management-gdk] | CE & EE |
-| [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | [⚙][geo-omnibus] | [❌][geo-charts] | [❌][geo-charts] | ✅ | [❌](../administration/geo/replication/configuration_source.md) | [⚙][geo-gdk] | EE Only |
+| [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | [⚙][geo-omnibus] | [❌][geo-charts] | [❌][geo-charts] | ✅ | ❌ | [⚙][geo-gdk] | EE Only |
 | [LDAP Authentication](#ldap-authentication) | Authenticate users against centralized LDAP directory | [⤓][ldap-omnibus] | [⤓][ldap-charts] | [⤓][ldap-charts] | [❌](https://about.gitlab.com/pricing/#gitlab-com) | [⤓][gitlab-yml] | [⤓][ldap-gdk] | CE & EE |
 | [Outbound email (SMTP)](#outbound-email) | Send email messages to users | [⤓][outbound-email-omnibus] | [⤓][outbound-email-charts] | [⤓][outbound-email-charts] | [✅](../user/gitlab_com/index.md#mail-configuration) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE |
 | [Inbound email (SMTP)](#inbound-email) | Receive messages to update issues | [⤓][inbound-email-omnibus] | [⤓][inbound-email-charts] | [⤓][inbound-email-charts] | [✅](../user/gitlab_com/index.md#mail-configuration) | [⤓][gitlab-yml] | [⤓][gitlab-yml] |  CE & EE |

doc/gitlab-geo/configuration_source.md

 ---
-redirect_to: '../administration/geo/replication/configuration_source.md'
+redirect_to: '../administration/geo/replication/configuration.md'
 ---
 
-This document was moved to [another location](../administration/geo/replication/configuration_source.md).
+This document was moved to [another location](../administration/geo/replication/configuration.md).

doc/gitlab-geo/database_source.md

 ---
-redirect_to: '../administration/geo/replication/database_source.md'
+redirect_to: '../administration/geo/replication/database.md'
 ---
 
-This document was moved to [another location](../administration/geo/replication/database_source.md).
+This document was moved to [another location](../administration/geo/replication/database.md).