Improve geo docs based on geo demo on Dec 5

doc/gitlab-geo/README.md

@@ -127,22 +127,29 @@ sync again.
 
 ## Setup instructions
 
-In order to set up one or more GitLab Geo instances, follow the steps below in
-the **exact order** they appear. **Make sure the GitLab version is the same on
-all nodes.**
+These instructions assume you have a working instance of GitLab. They will
+guide you through making your existing instance the primary Geo node and
+adding secondary Geo nodes.
+
+The steps below should be followed in the order they appear. **Make sure the
+GitLab version is the same on all nodes.**
 
 ### Using Omnibus GitLab
 
 If you installed GitLab using the Omnibus packages (highly recommended):
 
 1. [Install GitLab Enterprise Edition][install-ee] on the server that will serve
-   as the **secondary** Geo node. Do not login or set up anything else in the
-   secondary node for the moment.
-1. [Upload the GitLab License](../user/admin_area/license.md) on the **primary** Geo Node to unlock GitLab Geo.
-1. [Setup the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology).
-1. [Lookup authorized SSH keys in the database](../administration/operations/speed_up_ssh.html), do this step for both primary AND secondary nodes.
+   as the **secondary** Geo node. Do not create an account or login to the new
+   secondary node.
+1. [Upload the GitLab License](../user/admin_area/license.md) on the **primary**
+   Geo node to unlock GitLab Geo.
+1. [Setup the database replication](database.md) (`primary (read-write) <->
+   secondary (read-only)` topology).
+1. [Lookup authorized SSH keys in the database](../administration/operations/speed_up_ssh.html),
+   do this step for both primary AND secondary nodes.
 1. [Configure GitLab](configuration.md) to set the primary and secondary nodes.
-1. Optional: [Configure a secondary LDAP server](../administration/auth/ldap.md) for the secondary. See [notes on LDAP](#ldap).
+1. Optional: [Configure a secondary LDAP server](../administration/auth/ldap.md)
+   for the secondary. See [notes on LDAP](#ldap).
 1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md).
 
 [install-ee]: https://about.gitlab.com/downloads-ee/ "GitLab Enterprise Edition Omnibus packages downloads page"
@@ -152,11 +159,14 @@ If you installed GitLab using the Omnibus packages (highly recommended):
 If you installed GitLab from source:
 
 1. [Install GitLab Enterprise Edition][install-ee-source] on the server that
-   will serve as the **secondary** Geo node. Do not login or set up anything
-   else in the secondary node for the moment.
-1. [Upload the GitLab License](../user/admin_area/license.md) on the **primary** Geo Node to unlock GitLab Geo.
-1. [Setup the database replication](database_source.md) (`primary (read-write) <-> secondary (read-only)` topology).
-1. [Lookup authorized SSH keys in the database](../administration/operations/speed_up_ssh.html), do this step for both primary AND secondary nodes.
+   will serve as the **secondary** Geo node. Do not create an account or login
+   to the new secondary node.
+1. [Upload the GitLab License](../user/admin_area/license.md) on the **primary**
+   Geo node to unlock GitLab Geo.
+1. [Setup the database replication](database_source.md) (`primary (read-write)
+   <-> secondary (read-only)` topology).
+1. [Lookup authorized SSH keys in the database](../administration/operations/speed_up_ssh.html),
+   do this step for both primary AND secondary nodes.
 1. [Configure GitLab](configuration_source.md) to set the primary and secondary
    nodes.
 1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md).

doc/gitlab-geo/configuration_source.md

@@ -59,6 +59,13 @@ Then save and close the file.
     service gitlab restart
     ```
 
+The secondary will start automatically replicating missing data from the
+primary in a process known as backfill. Meanwhile, the primary node will start
+to notify changes to the secondary, which will act on those notifications
+immediately. Make sure the secondary instance is running and accessible.
+
+### Step 2. (Optional) Enabling hashed storage
+
 Once restarted, the secondary will automatically start replicating missing data
 from the primary in a process known as backfill. Meanwhile, the primary node
 will start to notify the secondary of any changes, so that the secondary can
@@ -87,7 +94,33 @@ cp primary.geo.example.com.crt /usr/local/share/ca-certificates
 update-ca-certificates
 ```
 
+### Step 4. Managing the secondary GitLab node
+
+Congratulations! Your secondary geo node is now configured!
+
+The initial replication, or 'backfill', will probably still be in progress.
+You can monitor the synchronization process on each geo node from the primary
+node's Geo Nodes dashboard (Admin Area ➔ Geo Nodes, `/admin/geo_nodes`) in your
+browser.
+
+![GitLab Geo dashboard](img/geo-node-dashboard.png)
+
+After the backfill is completed you can continue to monitor geo node health and
+replication delays from the dashboard.
+
+The two most obvious issues that can become apparent in the dashboard are:
+
+1. Database replication not working well
+1. Instance to instance notification not working. In that case, it can be
+   something of the following:
+     - You are using a custom certificate or custom CA (see the
 ### Step 4. Enable Git access over HTTP/HTTPS
+     - Instance is firewalled (check your firewall rules)
+
+Please note that disabling a secondary node will stop the sync process.
+
+Please note that if `git_data_dirs` is customized on the primary for multiple
+repository shards you must duplicate the same configuration on the secondary.
 
 GitLab Geo synchronizes repositories over HTTP/HTTPS, and so requires this clone
 method to be enabled. Navigate to **Admin Area ➔ Settings**

doc/gitlab-geo/database.md

@@ -160,17 +160,17 @@ will not be able to perform all necessary configuration steps. Refer to
 
     | GitLab Terminology | Amazon Web Services | Google Cloud Platform |
     |-----|-----|-----|-----|
-    | Interface address | Private address | Internal address |
+    | Private address | Private address | Internal address |
     | Public address | Public address | External address |
 
-    To lookup the address of a Geo node, on the Geo node execute:
+    To lookup the address of a Geo node, SSH in to the Geo node and execute:
 
     ```bash
-    # Interface address
-    ip route get 255.255.255.255 | awk '{print $NF; exit}'
+    # Private address
+    ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}'
 
     # Public address
-    curl ipinfo.io/ip
+    echo "External address: $(curl ipinfo.io/ip)"
     ```
 
     In most cases, the following addresses will be used to configure GitLab
@@ -178,8 +178,8 @@ will not be able to perform all necessary configuration steps. Refer to
 
     | Configuration | Address |
     |-----|-----|
-    | `postgresql['listen_address']` | Primary's interface address |
-    | `postgresql['trust_auth_cidr_addresses']` | Primary's interface address |
+    | `postgresql['listen_address']` | Primary's private address |
+    | `postgresql['trust_auth_cidr_addresses']` | Primary's private address |
     | `postgresql['md5_auth_cidr_addresses']` | Secondary's public addresses |
 
     The `listen_address` option opens PostgreSQL up to network connections
@@ -191,7 +191,7 @@ will not be able to perform all necessary configuration steps. Refer to
     be correct. If your primary and secondary connect over a local
     area network, or a virtual network connecting availability zones like
     Amazon's [VPC](https://aws.amazon.com/vpc/) of Google's [VPC](https://cloud.google.com/vpc/)
-    you should use the secondary's interface address for `postgresql['md5_auth_cidr_addresses']`.
+    you should use the secondary's private address for `postgresql['md5_auth_cidr_addresses']`.
 
     Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP
     addresses with addresses appropriate to your network configuration:
@@ -200,7 +200,7 @@ will not be able to perform all necessary configuration steps. Refer to
     geo_primary_role['enable'] = true
 
     # Primary address
-    # - replace '1.2.3.4' with the primary interface address
+    # - replace '1.2.3.4' with the primary private address
     postgresql['listen_address'] = '1.2.3.4'
     postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32']
 
@@ -214,8 +214,8 @@ will not be able to perform all necessary configuration steps. Refer to
     # postgresql['max_wal_senders'] = 10
     # postgresql['wal_keep_segments'] = 10
 
-    # Disable automatic database migrations for now
-    # (until PostgreSQL is restarted and listening on the interface address)
+    # Disable automatic database migrations temporarily
+    # (until PostgreSQL is restarted and listening on the private address)
     gitlab_rails['auto_migrate'] = false
     ```
 
@@ -232,8 +232,12 @@ will not be able to perform all necessary configuration steps. Refer to
     Replication documentation](https://www.postgresql.org/docs/9.6/static/runtime-config-replication.html)
     for more information.
 
-1. Save the file and [reconfigure GitLab][] for the database listen changes and
+1. Save the file and reconfigure GitLab for the database listen changes and
    the replication slot changes to be applied.
+    
+    ```bash
+    gitlab-ctl reconfigure
+    ```
 
     Restart PostgreSQL for its changes to take effect:
 
@@ -241,21 +245,24 @@ will not be able to perform all necessary configuration steps. Refer to
     gitlab-ctl restart postgresql
     ```
 
-1. Reenable migrations
+1. Re-enable migrations now that PostgreSQL is restarted and listening on the
+   private address.
 
-    Edit `/etc/gitlab/gitlab.rb` and **delete** the following lines:
+    Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`:
 
     ```ruby
-    # Disable automatic database migrations for now
-    # (until PostgreSQL is restarted and listening on the interface address)
-    gitlab_rails['auto_migrate'] = false
+    gitlab_rails['auto_migrate'] = true
     ```
 
-    Save the file and [reconfigure GitLab][].
+    Save the file and reconfigure GitLab:
+
+    ```bash
+    gitlab-ctl reconfigure
+    ```
 
 1. Now that the PostgreSQL server is set up to accept remote connections, run
    `netstat -plnt` to make sure that PostgreSQL is listening on port `5432` to
-   the server's interface address.
+   the primary server's private address.
 
 1. Verify that clock synchronization is enabled.
 
@@ -284,11 +291,14 @@ primary before the database is replicated.
    (`/admin/geo_nodes`) in your browser.
 1. Add the secondary node by providing its full URL. **Do NOT** check the box
    'This is a primary node'.
-1. Added in GitLab 9.5: Choose which namespaces should be replicated by the
+1. Optionally, choose which namespaces should be replicated by the
    secondary node. Leave blank to replicate all. Read more in
    [selective replication](#selective-replication).
 1. Click the **Add node** button.
 
+The new secondary geo node will have the status **Unhealthy**. This is expected
+because we have not yet configured the secondary server. This is the next step.
+
 ### Step 3. Configure the secondary server
 
 1. SSH into your GitLab **secondary** server and login as root:
@@ -330,12 +340,12 @@ primary before the database is replicated.
 
     ```
     # Certificate and key currently used by GitLab, and connecting by FQDN
-    sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h primary.geo.example.com -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-full" -W
+    sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-full" -W -h primary.geo.example.com
     ```
     
     ```
     # Self-signed certificate and key, or connecting by IP address
-    sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h 1.2.3.4 -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-ca" -W
+    sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-ca" -W -h 1.2.3.4
     ```
 
     When prompted enter the password you set in the first step for the
@@ -399,21 +409,21 @@ data before running `pg_basebackup`.
     sudo -i
     ```
 
-1. New for 9.4: Choose a database-friendly name to use for your secondary to
+1. Choose a database-friendly name to use for your secondary to
    use as the replication slot name. For example, if your domain is
    `secondary.geo.example.com`, you may use `secondary_example` as the slot
-   name.
+   name as shown in the commands below.
 
 1. Execute the command below to start a backup/restore and begin the replication:
 
     ```
     # Certificate and key currently used by GitLab, and connecting by FQDN
-    gitlab-ctl replicate-geo-database --host=primary.geo.example.com --slot-name=secondary_example
+    gitlab-ctl replicate-geo-database --slot-name=secondary_example --host=primary.geo.example.com
     ```
     
     ```
     # Self-signed certificate and key, or connecting by IP
-    gitlab-ctl replicate-geo-database --host=1.2.3.4 --slot-name=secondary_example --sslmode=verify-ca
+    gitlab-ctl replicate-geo-database --slot-name=secondary_example --sslmode=verify-ca --host=1.2.3.4
     ```
 
     If PostgreSQL is listening on a non-standard port, add `--port=` as well.
@@ -510,7 +520,6 @@ We don't support MySQL replication for GitLab Geo.
 Read the [troubleshooting document](troubleshooting.md).
 
 [pgback]: http://www.postgresql.org/docs/9.2/static/app-pgbasebackup.html
-[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
 [external postgresql]: #external-postgresql-instances
 [tracking]: database_source.md#enable-tracking-database-on-the-secondary-server
 [toc]: README.md#using-omnibus-gitlab

doc/gitlab-geo/database_source.md

@@ -229,11 +229,14 @@ primary before the database is replicated.
    (`/admin/geo_nodes`) in your browser.
 1. Add the secondary node by providing its full URL. **Do NOT** check the box
    'This is a primary node'.
-1. Added in GitLab 9.5: Choose which namespaces should be replicated by the
+1. Optionally, choose which namespaces should be replicated by the
    secondary node. Leave blank to replicate all. Read more in
    [selective replication](#selective-replication).
 1. Click the **Add node** button.
 
+The new secondary geo node will have the status **Unhealthy**. This is expected
+because we have not yet configured the secondary server. This is the next step.
+
 ### Step 3. Configure the secondary server
 
 1. SSH into your GitLab **secondary** server and login as root:
@@ -273,10 +276,10 @@ primary before the database is replicated.
 
     ```
     # Certificate and key currently used by GitLab, and connecting by FQDN
-    sudo -u postgres psql -h primary.geo.example.com -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-ca" -W
+    sudo -u postgres psql -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-ca" -W -h primary.geo.example.com
 
     # Self-signed certificate and key, or connecting by IP address
-    sudo -u postgres psql -h 1.2.3.4 -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-ca" -W
+    sudo -u postgres psql -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-ca" -W -h 1.2.3.4
     ```
 
     When prompted enter the password you set in the first step for the
@@ -473,5 +476,4 @@ We don't support MySQL replication for GitLab Geo.
 Read the [troubleshooting document](troubleshooting.md).
 
 [pgback]: http://www.postgresql.org/docs/9.6/static/app-pgbasebackup.html
-[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
 [toc]: README.md#using-gitlab-installed-from-source