Cleanup and improve Gitaly on seperate server over TLS setup

doc/administration/gitaly/index.md

@@ -177,15 +177,14 @@ Check the directory layout on your Gitaly server to be sure.
    # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server.
    gitlab_rails['internal_api_url'] = 'https://gitlab.example.com'
 
+   # Authentication token to ensure only authorized servers can communicate with
+   # Gitaly server
+   gitaly['auth_token'] = 'abc123secret'
+
    # Make Gitaly accept connections on all network interfaces. You must use
    # firewalls to restrict access to this address/port.
+   # Comment out following line if you only want to support TLS connections
    gitaly['listen_addr'] = "0.0.0.0:8075"
-   gitaly['auth_token'] = 'abc123secret'
-
-   # To use TLS for Gitaly you need to add
-   gitaly['tls_listen_addr'] = "0.0.0.0:9999"
-   gitaly['certificate_path'] = "path/to/cert.pem"
-   gitaly['key_path'] = "path/to/key.pem"
    ```
 
 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server:
@@ -219,11 +218,6 @@ Check the directory layout on your Gitaly server to be sure.
 
    ```toml
    listen_addr = '0.0.0.0:8075'
-   tls_listen_addr = '0.0.0.0:9999'
-
-   [tls]
-   certificate_path = /path/to/cert.pem
-   key_path = /path/to/key.pem
 
    [auth]
    token = 'abc123secret'
@@ -369,11 +363,12 @@ To disable Gitaly on a client node:
 > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/22602) in GitLab 11.8.
 
 Gitaly supports TLS encryption. To be able to communicate
-with a Gitaly instance that listens for secure connections you will need to use `tls://` url
+with a Gitaly instance that listens for secure connections you will need to use `tls://` URL
 scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration.
 
 You will need to bring your own certificates as this isn't provided automatically.
-The certificate to be used needs to be installed on all Gitaly nodes and on all
+The certificate to be used needs to be installed on all Gitaly nodes, and the
+certificate (or CA of certificate) on all
 client nodes that communicate with it following the procedure described in
 [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates).
 
@@ -395,7 +390,7 @@ To configure Gitaly with TLS:
 
 **For Omnibus GitLab**
 
-1. On the client nodes, edit `/etc/gitlab/gitlab.rb`:
+1. On the client node(s), edit `/etc/gitlab/gitlab.rb` as follows:
 
    ```ruby
    git_data_dirs({
@@ -407,20 +402,38 @@ To configure Gitaly with TLS:
    gitlab_rails['gitaly_token'] = 'abc123secret'
    ```
 
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
-1. On the Gitaly server nodes, edit `/etc/gitlab/gitlab.rb`:
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on client node(s).
+1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there:
+
+   ```sh
+   sudo mkdir -p /etc/gitlab/ssl
+   sudo chmod 700 /etc/gitlab/ssl
+   sudo cp key.pem cert.pem /etc/gitlab/ssl/
+   ```
+
+1. On the Gitaly server node(s), edit `/etc/gitlab/gitlab.rb` and add:
+
+   <!--
+   updates to following example must also be made at
+   https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab
+   -->
 
    ```ruby
    gitaly['tls_listen_addr'] = "0.0.0.0:9999"
-   gitaly['certificate_path'] = "path/to/cert.pem"
-   gitaly['key_path'] = "path/to/key.pem"
+   gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem"
+   gitaly['key_path'] = "/etc/gitlab/ssl/key.pem"
    ```
 
-1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly server node(s).
+1. (Optional) After [verifying that all Gitaly traffic is being served over TLS](#observe-type-of-gitaly-connections),
+   you can improve security by disabling non-TLS connections by commenting out
+   or deleting `gitaly['listen_addr']` in `/etc/gitlab/gitlab.rb`, saving the file,
+   and [reconfiguring GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure)
+   on Gitaly server node(s).
 
 **For installations from source**
 
-1. On the client nodes, edit `/home/git/gitlab/config/gitlab.yml`:
+1. On the client node(s), edit `/home/git/gitlab/config/gitlab.yml` as follows:
 
    ```yaml
    gitlab:
@@ -445,18 +458,33 @@ To configure Gitaly with TLS:
    data will be stored in this folder. This will no longer be necessary after
    [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved.
 
-1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
-1. On the Gitaly server nodes, edit `/home/git/gitaly/config.toml`:
+1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) on client node(s).
+1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there:
+
+   ```sh
+   sudo mkdir -p /etc/gitlab/ssl
+   sudo chmod 700 /etc/gitlab/ssl
+   sudo cp key.pem cert.pem /etc/gitlab/ssl/
+   ```
+
+1. On the Gitaly server node(s), edit `/home/git/gitaly/config.toml` and add:
 
    ```toml
    tls_listen_addr = '0.0.0.0:9999'
 
    [tls]
-   certificate_path = '/path/to/cert.pem'
-   key_path = '/path/to/key.pem'
+   certificate_path = '/etc/gitlab/ssl/cert.pem'
+   key_path = '/etc/gitlab/ssl/key.pem'
    ```
 
-1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source).
+1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) on Gitaly server node(s).
+1. (Optional) After [verifying that all Gitaly traffic is being served over TLS](#observe-type-of-gitaly-connections),
+   you can improve security by disabling non-TLS connections by commenting out
+   or deleting `listen_addr` in `/home/git/gitaly/config.toml`, saving the file,
+   and [restarting GitLab](../restart_gitlab.md#installations-from-source)
+   on Gitaly server node(s).
+
+### Observe type of Gitaly connections
 
 To observe what type of connections are actually being used in a
 production environment you can use the following Prometheus query: