Merge branch 'ab/db-extensions' into 'master'

More verbose error message when creating extension

Closes #243729

See merge request gitlab-org/gitlab!40956

db/migrate/20200515152649_enable_btree_gist_extension.rb

 # frozen_string_literal: true
 
 class EnableBtreeGistExtension < ActiveRecord::Migration[6.0]
+  include Gitlab::Database::MigrationHelpers
+
   DOWNTIME = false
 
   def up
-    execute 'CREATE EXTENSION IF NOT EXISTS btree_gist'
+    create_extension :btree_gist
   end
 
   def down
-    execute 'DROP EXTENSION IF EXISTS btree_gist'
+    drop_extension :btree_gist
   end
 end

doc/install/postgresql_extensions.md

+---
+last_updated: 2020-09-01
+---
+
+# Managing PostgreSQL extensions
+
+This guide documents how to manage PostgreSQL extensions for installations with an external
+PostgreSQL database.
+
+GitLab requires certain extensions to be installed into the GitLab database. For example,
+GitLab relies on `pg_trgm` and the `btree_gist` extensions.
+
+In order to install extensions, PostgreSQL requires the user to have superuser privileges.
+Typically, the GitLab database user is not a superuser. Therefore, regular database migrations
+cannot be used in installing extensions and instead, extensions have to be installed manually
+prior to upgrading GitLab to a newer version.
+
+## Installing PostgreSQL extensions manually
+
+In order to install a PostgreSQL extension, this procedure should be followed:
+
+1. Connect to the GitLab PostgreSQL database using a superuser, for example:
+
+   ```shell
+   sudo gitlab-psql -d gitlabhq_production
+   ```
+
+1. Install the extension (`btree_gist` in this example) using [`CREATE EXTENSION`](https://www.postgresql.org/docs/11/sql-createextension.html):
+
+   ```sql
+   CREATE EXTENSION IF NOT EXISTS btree_gist
+   ```
+
+1. Verify installed extensions:
+
+   ```shell
+    gitlabhq_production=# \dx
+                                        List of installed extensions
+        Name    | Version |   Schema   |                            Description
+    ------------+---------+------------+-------------------------------------------------------------------
+    btree_gist | 1.5     | public     | support for indexing common datatypes in GiST
+    pg_trgm    | 1.4     | public     | text similarity measurement and index searching based on trigrams
+    plpgsql    | 1.0     | pg_catalog | PL/pgSQL procedural language
+    (3 rows)
+   ```
+
+On some systems you may need to install an additional package (for example,
+`postgresql-contrib`) for certain extensions to become available.
+
+## A typical migration failure scenario
+
+The following is an example of a situation when the extension hasn't been installed before running migrations.
+In this scenario, the database migration fails to create the extension `btree_gist` because of insufficient
+privileges.
+
+```shell
+== 20200515152649 EnableBtreeGistExtension: migrating =========================
+-- execute("CREATE EXTENSION IF NOT EXISTS btree_gist")
+
+GitLab requires the PostgreSQL extension 'btree_gist' installed in database 'gitlabhq_production', but
+the database user is not allowed to install the extension.
+
+You can either install the extension manually using a database superuser:
+
+  CREATE EXTENSION IF NOT EXISTS btree_gist
+
+Or, you can solve this by logging in to the GitLab database (gitlabhq_production) using a superuser and running:
+
+    ALTER regular WITH SUPERUSER
+
+This query will grant the user superuser permissions, ensuring any database extensions
+can be installed through migrations.
+```
+
+In order to recover from this situation, the extension needs to be installed manually using a superuser, and
+the database migration (or GitLab upgrade) can be retried afterwards.

doc/install/requirements.md

@@ -143,11 +143,8 @@ GitLab version | Minimum PostgreSQL version
 12.10 | 11
 13.0 | 11
 
-You must also ensure the `pg_trgm` and `btree_gist` extensions are loaded into every
-GitLab database. These extensions [can be enabled](https://www.postgresql.org/docs/11/sql-createextension.html) using a PostgreSQL super user.
-
-On some systems you may need to install an additional package (for example,
-`postgresql-contrib`) for this extension to become available.
+You must also ensure the `pg_trgm` and `btree_gist` extensions are [loaded into every
+GitLab database](postgresql_extensions.html).
 
 NOTE: **Note:**
 Support for [PostgreSQL 9.6 and 10 has been removed in GitLab 13.0](https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/#postgresql-11-is-now-the-minimum-required-version-to-install-gitlab) so that GitLab can benefit from PostgreSQL 11 improvements, such as partitioning. For the schedule of transitioning to PostgreSQL 12, see [the related epic](https://gitlab.com/groups/gitlab-org/-/epics/2184).

doc/update/README.md

@@ -310,3 +310,4 @@ for more information.
 - [Restoring from backup after a failed upgrade](restore_after_failure.md)
 - [Upgrading PostgreSQL Using Slony](upgrading_postgresql_using_slony.md), for
   upgrading a PostgreSQL database with minimal downtime.
+- [Managing PostgreSQL extensions](../install/postgresql_extensions.md)

lib/gitlab/database/migration_helpers.rb

@@ -1212,6 +1212,63 @@ into similar problems in the future (e.g. when new tables are created).
         )
       end
 
+      def create_extension(extension)
+        execute('CREATE EXTENSION IF NOT EXISTS %s' % extension)
+      rescue ActiveRecord::StatementInvalid => e
+        dbname = Database.database_name
+        user = Database.username
+
+        warn(<<~MSG) if e.to_s =~ /permission denied/
+          GitLab requires the PostgreSQL extension '#{extension}' installed in database '#{dbname}', but
+          the database user is not allowed to install the extension.
+
+          You can either install the extension manually using a database superuser:
+
+            CREATE EXTENSION IF NOT EXISTS #{extension}
+
+          Or, you can solve this by logging in to the GitLab
+          database (#{dbname}) using a superuser and running:
+
+              ALTER #{user} WITH SUPERUSER
+
+          This query will grant the user superuser permissions, ensuring any database extensions
+          can be installed through migrations.
+
+          For more information, refer to https://docs.gitlab.com/ee/install/postgresql_extensions.html.
+        MSG
+
+        raise
+      end
+
+      def drop_extension(extension)
+        execute('DROP EXTENSION IF EXISTS %s' % extension)
+      rescue ActiveRecord::StatementInvalid => e
+        dbname = Database.database_name
+        user = Database.username
+
+        warn(<<~MSG) if e.to_s =~ /permission denied/
+          This migration attempts to drop the PostgreSQL extension '#{extension}'
+          installed in database '#{dbname}', but the database user is not allowed
+          to drop the extension.
+
+          You can either drop the extension manually using a database superuser:
+
+            DROP EXTENSION IF EXISTS #{extension}
+
+          Or, you can solve this by logging in to the GitLab
+          database (#{dbname}) using a superuser and running:
+
+              ALTER #{user} WITH SUPERUSER
+
+          This query will grant the user superuser permissions, ensuring any database extensions
+          can be dropped through migrations.
+
+          For more information, refer to https://docs.gitlab.com/ee/install/postgresql_extensions.html.
+        MSG
+
+        raise
+      end
+
       private
 
       def validate_check_constraint_name!(constraint_name)

spec/lib/gitlab/database/migration_helpers_spec.rb

@@ -2329,4 +2329,56 @@ RSpec.describe Gitlab::Database::MigrationHelpers do
       end
     end
   end
+
+  describe '#create_extension' do
+    subject { model.create_extension(extension) }
+
+    let(:extension) { :btree_gist }
+
+    it 'executes CREATE EXTENSION statement' do
+      expect(model).to receive(:execute).with(/CREATE EXTENSION IF NOT EXISTS #{extension}/)
+
+      subject
+    end
+
+    context 'without proper permissions' do
+      before do
+        allow(model).to receive(:execute).with(/CREATE EXTENSION IF NOT EXISTS #{extension}/).and_raise(ActiveRecord::StatementInvalid, 'InsufficientPrivilege: permission denied')
+      end
+
+      it 'raises the exception' do
+        expect { subject }.to raise_error(ActiveRecord::StatementInvalid, /InsufficientPrivilege/)
+      end
+
+      it 'prints an error message' do
+        expect { subject }.to output(/user is not allowed/).to_stderr.and raise_error
+      end
+    end
+  end
+
+  describe '#drop_extension' do
+    subject { model.drop_extension(extension) }
+
+    let(:extension) { 'btree_gist' }
+
+    it 'executes CREATE EXTENSION statement' do
+      expect(model).to receive(:execute).with(/DROP EXTENSION IF EXISTS #{extension}/)
+
+      subject
+    end
+
+    context 'without proper permissions' do
+      before do
+        allow(model).to receive(:execute).with(/DROP EXTENSION IF EXISTS #{extension}/).and_raise(ActiveRecord::StatementInvalid, 'InsufficientPrivilege: permission denied')
+      end
+
+      it 'raises the exception' do
+        expect { subject }.to raise_error(ActiveRecord::StatementInvalid, /InsufficientPrivilege/)
+      end
+
+      it 'prints an error message' do
+        expect { subject }.to output(/user is not allowed/).to_stderr.and raise_error
+      end
+    end
+  end
 end