Reload the schema before restoring a database backup

If a user tries to downgrade and restore after a failed upgrade,
the database may still contain newer tables. Reload the older
schema before restoring the database to avoid future upgrade
problems. Also, add a rake task to help users add migration
versions to the database so it's easier to recover from these
errors if they do occur. Fixes #13419

CHANGELOG

@@ -2,6 +2,7 @@ Please view this file on the master branch, on stable branches it's out of date.
 
 v 8.6.0 (unreleased)
   - Improve the formatting for the user page bio (Connor Shea)
+  - Add option to reload the schema before restoring a database backup. !2807
 
 v 8.5.1
   - Fix group projects styles

doc/raketasks/backup_restore.md

@@ -249,6 +249,9 @@ reconfigure` after changing `gitlab-secrets.json`.
 ### Installation from source
 
 ```
+# Stop processes that are connected to the database
+sudo service gitlab stop
+
 bundle exec rake gitlab:backup:restore RAILS_ENV=production
 ```
 

doc/update/README.md

@@ -15,3 +15,4 @@ Depending on the installation method and your GitLab version, there are multiple
 
 - [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating your database from MySQL to PostgreSQL.
 - [MySQL installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/database_mysql.md) contains additional information about configuring GitLab to work with a MySQL database.
+- [Restoring from backup after a failed upgrade](restore_after_failure.md)

doc/update/restore_after_failure.md

+# Restoring from backup after a failed upgrade
+
+Upgrades are usually smooth and restoring from backup is a rare occurrence.
+However, it's important to know how to recover when problems do arise.
+
+## Roll back to an earlier version and restore a backup
+
+In some cases after a failed upgrade, the fastest solution is to roll back to
+the previous version you were using.
+
+First, roll back the code or package. For source installations this involves
+checking out the older version (branch or tag). For Omnibus installations this
+means installing the older .deb or .rpm package. Then, restore from a backup.
+Follow the instructions in the
+[Backup and Restore](../raketasks/backup_restore.md#restore-a-previously-created-backup)
+documentation.
+
+## Potential problems on the next upgrade
+
+When a rollback is necessary it can produce problems on subsequent upgrade
+attempts. This is because some tables may have been added during the failed
+upgrade. If these tables are still present after you restore from the
+older backup it can lead to migration failures on future upgrades.
+
+Starting in GitLab 8.5 we drop all tables prior to importing the backup to
+prevent this problem. If you've restored a backup to a version prior to 8.5 you
+may need to manually correct the problem next time you upgrade.
+
+Example error:
+
+```
+== 20151103134857 CreateLfsObjects: migrating =================================
+-- create_table(:lfs_objects)
+rake aborted!
+StandardError: An error has occurred, this and all later migrations canceled:
+
+PG::DuplicateTable: ERROR:  relation "lfs_objects" already exists
+```
+
+Copy the version from the error. In this case the version number is
+`20151103134857`.
+
+_**WARNING:** Use the following steps only if you are certain this is what you
+need to do._
+
+### GitLab 8.5+
+
+Pass the version to a database rake task to manually mark the migration as
+complete.
+
+```
+# Source install
+sudo -u git -H bundle exec rake gitlab:db:mark_migration_complete[20151103134857] RAILS_ENV=production
+
+# Omnibus install
+sudo gitlab-rake gitlab:db:mark_migration_complete[20151103134857]
+```
+
+Once the migration is successfully marked, run the rake `db:migrate` task again.
+You will likely have to repeat this process several times until all failed
+migrations are marked complete.
+
+### GitLab < 8.5
+
+```
+# Source install
+sudo -u git -H bundle exec rails console production
+
+# Omnibus install
+sudo gitlab-rails console
+```
+
+At the rails console, type the following commands.
+
+```
+ActiveRecord::Base.connection.execute("INSERT INTO schema_migrations (version) VALUES('20151103134857')")
+exit
+```
+
+Once the migration is successfully marked, run the rake `db:migrate` task again.
+You will likely have to repeat this process several times until all failed
+migrations are marked complete.
+

lib/tasks/gitlab/backup.rake

@@ -22,7 +22,7 @@ namespace :gitlab do
     end
 
     # Restore backup of GitLab system
-    desc "GitLab | Restore a previously created backup"
+    desc 'GitLab | Restore a previously created backup'
     task restore: :environment do
       warn_user_is_not_gitlab
       configure_cron_mode
@@ -30,13 +30,32 @@ namespace :gitlab do
       backup = Backup::Manager.new
       backup.unpack
 
-      Rake::Task["gitlab:backup:db:restore"].invoke unless backup.skipped?("db")
-      Rake::Task["gitlab:backup:repo:restore"].invoke unless backup.skipped?("repositories")
-      Rake::Task["gitlab:backup:uploads:restore"].invoke unless backup.skipped?("uploads")
-      Rake::Task["gitlab:backup:builds:restore"].invoke unless backup.skipped?("builds")
-      Rake::Task["gitlab:backup:artifacts:restore"].invoke unless backup.skipped?("artifacts")
-      Rake::Task["gitlab:backup:lfs:restore"].invoke unless backup.skipped?("lfs")
-      Rake::Task["gitlab:shell:setup"].invoke
+      unless backup.skipped?('db')
+        warning = warning = <<-MSG.strip_heredoc
+          Before restoring the database we recommend removing all existing
+          tables to avoid future upgrade problems. Be aware that if you have
+          custom tables in the GitLab database these tables and all data will be
+          removed.
+        MSG
+        puts warning.red
+        answer = prompt('Confirm whether this task should remove all tables (yes/no): '.red, %w{yes no})
+        if answer == 'yes'
+          puts 'Removing all tables. Press `Ctrl-C` within 5 seconds to abort'.yellow
+          sleep(5)
+          # Drop all tables Load the schema to ensure we don't have any newer tables
+          # hanging out from a failed upgrade
+          $progress.puts 'Cleaning the database ... '.blue
+          Rake::Task['gitlab:db:drop_tables'].invoke
+        end
+        $progress.puts 'done'.green
+        Rake::Task['gitlab:backup:db:restore'].invoke
+      end
+      Rake::Task['gitlab:backup:repo:restore'].invoke unless backup.skipped?('repositories')
+      Rake::Task['gitlab:backup:uploads:restore'].invoke unless backup.skipped?('uploads')
+      Rake::Task['gitlab:backup:builds:restore'].invoke unless backup.skipped?('builds')
+      Rake::Task['gitlab:backup:artifacts:restore'].invoke unless backup.skipped?('artifacts')
+      Rake::Task['gitlab:backup:lfs:restore'].invoke unless backup.skipped?('lfs')
+      Rake::Task['gitlab:shell:setup'].invoke
 
       backup.cleanup
     end

lib/tasks/gitlab/db.rake

+namespace :gitlab do
+  namespace :db do
+    desc 'GitLab | Manually insert schema migration version'
+    task :mark_migration_complete, [:version] => :environment do |_, args|
+      unless args[:version]
+        puts "Must specify a migration version as an argument".red
+        exit 1
+      end
+
+      version = args[:version].to_i
+      if version == 0
+        puts "Version '#{args[:version]}' must be a non-zero integer".red
+        exit 1
+      end
+
+      sql = "INSERT INTO schema_migrations (version) VALUES (#{version})"
+      begin
+        ActiveRecord::Base.connection.execute(sql)
+        puts "Successfully marked '#{version}' as complete".green
+      rescue ActiveRecord::RecordNotUnique
+        puts "Migration version '#{version}' is already marked complete".yellow
+      end
+    end
+
+    desc 'Drop all tables'
+    task :drop_tables => :environment do
+      connection = ActiveRecord::Base.connection
+      tables = connection.tables
+      tables.delete 'schema_migrations'
+      # Truncate schema_migrations to ensure migrations re-run
+      connection.execute('TRUNCATE schema_migrations')
+      tables.each { |t| connection.execute("DROP TABLE #{t}") }
+    end
+  end
+end

spec/tasks/gitlab/backup_rake_spec.rb

@@ -3,9 +3,10 @@ require 'rake'
 
 describe 'gitlab:app namespace rake task' do
   before :all do
-    Rake.application.rake_require "tasks/gitlab/task_helpers"
-    Rake.application.rake_require "tasks/gitlab/backup"
-    Rake.application.rake_require "tasks/gitlab/shell"
+    Rake.application.rake_require 'tasks/gitlab/task_helpers'
+    Rake.application.rake_require 'tasks/gitlab/backup'
+    Rake.application.rake_require 'tasks/gitlab/shell'
+    Rake.application.rake_require 'tasks/gitlab/db'
     # empty task as env is already loaded
     Rake::Task.define_task :environment
   end
@@ -52,13 +53,14 @@ describe 'gitlab:app namespace rake task' do
       it 'should invoke restoration on match' do
         allow(YAML).to receive(:load_file).
           and_return({ gitlab_version: gitlab_version })
-        expect(Rake::Task["gitlab:backup:db:restore"]).to receive(:invoke)
-        expect(Rake::Task["gitlab:backup:repo:restore"]).to receive(:invoke)
-        expect(Rake::Task["gitlab:backup:builds:restore"]).to receive(:invoke)
-        expect(Rake::Task["gitlab:backup:uploads:restore"]).to receive(:invoke)
-        expect(Rake::Task["gitlab:backup:artifacts:restore"]).to receive(:invoke)
-        expect(Rake::Task["gitlab:backup:lfs:restore"]).to receive(:invoke)
-        expect(Rake::Task["gitlab:shell:setup"]).to receive(:invoke)
+        expect(Rake::Task['gitlab:db:drop_tables']).to receive(:invoke)
+        expect(Rake::Task['gitlab:backup:db:restore']).to receive(:invoke)
+        expect(Rake::Task['gitlab:backup:repo:restore']).to receive(:invoke)
+        expect(Rake::Task['gitlab:backup:builds:restore']).to receive(:invoke)
+        expect(Rake::Task['gitlab:backup:uploads:restore']).to receive(:invoke)
+        expect(Rake::Task['gitlab:backup:artifacts:restore']).to receive(:invoke)
+        expect(Rake::Task['gitlab:backup:lfs:restore']).to receive(:invoke)
+        expect(Rake::Task['gitlab:shell:setup']).to receive(:invoke)
         expect { run_rake_task('gitlab:backup:restore') }.not_to raise_error
       end
     end
@@ -177,17 +179,18 @@ describe 'gitlab:app namespace rake task' do
     end
 
     it 'does not invoke repositories restore' do
-      allow(Rake::Task["gitlab:shell:setup"]).
+      allow(Rake::Task['gitlab:shell:setup']).
         to receive(:invoke).and_return(true)
       allow($stdout).to receive :write
 
-      expect(Rake::Task["gitlab:backup:db:restore"]).to receive :invoke
-      expect(Rake::Task["gitlab:backup:repo:restore"]).not_to receive :invoke
-      expect(Rake::Task["gitlab:backup:uploads:restore"]).not_to receive :invoke
-      expect(Rake::Task["gitlab:backup:builds:restore"]).to receive :invoke
-      expect(Rake::Task["gitlab:backup:artifacts:restore"]).to receive :invoke
-      expect(Rake::Task["gitlab:backup:lfs:restore"]).to receive :invoke
-      expect(Rake::Task["gitlab:shell:setup"]).to receive :invoke
+      expect(Rake::Task['gitlab:db:drop_tables']).to receive :invoke
+      expect(Rake::Task['gitlab:backup:db:restore']).to receive :invoke
+      expect(Rake::Task['gitlab:backup:repo:restore']).not_to receive :invoke
+      expect(Rake::Task['gitlab:backup:uploads:restore']).not_to receive :invoke
+      expect(Rake::Task['gitlab:backup:builds:restore']).to receive :invoke
+      expect(Rake::Task['gitlab:backup:artifacts:restore']).to receive :invoke
+      expect(Rake::Task['gitlab:backup:lfs:restore']).to receive :invoke
+      expect(Rake::Task['gitlab:shell:setup']).to receive :invoke
       expect { run_rake_task('gitlab:backup:restore') }.not_to raise_error
     end
   end