Merge branch '291980-docs-on-removing-elasticsearch-migrations-in-major-version' into 'master'

Document Elasticsearch migrations must be finished before major upgrade

See merge request gitlab-org/gitlab!55580

doc/administration/troubleshooting/elasticsearch.md

@@ -36,7 +36,7 @@ The type of problem will determine what steps to take. The possible troubleshoot
 - Indexing.
 - Integration.
 - Performance.
-- Background Migrations.
+- Advanced Search Migrations.
 
 ### Search Results workflow
 
@@ -148,7 +148,7 @@ graph TD;
   F7(Escalate to<br>GitLab support.)
 ```
 
-### Background Migrations workflow
+### Advanced Search Migrations workflow
 
 ```mermaid
 graph TD;
@@ -180,7 +180,7 @@ Most Elasticsearch troubleshooting can be broken down into 4 categories:
 - [Troubleshooting indexing](#troubleshooting-indexing)
 - [Troubleshooting integration](#troubleshooting-integration)
 - [Troubleshooting performance](#troubleshooting-performance)
-- [Troubleshooting background migrations](#troubleshooting-background-migrations)
+- [Troubleshooting Advanced Search migrations](#troubleshooting-advanced-search-migrations)
 
 Generally speaking, if it does not fall into those four categories, it is either:
 
@@ -356,15 +356,16 @@ dig further into these.
 Feel free to reach out to GitLab support, but this is likely to be something a skilled
 Elasticsearch administrator has more experience with.
 
-### Troubleshooting background migrations
+### Troubleshooting Advanced Search migrations
 
-Troubleshooting background migration failures can be difficult and may require contacting
-an Elasticsearch administrator or GitLab Support.
+Troubleshooting Advanced Search migration failures can be difficult and may
+require contacting an Elasticsearch administrator or GitLab Support.
 
-The best place to start while debugging issues with a background migration is the
-[`elasticsearch.log` file](../logs.md#elasticsearchlog). Migrations will
-print information while a migration is in progress and any errors encountered.
-Apply fixes for any errors found in the log and retry the migration.
+The best place to start while debugging issues with an Advanced Search
+migration is the [`elasticsearch.log` file](../logs.md#elasticsearchlog).
+Migrations will log information while a migration is in progress and any
+errors encountered. Apply fixes for any errors found in the log and retry
+the migration.
 
 If you still encounter issues after retrying the migration, reach out to GitLab support.
 

doc/development/elasticsearch.md

@@ -249,12 +249,12 @@ end
 
 ### Multi-version compatibility
 
-These Elasticsearch migrations, like any other GitLab changes, need to support the case where
+These Advanced Search migrations, like any other GitLab changes, need to support the case where
 [multiple versions of the application are running at the same time](multi_version_compatibility.md).
 
 Depending on the order of deployment, it's possible that the migration
 has started or finished and there's still a server running the application code from before the
-migration. We need to take this into consideration until we can [ensure all Elasticsearch migrations
+migration. We need to take this into consideration until we can [ensure all Advanced Search migrations
 start after the deployment has finished](https://gitlab.com/gitlab-org/gitlab/-/issues/321619).
 
 ### Reverting a migration
@@ -268,7 +268,7 @@ some data is moved) to a later merge request after the migrations have
 completed successfully. To be safe, for self-managed customers we should also
 defer it to another release if there is risk of important data loss.
 
-### Best practices for Elasticsearch migrations
+### Best practices for Advanced Search migrations
 
 Follow these best practices for best results:
 
@@ -286,6 +286,63 @@ Follow these best practices for best results:
 - Consider adding a retry limit if there is potential for the migration to fail.
   This ensures that migrations can be halted if an issue occurs.
 
+## Deleting Advanced Search migrations in a major version upgrade
+
+Since our Advanced Search migrations usually require us to support multiple
+code paths for a long period of time, it's important to clean those up when we
+safely can.
+
+We choose to use GitLab major version upgrades as a safe time to remove
+backwards compatibility for indices that have not been fully migrated. We
+[document this in our upgrade
+documentation](../update/index.md#upgrading-to-a-new-major-version). We also
+choose to remove the migration code and tests so that:
+
+- We don't need to maintain any code that is called from our Advanced Search
+  migrations.
+- We don't waste CI time running tests for migrations that we don't support
+  anymore.
+
+To be extra safe, we will not delete migrations that were created in the last
+minor version before the major upgrade. So, if the we are upgrading to `%14.0`,
+we should not delete migrations that were only added in `%13.11`. This is an
+extra safety net as we expect there are migrations that get merged that may
+take multiple weeks to finish on GitLab.com. It would be bad if we upgraded
+GitLab.com to `%14.0` before the migrations in `%13.11` were finished. Since
+our deployments to GitLab.com are automated and we currently don't have
+automated checks to prevent this, the extra precaution is warranted.
+Additionally, even if we did have automated checks to prevent it, we wouldn't
+actually want to hold up GitLab.com deployments on Advanced Search migrations,
+as they may still have another week to go, and that's too long to block
+deployments.
+
+### Process for removing migrations
+
+For every migration that was created 2 minor versions before the major version
+being upgraded to, we do the following:
+
+1. Confirm the migration has actually completed successfully for GitLab.com.
+1. Replace the content of `migrate` and `completed?` methods as follows:
+
+   ```ruby
+   def migrate
+     log_raise "Migration has been deleted in the last major version upgrade." \
+       "Migrations are supposed to be finished before upgrading major version https://docs.gitlab.com/ee/update/#upgrading-to-a-new-major-version ." \
+       "In order to correct this issue you will need to reacreate your index from scratch https://docs.gitlab.com/ee/integration/elasticsearch.html#last-resort-to-recreate-an-index ."
+   end
+
+   def completed?
+     false
+   end
+   ```
+
+1. Delete any spec files to support this migration.
+1. Remove any logic handling backwards compatibility for this migration. You
+   can find this by looking for
+   `Elastic::DataMigrationService.migration_has_finished?(:migration_name_in_lowercase)`.
+1. Create a merge request with these changes. Noting that we should not
+   accidentally merge this before the major release is started.
+
 ## Performance Monitoring
 
 ### Prometheus

doc/integration/elasticsearch.md

@@ -344,7 +344,7 @@ Sometimes, you might want to abandon the unfinished reindex job and resume the i
 
 1. Uncheck the "Pause Elasticsearch indexing" checkbox in **Admin Area > Settings > Advanced Search**.
 
-## Background migrations
+## Advanced Search migrations
 
 > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/234046) in GitLab 13.6.
 
@@ -352,7 +352,7 @@ With reindex migrations running in the background, there's no need for a manual
 intervention. This usually happens in situations where new features are added to
 Advanced Search, which means adding or changing the way content is indexed.
 
-To confirm that the background migrations ran, you can check with:
+To confirm that the Advanced Search migrations ran, you can check with:
 
 ```shell
 curl "$CLUSTER_URL/gitlab-production-migrations/_search?q=*" | jq .
@@ -402,6 +402,21 @@ debug why the migration was halted and make any changes before retrying the migr
 fixed the cause of the failure, click "Retry migration", and the migration will be scheduled to be retried
 in the background.
 
+If you cannot get the migration to succeed, you may
+consider the [last resort to recreate the index from
+scratch](#last-resort-to-recreate-an-index). This may allow you to skip over
+the problem because a newly created index will skip all migrations as the index
+will be recreated with the correct up-to-date schema.
+
+### All migrations must be finished before doing a major upgrade
+
+Before doing a major version GitLab upgrade, you should have completed all
+migrations that exist up until the latest minor version before that major
+version. If you have halted migrations, these will need to be resolved and
+[retried](#retry-a-halted-migration) before proceeding with a major version
+upgrade. Read more about [upgrading to a new major
+version](../update/index.md#upgrading-to-a-new-major-version).
+
 ## GitLab Advanced Search Rake tasks
 
 Rake tasks are available to:
@@ -889,3 +904,35 @@ Improvements to the `code_analyzer` pattern and filters are being discussed in [
 ### Some binary files may not be searchable by name
 
 In GitLab 13.9, a change was made where [binary file names are being indexed](https://gitlab.com/gitlab-org/gitlab/-/issues/301083). However, without indexing all projects' data from scratch, only binary files that are added or updated after the GitLab 13.9 release are searchable.
+
+### Last resort to recreate an index
+
+There may be cases where somehow data never got indexed and it's not in the
+queue, or the index is somehow in a state where migrations just simply cannot
+proceed. It is always best to try to troubleshoot the root cause of the problem
+using the above [troubleshooting](#troubleshooting) steps.
+
+If there are no other options, then you always have the option of recreating the
+entire index from scratch. If you have a small GitLab installation, this can
+sometimes be a quick way to resolve a problem, but if you have a large GitLab
+installation, then this will likely take a very long time to complete. Until the
+index is fully recreated, your index will not be serving correct search results,
+so you may want to disable **Search with Elasticsearch** while it is running.
+
+If you are sure you've read the above caveats and want to proceed, then you
+should run the following Rake task to recreate the entire index from scratch:
+
+**For Omnibus installations**
+
+```shell
+# WARNING: DO NOT RUN THIS UNTIL YOU READ THE DESCRIPTION ABOVE
+sudo gitlab-rake gitlab:elastic:index
+```
+
+**For installations from source**
+
+```shell
+# WARNING: DO NOT RUN THIS UNTIL YOU READ THE DESCRIPTION ABOVE
+cd /home/git/gitlab
+sudo -u git -H bundle exec rake gitlab:elastic:index
+```

doc/update/index.md

@@ -138,14 +138,16 @@ pending_job_classes = scheduled_queue.select { |job| job["class"] == "Background
 pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) }
 ```
 
-## Checking for pending Elasticsearch migrations
+## Checking for pending Advanced Search migrations
 
 This section is only applicable if you have enabled the [Elasticsearch
 integration](../integration/elasticsearch.md).
 
-Certain major releases might require [Elasticsearch
-migrations](../integration/elasticsearch.md#background-migrations) to be
-finished. You can find pending migrations by running the following command:
+Major releases require all [Advanced Search
+migrations](../integration/elasticsearch.md#advanced-search-migrations) to
+be finished from the most recent minor release in your current version
+before the major version upgrade. You can find pending migrations by
+running the following command:
 
 **For Omnibus installations**
 
@@ -160,7 +162,7 @@ cd /home/git/gitlab
 sudo -u git -H bundle exec rake gitlab:elastic:list_pending_migrations
 ```
 
-### What do I do if my Elasticsearch migrations are stuck?
+### What do I do if my Advanced Search migrations are stuck?
 
 See [how to retry a halted
 migration](../integration/elasticsearch.md#retry-a-halted-migration).
@@ -208,6 +210,13 @@ It's also important to ensure that any background migrations have been fully com
 before upgrading to a new major version. To see the current size of the `background_migration` queue,
 [Check for background migrations before upgrading](#checking-for-background-migrations-before-upgrading).
 
+If you have enabled the [Elasticsearch
+integration](../integration/elasticsearch.md), then you will also need to ensure
+all Advanced Search migrations are completed in the last minor version within
+your current version. Be sure to [check for pending Advanced Search
+migrations](#checking-for-pending-advanced-search-migrations) before proceeding
+with the major version upgrade.
+
 If your GitLab instance has any runners associated with it, it is very
 important to upgrade GitLab Runner to match the GitLab minor version that was
 upgraded to. This is to ensure [compatibility with GitLab versions](https://docs.gitlab.com/runner/#compatibility-with-gitlab-versions).

ee/app/views/admin/application_settings/_elasticsearch_form.html.haml

@@ -28,7 +28,7 @@
                 .gl-alert-body
                   %h4.gl-alert-title= _('There is a halted Elasticsearch migration')
                   = html_escape_once(_('Check the elasticsearch.log file to debug why the migration was halted and make any changes before retrying the migration. When you fix the cause of the failure, click "Retry migration", and the migration will be scheduled to be retried in the background.')).html_safe
-                  = link_to _('Learn more.'), help_page_path('integration/elasticsearch', anchor: 'background-migrations')
+                  = link_to _('Learn more.'), help_page_path('integration/elasticsearch', anchor: 'advanced-search-migrations')
                 .gl-alert-actions
                   - migration = Elastic::DataMigrationService.halted_migration
                   = link_to _('Retry migration'), admin_elasticsearch_retry_migration_path(version: migration.version), class: 'btn gl-alert-action btn-warning gl-button', disabled: @elasticsearch_reindexing_task&.in_progress?, data: { confirm: _('Are you sure you want to retry this migration?') }, method: :post