Merge branch '213628-es-use-alias' into 'master'

Elasticsearch integration should only ever read/write from alias

See merge request gitlab-org/gitlab!32107

doc/integration/elasticsearch.md

@@ -418,14 +418,14 @@ The following are some available Rake tasks:
 | [`sudo gitlab-rake gitlab:elastic:index_projects`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)                   | Iterates over all projects and queues Sidekiq jobs to index them in the background.                                                                                                       |
 | [`sudo gitlab-rake gitlab:elastic:index_projects_status`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)            | Determines the overall status of the indexing. It is done by counting the total number of indexed projects, dividing by a count of the total number of projects, then multiplying by 100. |
 | [`sudo gitlab-rake gitlab:elastic:clear_index_status`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)               | Deletes all instances of IndexStatus for all projects.                                                                                                                                    |
-| [`sudo gitlab-rake gitlab:elastic:create_empty_index[<INDEX_NAME>]`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Generates an empty index on the Elasticsearch side only if it doesn't already exist.                                                                                                      |
-| [`sudo gitlab-rake gitlab:elastic:delete_index[<INDEX_NAME>]`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)       | Removes the GitLab index on the Elasticsearch instance.                                                                                                                                   |
-| [`sudo gitlab-rake gitlab:elastic:recreate_index[<INDEX_NAME>]`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)     | Wrapper task for `gitlab:elastic:delete_index[<INDEX_NAME>]` and `gitlab:elastic:create_empty_index[<INDEX_NAME>]`.                                                                       |
+| [`sudo gitlab-rake gitlab:elastic:create_empty_index[<TARGET_NAME>]`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Generates an empty index and assigns an alias for it on the Elasticsearch side only if it doesn't already exist.                                                                                                      |
+| [`sudo gitlab-rake gitlab:elastic:delete_index[<TARGET_NAME>]`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)       | Removes the GitLab index and alias (if exists) on the Elasticsearch instance.                                                                                                                                   |
+| [`sudo gitlab-rake gitlab:elastic:recreate_index[<TARGET_NAME>]`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)     | Wrapper task for `gitlab:elastic:delete_index[<TARGET_NAME>]` and `gitlab:elastic:create_empty_index[<TARGET_NAME>]`.                                                                       |
 | [`sudo gitlab-rake gitlab:elastic:index_snippets`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)                   | Performs an Elasticsearch import that indexes the snippets data.                                                                                                                          |
 | [`sudo gitlab-rake gitlab:elastic:projects_not_indexed`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake)             | Displays which projects are not indexed.                                                                                                                                                  |
 
 NOTE: **Note:**
-The `INDEX_NAME` parameter is optional and will use the default index name from the current `RAILS_ENV` if not set.
+The `TARGET_NAME` parameter is optional and will use the default index/alias name from the current `RAILS_ENV` if not set.
 
 ### Environment variables
 

ee/changelogs/unreleased/213628-es-use-alias.yml

+---
+title: Elasticsearch integration started using aliases instead of using indices directly
+merge_request: 32107
+author:
+type: changed

ee/lib/gitlab/elastic/helper.rb

@@ -4,17 +4,17 @@ module Gitlab
   module Elastic
     class Helper
       attr_reader :version, :client
-      attr_accessor :index_name
+      attr_accessor :target_name
 
       def initialize(
         version: ::Elastic::MultiVersionUtil::TARGET_VERSION,
         client: nil,
-        index_name: nil)
+        target_name: nil)
 
         proxy = self.class.create_proxy(version)
 
         @client = client || proxy.client
-        @index_name = index_name || proxy.index_name
+        @target_name = target_name || proxy.index_name
         @version = version
       end
 
@@ -28,11 +28,16 @@ module Gitlab
         end
       end
 
-      # rubocop: disable CodeReuse/ActiveRecord
-      def create_empty_index
+      def create_empty_index(with_alias: true)
+        if index_exists?
+          raise "Index under '#{target_name}' already exists, use `recreate_index` to recreate it."
+        end
+
         settings = {}
         mappings = {}
 
+        new_index_name = with_alias ? "#{target_name}-#{Time.now.strftime("%Y%m%d-%H%M%S")}" : target_name
+
         [
           Project,
           Issue,
@@ -48,7 +53,7 @@ module Gitlab
         end
 
         create_index_options = {
-          index: index_name,
+          index: new_index_name,
           body: {
             settings: settings.to_hash,
             mappings: mappings.to_hash
@@ -65,16 +70,12 @@ module Gitlab
           create_index_options[:include_type_name] = true
         end
 
-        if client.indices.exists?(index: index_name)
-          raise "Index '#{index_name}' already exists, use `recreate_index` to recreate it."
-        end
-
         client.indices.create create_index_options
+        client.indices.put_alias(name: target_name, index: new_index_name) if with_alias
       end
-      # rubocop: enable CodeReuse/ActiveRecord
 
       def delete_index
-        result = client.indices.delete(index: index_name)
+        result = client.indices.delete(index: target_index_name)
         result['acknowledged']
       rescue ::Elasticsearch::Transport::Transport::Errors::NotFound => e
         Gitlab::ErrorTracking.log_exception(e)
@@ -82,18 +83,33 @@ module Gitlab
       end
 
       def index_exists?
-        client.indices.exists?(index: index_name) # rubocop:disable CodeReuse/ActiveRecord
+        client.indices.exists?(index: target_name) # rubocop:disable CodeReuse/ActiveRecord
+      end
+
+      def alias_exists?
+        client.indices.exists_alias(name: target_name)
       end
 
       # Calls Elasticsearch refresh API to ensure data is searchable
       # immediately.
       # https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html
       def refresh_index
-        client.indices.refresh(index: index_name)
+        client.indices.refresh(index: target_name)
       end
 
       def index_size
-        client.indices.stats['indices'][index_name]['total']
+        client.indices.stats['indices'][target_index_name]['total']
+      end
+
+      private
+
+      # This method is used when we need to get an actual index name (if it's used through an alias)
+      def target_index_name
+        if alias_exists?
+          client.indices.get_alias(name: target_name).each_key.first
+        else
+          target_name
+        end
       end
     end
   end

ee/lib/tasks/gitlab/elastic.rake

@@ -55,27 +55,27 @@ namespace :gitlab do
       logger.info("Indexing snippets... " + "done".color(:green))
     end
 
-    desc "GitLab | Elasticsearch | Create empty index"
-    task :create_empty_index, [:index_name] => [:environment] do |t, args|
-      helper = Gitlab::Elastic::Helper.new(index_name: args[:index_name])
+    desc "GitLab | Elasticsearch | Create empty index and assign alias"
+    task :create_empty_index, [:target_name] => [:environment] do |t, args|
+      helper = Gitlab::Elastic::Helper.new(target_name: args[:target_name])
       helper.create_empty_index
 
-      puts "Index '#{helper.index_name}' has been created.".color(:green)
+      puts "Index and underlying alias '#{helper.target_name}' has been created.".color(:green)
     end
 
     desc "GitLab | Elasticsearch | Delete index"
-    task :delete_index, [:index_name] => [:environment] do |t, args|
-      helper = Gitlab::Elastic::Helper.new(index_name: args[:index_name])
+    task :delete_index, [:target_name] => [:environment] do |t, args|
+      helper = Gitlab::Elastic::Helper.new(target_name: args[:target_name])
 
       if helper.delete_index
-        puts "Index '#{helper.index_name}' has been deleted".color(:green)
+        puts "Index/alias '#{helper.target_name}' has been deleted".color(:green)
       else
-        puts "Index '#{helper.index_name}' was not found".color(:green)
+        puts "Index/alias '#{helper.target_name}' was not found".color(:green)
       end
     end
 
     desc "GitLab | Elasticsearch | Recreate index"
-    task :recreate_index, [:index_name] => [:environment] do |t, args|
+    task :recreate_index, [:target_name] => [:environment] do |t, args|
       Rake::Task["gitlab:elastic:delete_index"].invoke(*args)
       Rake::Task["gitlab:elastic:create_empty_index"].invoke(*args)
     end

ee/spec/lib/ee/gitlab/elastic/helper_spec.rb

@@ -5,9 +5,15 @@ require 'spec_helper'
 describe Gitlab::Elastic::Helper do
   subject(:helper) { described_class.default }
 
-  shared_context 'with an existing index' do
+  shared_context 'with a legacy index' do
     before do
-      helper.create_empty_index
+      helper.create_empty_index(with_alias: false)
+    end
+  end
+
+  shared_context 'with an existing index and alias' do
+    before do
+      helper.create_empty_index(with_alias: true)
     end
   end
 
@@ -19,34 +25,50 @@ describe Gitlab::Elastic::Helper do
     it 'has the proper default values' do
       expect(helper).to have_attributes(
         version: ::Elastic::MultiVersionUtil::TARGET_VERSION,
-        index_name: ::Elastic::Latest::Config.index_name)
+        target_name: ::Elastic::Latest::Config.index_name)
     end
 
     context 'with a custom `index_name`' do
       let(:index_name) { 'custom-index-name' }
 
-      subject(:helper) { described_class.new(index_name: index_name) }
+      subject(:helper) { described_class.new(target_name: index_name) }
 
       it 'has the proper `index_name`' do
-        expect(helper).to have_attributes(index_name: index_name)
+        expect(helper).to have_attributes(target_name: index_name)
       end
     end
   end
 
   describe '#create_empty_index' do
-    context 'without an existing index' do
-      it 'creates the index' do
+    context 'with an empty cluster' do
+      it 'creates index and alias' do
         helper.create_empty_index
 
         expect(helper.index_exists?).to eq(true)
+        expect(helper.alias_exists?).to eq(true)
+      end
+
+      it 'creates the index only' do
+        helper.create_empty_index(with_alias: false)
+
+        expect(helper.index_exists?).to eq(true)
+        expect(helper.alias_exists?).to eq(false)
       end
     end
 
-    context 'when there is an index' do
-      include_context 'with an existing index'
+    context 'when there is an alias' do
+      include_context 'with an existing index and alias'
 
       it 'raises an error' do
-        expect { helper.create_empty_index }.to raise_error
+        expect { helper.create_empty_index }.to raise_error(RuntimeError)
+      end
+    end
+
+    context 'when there is a legacy index' do
+      include_context 'with a legacy index'
+
+      it 'raises an error' do
+        expect { helper.create_empty_index }.to raise_error(RuntimeError)
       end
     end
   end
@@ -60,8 +82,14 @@ describe Gitlab::Elastic::Helper do
       end
     end
 
-    context 'when there is an index' do
-      include_context 'with an existing index'
+    context 'when there is an alias' do
+      include_context 'with an existing index and alias'
+
+      it { is_expected.to be_truthy }
+    end
+
+    context 'when there is a legacy index' do
+      include_context 'with a legacy index'
 
       it { is_expected.to be_truthy }
     end
@@ -74,8 +102,34 @@ describe Gitlab::Elastic::Helper do
       it { is_expected.to be_falsy }
     end
 
-    context 'when there is an index' do
-      include_context 'with an existing index'
+    context 'when there is a legacy index' do
+      include_context 'with a legacy index'
+
+      it { is_expected.to be_truthy }
+    end
+
+    context 'when there is an alias' do
+      include_context 'with an existing index and alias'
+
+      it { is_expected.to be_truthy }
+    end
+  end
+
+  describe '#alias_exists?' do
+    subject { helper.alias_exists? }
+
+    context 'without an existing index' do
+      it { is_expected.to be_falsy }
+    end
+
+    context 'when there is a legacy index' do
+      include_context 'with a legacy index'
+
+      it { is_expected.to be_falsy }
+    end
+
+    context 'when there is an alias' do
+      include_context 'with an existing index and alias'
 
       it { is_expected.to be_truthy }
     end