Commit 03c8bb12 authored by Nick Thomas's avatar Nick Thomas

Merge branch '328-versioned-search-ee' into 'master'

Elasticsearch versioned schema for Snippet

See merge request gitlab-org/gitlab-ce!31465
parents 13581fe0 235b4e01
......@@ -5,46 +5,42 @@
require 'gitlab/current_settings'
Gitlab.ee do
require 'elasticsearch/model'
### Modified from elasticsearch-model/lib/elasticsearch/model.rb
[
Elasticsearch::Model::Client::ClassMethods,
Elasticsearch::Model::Naming::ClassMethods,
Elasticsearch::Model::Indexing::ClassMethods,
Elasticsearch::Model::Searching::ClassMethods
].each do |mod|
Elasticsearch::Model::Proxy::ClassMethodsProxy.include mod
end
[
Elasticsearch::Model::Client::InstanceMethods,
Elasticsearch::Model::Naming::InstanceMethods,
Elasticsearch::Model::Indexing::InstanceMethods,
Elasticsearch::Model::Serializing::InstanceMethods
].each do |mod|
Elasticsearch::Model::Proxy::InstanceMethodsProxy.include mod
end
Elasticsearch::Model::Proxy::InstanceMethodsProxy.class_eval <<-CODE, __FILE__, __LINE__ + 1
def as_indexed_json(options={})
target.respond_to?(:as_indexed_json) ? target.__send__(:as_indexed_json, options) : super
end
CODE
### Monkey patches
Elasticsearch::Model::Response::Records.prepend GemExtensions::Elasticsearch::Model::Response::Records
Elasticsearch::Model::Adapter::Multiple::Records.prepend GemExtensions::Elasticsearch::Model::Adapter::Multiple::Records
Elasticsearch::Model::Indexing::InstanceMethods.prepend GemExtensions::Elasticsearch::Model::Indexing::InstanceMethods
module Elasticsearch
module Model
module Client
# This mutex is only used to synchronize *creation* of a new client, so
# all including classes can share the same client instance
CLIENT_MUTEX = Mutex.new
cattr_accessor :cached_client
cattr_accessor :cached_config
module ClassMethods
# Override the default ::Elasticsearch::Model::Client implementation to
# return a client configured from application settings. All including
# classes will use the same instance, which is refreshed automatically
# if the settings change.
#
# _client is present to match the arity of the overridden method, where
# it is also not used.
#
# @return [Elasticsearch::Transport::Client]
def client(_client = nil)
store = ::Elasticsearch::Model::Client
store::CLIENT_MUTEX.synchronize do
config = Gitlab::CurrentSettings.elasticsearch_config
if store.cached_client.nil? || config != store.cached_config
store.cached_client = ::Gitlab::Elastic::Client.build(config)
store.cached_config = config
end
end
store.cached_client
end
end
end
end
end
Elasticsearch::Model::Adapter::ActiveRecord::Importing.prepend GemExtensions::Elasticsearch::Model::Adapter::ActiveRecord::Importing
Elasticsearch::Model::Client::InstanceMethods.prepend GemExtensions::Elasticsearch::Model::Client
Elasticsearch::Model::Client::ClassMethods.prepend GemExtensions::Elasticsearch::Model::Client
Elasticsearch::Model::ClassMethods.prepend GemExtensions::Elasticsearch::Model::Client
Elasticsearch::Model.singleton_class.prepend GemExtensions::Elasticsearch::Model::Client
end
......@@ -148,6 +148,36 @@ Uses an [Edge NGram token filter](https://www.elastic.co/guide/en/elasticsearch/
- Searches can have their own analyzers. Remember to check when editing analyzers
- `Character` filters (as opposed to token filters) always replace the original character, so they're not a good choice as they can hinder exact searches
## Architecture
GitLab uses `elasticsearch-rails` for handling communication with Elasticsearch server. However, in order to achieve zero-downtime deployment during schema changes, an extra abstraction layer is built to allow:
* Indexing (writes) to multiple indexes, with different mappings
* Switching to different index for searches (reads) on the fly
Currently we are on the process of migrating models to this new design (e.g. `Snippet`), and it is hardwired to work with a single version for now.
Traditionally, `elasticsearch-rails` provides class and instance level `__elasticsearch__` proxy methods. If you call `Issue.__elasticsearch__`, you will get an instance of `Elasticsearch::Model::Proxy::ClassMethodsProxy`, and if you call `Issue.first.__elasticsearch__`, you will get an instance of `Elasticsearch::Model::Proxy::InstanceMethodsProxy`. These proxy objects would talk to Elasticsearch server directly.
In the new design, `__elasticsearch__` instead represents one extra layer of proxy. It would keep multiple versions of the actual proxy objects, and it would forward read and write calls to the proxy of the intended version.
The `elasticsearch-rails`'s way of specifying each model's mappings and other settings is to create a module for the model to include. However in the new design, each model would have its own corresponding subclassed proxy object, where the settings reside in. For example, snippet related setting in the past reside in `SnippetsSearch` module, but in the new design would reside in `SnippetClassProxy` (which is a subclass of `Elasticsearch::Model::Proxy::ClassMethodsProxy`). This reduces namespace pollution in model classes.
The global configurations per version are now in the `Elastic::(Version)::Config` class. You can change mappings there.
### Creating new version of schema
Currently GitLab would still work with a single version of setting. Once it is implemented, multiple versions of setting can exists in different folders (e.g. `ee/lib/elastic/v12p1` and `ee/lib/elastic/v12p3`). To keep a continuous git history, the latest version lives under the `/latest` folder, but is aliased as the latest version.
If the current version is `v12p1`, and we need to create a new version for `v12p3`, the steps are as follows:
1. Copy the entire folder of `v12p1` as `v12p3`
1. Change the namespace for files under `v12p3` folder from `V12p1` to `V12p3` (which are still aliased to `Latest`)
1. Delete `v12p1` folder
1. Copy the entire folder of `latest` as `v12p1`
1. Change the namespace for files under `v12p1` folder from `Latest` to `V12p1`
1. Make changes to `Latest` as needed
## Troubleshooting
### Getting `flood stage disk watermark [95%] exceeded`
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment