Commit 334c9108 authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Split Elasticsearch docs to admin and user

parent 387d1fd0
......@@ -54,6 +54,8 @@ Shortcuts to GitLab's most visited docs:
- [Create a group](gitlab-basics/create-group.md)
- [GitLab Subgroups](user/group/subgroups/index.md)
- [Search through GitLab](user/search/index.md): Search for issues, merge requests, projects, groups, todos, and issues in Issue Boards.
- **(EES/EEP)** [Global code search](user/search/global_code_search.md): Leverage Elasticsearch for faster, more advanced code search across your entire GitLab instance.
- **(EES/EEP)** [Advanced search syntax](user/search/advanced_search_syntax.md): Use advanced queries for more targeted search results.
- [Snippets](user/snippets.md): Snippets allow you to create little bits of code.
- [Wikis](user/project/wiki/index.md): Enhance your repository documentation with built-in wikis.
- [GitLab Pages](user/project/pages/index.md): Build, test, and deploy your static website with GitLab Pages.
......@@ -161,7 +163,8 @@ have access to GitLab administration tools and settings.
- [User cohorts](user/admin_area/user_cohorts.md) View user activity over time.
- [Web terminals](administration/integration/terminal.md): Provide terminal access to environments from within GitLab.
- **(EES/EEP)** [Audit logs and events](administration/audit_events.md): View the changes made within the GitLab server.
- **(EES/EEP)** [Elasticsearch](integration/elasticsearch.md): A flexible, scalable and powerful search service to keep GitLab's search fast when dealing with huge amount of data.
- **(EES/EEP)** [Elasticsearch](integration/elasticsearch.md): Enable Elasticsearch,
a flexible, scalable and powerful search service to keep GitLab's search fast when dealing with huge amount of data.
- **(EES/EEP)** [Email users](tools/email.md): Email GitLab users from within GitLab.
- **(EES/EEP)** [Limit project size](user/admin_area/settings/account_and_limit_settings.md): Set a hard limit for your repositories' size.
- **(EEP)** [Auditor users](administration/auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance.
......
# Elasticsearch integration
> Elasticsearch was [Introduced][ee-109] in GitLab EE 8.4. Support for
> [Amazon Elasticsearch][aws-elasticsearch] was [introduced][ee-1305] in GitLab
> EE 9.0.
>
[Introduced][ee-109] in GitLab [Enterprise Edition Starter][ee] 8.4. Support
for [Amazon Elasticsearch][aws-elastic] was [introduced][ee-1305] in GitLab
[Enterprise Edition Starter][ee] 9.0.
## Why do you need this?
This document describes how to set up Elasticsearch with GitLab. Once enabled,
you'll have the benefit of fast search response times and the advantage of two
special searches:
[Elasticsearch] is a flexible, scalable and powerful search service that saves developers time. Instead of developers creating duplicate code and wasting time, they can now search for code within other teams that will help their own project.
## Who needs this?
1. My team uses a plugin to find code from different teams
2. Are developers from different teams creating the same code for their own projects?
3. Are you looking to enable innersourcing?
4. Do you you want to keep GitLab's search fast when dealing with huge amount of data?
If you answered yes to any of these, you should consider [enabling Elasticsearch](#enable-elasticsearch).
GitLab leverages the search capabilities of Elasticsearch and enables it when
searching in:
- GitLab application
- Issues
- Merge requests
- Milestones
- Notes
- Projects
- Repositories
- Snippets
- Wiki
Once the data is added to the database or repository and [Elasticsearch is
enabled in the admin area](#enable-elasticsearch) the search index will be
updated automatically. Elasticsearch can be installed on the same machine as
GitLab, or on a separate server, or you can use the [Amazon Elasticsearch][aws-elastic]
service.
- [Global code search](../user/search/global_code_search.md)
- [Advanced search syntax](../user/search/advanced_search_syntax.md)
## Requirements
......@@ -42,20 +19,26 @@ service.
| GitLab Enterprise Edition 8.4 - 8.17 | Elasticsearch 2.4 with [Delete By Query Plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/2.4/plugins-delete-by-query.html) installed |
| GitLab Enterprise Edition 9.0+ | Elasticsearch 5.1 - 5.3 |
## Install Elasticsearch
## Installing Elasticsearch
Elasticsearch is _not_ included in the Omnibus packages. You will have to
install it yourself whether you are using the Omnibus package or installed
GitLab from source. Providing detailed information on installing Elasticsearch
is out of the scope of this document.
Once the data is added to the database or repository and [Elasticsearch is
enabled in the admin area](#enable-elasticsearch) the search index will be
updated automatically. Elasticsearch can be installed on the same machine as
GitLab, or on a separate server, or you can use the [Amazon Elasticsearch][aws-elastic]
service.
You can follow the steps as described in the [official web site][install] or
use the packages that are available for your OS.
## Enabling Elasticsearch
## Enable Elasticsearch
In order to enable Elasticsearch you need to have access to the server that GitLab is hosted on, and an administrator account on your GitLab instance. Go to **Admin > Settings** and find the "Elasticsearch" section.
In order to enable Elasticsearch, you need to have admin access. Go to
**Admin > Settings** and find the "Elasticsearch" section.
The following Elasticsearch settings are available:
......@@ -70,7 +53,16 @@ The following Elasticsearch settings are available:
| `AWS Access Key` | The AWS access key. |
| `AWS Secret Access Key` | The AWS secret access key. |
## Add GitLab's data to the Elasticsearch index
## Disabling Elasticsearch
To disable the Elasticsearch integration:
1. Navigate to the **Admin area > Settings**
1. Find the 'Elasticsearch' section and uncheck 'Search with Elasticsearch enabled'
and 'Elasticsearch indexing'
1. Click **Save** for the changes to take effect
## Adding GitLab data to the Elasticsearch index
Configure Elasticsearch's host and port in **Admin > Settings**. Then create empty indexes using one of the following commands:
......@@ -192,17 +184,10 @@ sudo gitlab-rake gitlab:elastic:index
bundle exec rake gitlab:elastic:index RAILS_ENV=production
```
## Disable Elasticsearch
Disabling the Elasticsearch integration is as easy as unchecking `Search with Elasticsearch enabled` and `Elasticsearch indexing` in **Admin > Settings**.
## Special recommendations
Here are some tips to use Elasticsearch with GitLab more efficiently.
### Indexing large repositories
Indexing large Git repositories can take a while. To speed up the process, you
......@@ -280,57 +265,58 @@ To minimize downtime of the search feature we recommend the following:
repositories and commits that are already indexed, so it will be much
shorter than the first run.
## Troubleshooting
### Exception "Can't specify parent if no parent field has been configured"
Here are some common pitfalls and how to overcome them:
If you enabled Elasticsearch before GitLab 8.12 and have not rebuilt indexes you will get
exception in lots of different cases:
- **I indexed all the repositories but I can't find anything**
```
Elasticsearch::Transport::Transport::Errors::BadRequest([400] {
"error": {
"root_cause": [{
"type": "illegal_argument_exception",
"reason": "Can't specify parent if no parent field has been configured"
}],
"type": "illegal_argument_exception",
"reason": "Can't specify parent if no parent field has been configured"
},
"status": 400
}):
```
Make sure you indexed all the database data [as stated above](#adding-gitlab-data-to-the-elasticsearch-index).
This is because we changed the index mapping in GitLab 8.12 and the old indexes should be removed and built from scratch again,
see details in the [8-11-to-8-12 update guide](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/update/8.11-to-8.12.md#11-elasticsearch-index-update-if-you-currently-use-elasticsearch).
- **"Can't specify parent if no parent field has been configured"**
### Exception Elasticsearch::Transport::Transport::Errors::BadRequest
If you enabled Elasticsearch before GitLab 8.12 and have not rebuilt indexes you will get
exception in lots of different cases:
If you have this exception (just like in the case above but the actual message is different) please check if you have the correct Elasticsearch version and you met the other [requirements](#requirements).
There is also an easy way to check it automatically with `sudo gitlab-rake gitlab:check` command.
```
Elasticsearch::Transport::Transport::Errors::BadRequest([400] {
"error": {
"root_cause": [{
"type": "illegal_argument_exception",
"reason": "Can't specify parent if no parent field has been configured"
}],
"type": "illegal_argument_exception",
"reason": "Can't specify parent if no parent field has been configured"
},
"status": 400
}):
```
### Exception Elasticsearch::Transport::Transport::Errors::RequestEntityTooLarge
This is because we changed the index mapping in GitLab 8.12 and the old indexes should be removed and built from scratch again,
see details in the [8-11-to-8-12 update guide](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/update/8.11-to-8.12.md#11-elasticsearch-index-update-if-you-currently-use-elasticsearch).
```
[413] {"Message":"Request size exceeded 10485760 bytes"}
```
- Exception `Elasticsearch::Transport::Transport::Errors::BadRequest`
If you have this exception (just like in the case above but the actual message is different) please check if you have the correct Elasticsearch version and you met the other [requirements](#requirements).
There is also an easy way to check it automatically with `sudo gitlab-rake gitlab:check` command.
This exception is seen when your Elasticsearch cluster is configured to reject
requests above a certain size (10MiB in this case). This corresponds to the
`http.max_content_length` setting in `elasticsearch.yml`. Increase it to a
larger size and restart your Elasticsearch cluster.
- Exception `Elasticsearch::Transport::Transport::Errors::RequestEntityTooLarge`
AWS has [fixed limits](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html)
for this setting ("Maximum Size of HTTP Request Payloads"), based on the size of
the underlying instance.
```
[413] {"Message":"Request size exceeded 10485760 bytes"}
```
### I indexed all the repositories but I can't find anything
This exception is seen when your Elasticsearch cluster is configured to reject
requests above a certain size (10MiB in this case). This corresponds to the
`http.max_content_length` setting in `elasticsearch.yml`. Increase it to a
larger size and restart your Elasticsearch cluster.
Make sure you indexed all the database data as stated above (`sudo gitlab-rake gitlab:elastic:index`)
AWS has [fixed limits](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html)
for this setting ("Maximum Size of HTTP Request Payloads"), based on the size of
the underlying instance.
[ee-1305]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1305
[aws-elasticsearch]: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html
[aws-elastic]: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html
[aws-iam]: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
[aws-instance-profile]: http://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli
[ee-109]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/109 "Elasticsearch Merge Request"
......@@ -338,3 +324,4 @@ Make sure you indexed all the database data as stated above (`sudo gitlab-rake g
[install]: https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html "Elasticsearch installation documentation"
[pkg]: https://about.gitlab.com/downloads/ "Download Omnibus GitLab"
[elastic-settings]: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html#settings "Elasticsearch configuration settings"
[ee]: https://about.gitlab.com/gitlab-ee/
......@@ -2,12 +2,23 @@
>**Notes:**
- Introduced in [GitLab Enterprise Starter][ee] 9.2
- The [Elasticsearch integration](../../integration/elasticsearch.md) needs to
be enabled
- This is the user documentation. To install and configure Elasticsearch,
visit the [admin docs](../../integration/elasticsearch.md).
Full details can be found in the
[Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html#_simple_query_string_syntax)
but here's a quick guide:
Use advanced queries for more targeted search results.
## Overview
The advanced search syntax is a subset of the [global code search](global_code_search.md)
which you can use to have more specific search results.
## Use cases
## Using the advanced search syntax
Full details can be found in the [Elasticsearch documentation][elastic], but
here's a quick guide:
* Searches look for all the words in a query, in any order - e.g.: searching
issues for `display bug` will return all issues matching both those words, in any order.
......@@ -19,3 +30,4 @@ but here's a quick guide:
* To find a term containing one of these symbols, use `\`: `argument \-last`
[ee]: https://about.gitlab.com/gitlab-ee/
[elastic]: https://www.elastic.co/guide/en/elasticsearch/reference/5.3/query-dsl-simple-query-string-query.html#_simple_query_string_syntax
# Global code search
>
- [Introduced][ee-109] in GitLab [Enterprise Edition Starter][ee] 8.4.
- This is the user documentation. To install and configure Elasticsearch,
visit the [admin docs](../../integration/elasticsearch.md).
Leverage Elasticsearch for faster, more advanced code search across your entire
GitLab instance.
## Overview
Global code search in GitLab is a powerful search service that saves developers
time. Instead of creating duplicate code and wasting time, they can now search
for code within other teams that will help their own project.
GitLab leverages the search capabilities of Elasticsearch and enables it when
searching in:
- GitLab application
- Issues
- Merge requests
- Milestones
- Notes
- Projects
- Repositories
- Snippets
- Wiki
## Use cases
1. If you are looking to enable innersourcing, with Global code search ...
1. You want to keep GitLab's search fast when dealing with huge amount of data.
## Searching globally
You can also use the [advanced search syntax](advanced_search_syntax.md) which
provides some useful queries.
[ee-1305]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1305
[aws-elastic]: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html
[aws-iam]: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
[aws-instance-profile]: http://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli
[ee-109]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/109 "Elasticsearch Merge Request"
[elasticsearch]: https://www.elastic.co/products/elasticsearch "Elasticsearch website"
[install]: https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html "Elasticsearch installation documentation"
[pkg]: https://about.gitlab.com/downloads/ "Download Omnibus GitLab"
[elastic-settings]: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html#settings "Elasticsearch configuration settings"
[ee]: https://about.gitlab.com/gitlab-ee/
......@@ -103,12 +103,16 @@ and **Labels**, select multiple issues to add to a list of your choice:
![search and select issues to add to board](img/search_issues_board.png)
## Advanced search syntax
## Global code search
Leverage Elasticsearch for faster, more advanced code search across your entire
GitLab instance.
> Available in [GitLab Enterprise Edition Premium][ee].
[Learn how to use the global code search.](global_code_search.md)
## Advanced search syntax
If the [Elasticsearch integration](../../integration/elasticsearch.md) is
enabled, then some advanced search functionality is available.
Use advanced queries for more targeted search results.
[Learn how to use the advanced search syntax.](advanced_search_syntax.md)
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment