Commit c59047da authored by Achilleas Pipinellis's avatar Achilleas Pipinellis

Merge branch 'docs/elastic' into 'master'

Split Elasticsearch docs to admin and user

Closes #2056

See merge request !2375
parents 43d59a66 4976b9ea
......@@ -54,6 +54,8 @@ Shortcuts to GitLab's most visited docs:
- [Create a group](gitlab-basics/create-group.md)
- [GitLab Subgroups](user/group/subgroups/index.md)
- [Search through GitLab](user/search/index.md): Search for issues, merge requests, projects, groups, todos, and issues in Issue Boards.
- **(EES/EEP)** [Advanced Global Search](user/search/advanced_global_search.md): Leverage Elasticsearch for faster, more advanced code search across your entire GitLab instance.
- **(EES/EEP)** [Advanced Syntax Search](user/search/advanced_search_syntax.md): Use advanced queries for more targeted search results.
- [Snippets](user/snippets.md): Snippets allow you to create little bits of code.
- [Wikis](user/project/wiki/index.md): Enhance your repository documentation with built-in wikis.
- [GitLab Pages](user/project/pages/index.md): Build, test, and deploy your static website with GitLab Pages.
......@@ -161,7 +163,7 @@ have access to GitLab administration tools and settings.
- [User cohorts](user/admin_area/user_cohorts.md) View user activity over time.
- [Web terminals](administration/integration/terminal.md): Provide terminal access to environments from within GitLab.
- **(EES/EEP)** [Audit logs and events](administration/audit_events.md): View the changes made within the GitLab server.
- **(EES/EEP)** [Elasticsearch](integration/elasticsearch.md): A flexible, scalable and powerful search service to keep GitLab's search fast when dealing with huge amount of data.
- **(EES/EEP)** [Elasticsearch](integration/elasticsearch.md): Enable Elasticsearch which powers GitLab's Advanced Global Search. Useful when you deal with a huge amount of data.
- **(EES/EEP)** [Email users](tools/email.md): Email GitLab users from within GitLab.
- **(EES/EEP)** [Limit project size](user/admin_area/settings/account_and_limit_settings.md): Set a hard limit for your repositories' size.
- **(EEP)** [Auditor users](administration/auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance.
......
# Elasticsearch integration
> Elasticsearch was [Introduced][ee-109] in GitLab EE 8.4. Support for
> [Amazon Elasticsearch][aws-elasticsearch] was [introduced][ee-1305] in GitLab
> EE 9.0.
>
[Introduced][ee-109] in GitLab [Enterprise Edition Starter][ee] 8.4. Support
for [Amazon Elasticsearch][aws-elastic] was [introduced][ee-1305] in GitLab
[Enterprise Edition Starter][ee] 9.0.
## Why do you need this?
This document describes how to set up Elasticsearch with GitLab. Once enabled,
you'll have the benefit of fast search response times and the advantage of two
special searches:
[Elasticsearch] is a flexible, scalable and powerful search service that saves developers time. Instead of developers creating duplicate code and wasting time, they can now search for code within other teams that will help their own project.
## Who needs this?
1. My team uses a plugin to find code from different teams
2. Are developers from different teams creating the same code for their own projects?
3. Are you looking to enable innersourcing?
4. Do you you want to keep GitLab's search fast when dealing with huge amount of data?
If you answered yes to any of these, you should consider [enabling Elasticsearch](#enable-elasticsearch).
GitLab leverages the search capabilities of Elasticsearch and enables it when
searching in:
- GitLab application
- Issues
- Merge requests
- Milestones
- Notes
- Projects
- Repositories
- Snippets
- Wiki
Once the data is added to the database or repository and [Elasticsearch is
enabled in the admin area](#enable-elasticsearch) the search index will be
updated automatically. Elasticsearch can be installed on the same machine as
GitLab, or on a separate server, or you can use the [Amazon Elasticsearch][aws-elastic]
service.
- [Advance Global Search](../user/search/advanced_global_search.md)
- [Advanced Syntax Search](../user/search/advanced_search_syntax.md)
## Requirements
......@@ -42,20 +19,26 @@ service.
| GitLab Enterprise Edition 8.4 - 8.17 | Elasticsearch 2.4 with [Delete By Query Plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/2.4/plugins-delete-by-query.html) installed |
| GitLab Enterprise Edition 9.0+ | Elasticsearch 5.1 - 5.3 |
## Install Elasticsearch
## Installing Elasticsearch
Elasticsearch is _not_ included in the Omnibus packages. You will have to
install it yourself whether you are using the Omnibus package or installed
GitLab from source. Providing detailed information on installing Elasticsearch
is out of the scope of this document.
Once the data is added to the database or repository and [Elasticsearch is
enabled in the admin area](#enable-elasticsearch) the search index will be
updated automatically. Elasticsearch can be installed on the same machine as
GitLab, or on a separate server, or you can use the [Amazon Elasticsearch][aws-elastic]
service.
You can follow the steps as described in the [official web site][install] or
use the packages that are available for your OS.
## Enabling Elasticsearch
## Enable Elasticsearch
In order to enable Elasticsearch you need to have access to the server that GitLab is hosted on, and an administrator account on your GitLab instance. Go to **Admin > Settings** and find the "Elasticsearch" section.
In order to enable Elasticsearch, you need to have admin access. Go to
**Admin > Settings** and find the "Elasticsearch" section.
The following Elasticsearch settings are available:
......@@ -70,7 +53,16 @@ The following Elasticsearch settings are available:
| `AWS Access Key` | The AWS access key. |
| `AWS Secret Access Key` | The AWS secret access key. |
## Add GitLab's data to the Elasticsearch index
## Disabling Elasticsearch
To disable the Elasticsearch integration:
1. Navigate to the **Admin area > Settings**
1. Find the 'Elasticsearch' section and uncheck 'Search with Elasticsearch enabled'
and 'Elasticsearch indexing'
1. Click **Save** for the changes to take effect
## Adding GitLab data to the Elasticsearch index
Configure Elasticsearch's host and port in **Admin > Settings**. Then create empty indexes using one of the following commands:
......@@ -192,17 +184,10 @@ sudo gitlab-rake gitlab:elastic:index
bundle exec rake gitlab:elastic:index RAILS_ENV=production
```
## Disable Elasticsearch
Disabling the Elasticsearch integration is as easy as unchecking `Search with Elasticsearch enabled` and `Elasticsearch indexing` in **Admin > Settings**.
## Special recommendations
Here are some tips to use Elasticsearch with GitLab more efficiently.
### Indexing large repositories
Indexing large Git repositories can take a while. To speed up the process, you
......@@ -280,16 +265,21 @@ To minimize downtime of the search feature we recommend the following:
repositories and commits that are already indexed, so it will be much
shorter than the first run.
## Troubleshooting
### Exception "Can't specify parent if no parent field has been configured"
Here are some common pitfalls and how to overcome them:
If you enabled Elasticsearch before GitLab 8.12 and have not rebuilt indexes you will get
exception in lots of different cases:
- **I indexed all the repositories but I can't find anything**
```
Elasticsearch::Transport::Transport::Errors::BadRequest([400] {
Make sure you indexed all the database data [as stated above](#adding-gitlab-data-to-the-elasticsearch-index).
- **"Can't specify parent if no parent field has been configured"**
If you enabled Elasticsearch before GitLab 8.12 and have not rebuilt indexes you will get
exception in lots of different cases:
```
Elasticsearch::Transport::Transport::Errors::BadRequest([400] {
"error": {
"root_cause": [{
"type": "illegal_argument_exception",
......@@ -299,38 +289,34 @@ Elasticsearch::Transport::Transport::Errors::BadRequest([400] {
"reason": "Can't specify parent if no parent field has been configured"
},
"status": 400
}):
```
}):
```
This is because we changed the index mapping in GitLab 8.12 and the old indexes should be removed and built from scratch again,
see details in the [8-11-to-8-12 update guide](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/update/8.11-to-8.12.md#11-elasticsearch-index-update-if-you-currently-use-elasticsearch).
This is because we changed the index mapping in GitLab 8.12 and the old indexes should be removed and built from scratch again,
see details in the [8-11-to-8-12 update guide](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/update/8.11-to-8.12.md#11-elasticsearch-index-update-if-you-currently-use-elasticsearch).
### Exception Elasticsearch::Transport::Transport::Errors::BadRequest
- Exception `Elasticsearch::Transport::Transport::Errors::BadRequest`
If you have this exception (just like in the case above but the actual message is different) please check if you have the correct Elasticsearch version and you met the other [requirements](#requirements).
There is also an easy way to check it automatically with `sudo gitlab-rake gitlab:check` command.
If you have this exception (just like in the case above but the actual message is different) please check if you have the correct Elasticsearch version and you met the other [requirements](#requirements).
There is also an easy way to check it automatically with `sudo gitlab-rake gitlab:check` command.
### Exception Elasticsearch::Transport::Transport::Errors::RequestEntityTooLarge
- Exception `Elasticsearch::Transport::Transport::Errors::RequestEntityTooLarge`
```
[413] {"Message":"Request size exceeded 10485760 bytes"}
```
This exception is seen when your Elasticsearch cluster is configured to reject
requests above a certain size (10MiB in this case). This corresponds to the
`http.max_content_length` setting in `elasticsearch.yml`. Increase it to a
larger size and restart your Elasticsearch cluster.
AWS has [fixed limits](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html)
for this setting ("Maximum Size of HTTP Request Payloads"), based on the size of
the underlying instance.
```
[413] {"Message":"Request size exceeded 10485760 bytes"}
```
### I indexed all the repositories but I can't find anything
This exception is seen when your Elasticsearch cluster is configured to reject
requests above a certain size (10MiB in this case). This corresponds to the
`http.max_content_length` setting in `elasticsearch.yml`. Increase it to a
larger size and restart your Elasticsearch cluster.
Make sure you indexed all the database data as stated above (`sudo gitlab-rake gitlab:elastic:index`)
AWS has [fixed limits](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html)
for this setting ("Maximum Size of HTTP Request Payloads"), based on the size of
the underlying instance.
[ee-1305]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1305
[aws-elasticsearch]: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html
[aws-elastic]: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html
[aws-iam]: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
[aws-instance-profile]: http://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli
[ee-109]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/109 "Elasticsearch Merge Request"
......@@ -338,3 +324,4 @@ Make sure you indexed all the database data as stated above (`sudo gitlab-rake g
[install]: https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html "Elasticsearch installation documentation"
[pkg]: https://about.gitlab.com/downloads/ "Download Omnibus GitLab"
[elastic-settings]: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html#settings "Elasticsearch configuration settings"
[ee]: https://about.gitlab.com/gitlab-ee/
# Advanced Global Search
>
- [Introduced][ee-109] in GitLab [Enterprise Edition Starter][ee] 8.4.
- This is the user documentation. To install and configure Elasticsearch,
visit the [admin docs](../../integration/elasticsearch.md).
Leverage Elasticsearch for faster, more advanced code search across your entire
GitLab instance.
## Overview
The Advanced Global Search in GitLab is a powerful search service that saves
you time. Instead of creating duplicate code and wasting time, you can
now search for code within other teams that can help your own project.
GitLab leverages the search capabilities of [Elasticsearch] and enables it when
searching in:
- GitLab application
- Projects
- Repositories
- Commits
- Issues
- Merge requests
- Milestones
- Notes (comments)
- Snippets
- Wiki
## Use cases
The Advanced Global Search can be useful in various scenarios.
### Faster searches
If you are dealing with huge amount of data and want to keep GitLab's search
fast, the Advanced Global Search will help you achieve that.
### Promote innersourcing
Your company may consist of many different developer teams each of which has
their own group where the various projects are hosted. Some of your applications
may be connected to each other, so your developers need to instantly search
throughout the GitLab instance and find the code they search for.
## Searching globally
Just use the search as before and GitLab will show you matching code from each
project you have access to.
![Advanced Global Search](img/advanced_global_search.png)
You can also use the [Advanced Syntax Search](advanced_search_syntax.md) which
provides some useful queries.
>**Note:**
Elasticsearch has only data for the default branch. That means that if you go
to the repository tree and switch the branch from the default to something else,
then the "Code" tab in the search result page will be served by the regular
search even if Elasticsearch is enabled.
[ee-1305]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1305
[aws-elastic]: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html
[aws-iam]: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html
[aws-instance-profile]: http://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli
[ee-109]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/109 "Elasticsearch Merge Request"
[elasticsearch]: https://www.elastic.co/products/elasticsearch "Elasticsearch website"
[install]: https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html "Elasticsearch installation documentation"
[pkg]: https://about.gitlab.com/downloads/ "Download Omnibus GitLab"
[elastic-settings]: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html#settings "Elasticsearch configuration settings"
[ee]: https://about.gitlab.com/gitlab-ee/
[es]: https://www.elastic.co/products/elasticsearch
# Advanced search syntax
# Advanced Syntax Search
>**Notes:**
- Introduced in [GitLab Enterprise Starter][ee] 9.2
- The [Elasticsearch integration](../../integration/elasticsearch.md) needs to
be enabled
- This is the user documentation. To install and configure Elasticsearch,
visit the [admin docs](../../integration/elasticsearch.md).
Full details can be found in the
[Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html#_simple_query_string_syntax)
but here's a quick guide:
Use advanced queries for more targeted search results.
## Overview
The Advanced Syntax Search is a subset of the
[Advanced Global Search](advanced_global_search.md), which you can use if you
want to have more specific search results.
## Use cases
Let's say for example that the product you develop relies on the code of another
product that's hosted under some other group.
Since under your GitLab instance there are hosted hundreds of different projects,
you need the search results to be as efficient as possible. You have a feeling
of what you want to find (e.g., a function name), but at the same you're also
not so sure.
In that case, using the regular expressions in your query will yield much better
results.
## Using the Advanced Syntax Search
The Advanced Syntax Search supports queries of ranges, wildcards, regular
expressions, fuzziness and much more.
Full details can be found in the [Elasticsearch documentation][elastic], but
here's a quick guide:
* Searches look for all the words in a query, in any order - e.g.: searching
issues for `display bug` will return all issues matching both those words, in any order.
......@@ -17,5 +42,7 @@ but here's a quick guide:
* To group terms together, use parentheses: `bug | (display +sound)`
* To match a partial word, use `*`: `bug find_by_*`
* To find a term containing one of these symbols, use `\`: `argument \-last`
* To limit the results based on the time "created_at:[2012-01-01 TO 2012-12-31]" and other sweet stuff
[ee]: https://about.gitlab.com/gitlab-ee/
[elastic]: https://www.elastic.co/guide/en/elasticsearch/reference/5.3/query-dsl-simple-query-string-query.html#_simple_query_string_syntax
......@@ -103,13 +103,17 @@ and **Labels**, select multiple issues to add to a list of your choice:
![search and select issues to add to board](img/search_issues_board.png)
## Advanced search syntax
## Advanced Global Search
> Available in [GitLab Enterprise Edition Premium][ee].
Leverage Elasticsearch for faster, more advanced code search across your entire
GitLab instance.
If the [Elasticsearch integration](../../integration/elasticsearch.md) is
enabled, then some advanced search functionality is available.
[Learn how to use the Advanced Global Search.](advanced_global_search.md)
[Learn how to use the advanced search syntax.](advanced_search_syntax.md)
## Advanced Syntax Search
Use advanced queries for more targeted search results.
[Learn how to use the Advanced Syntax Search.](advanced_search_syntax.md)
[ee]: https://about.gitlab.com/gitlab-ee/
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment