API: Share projects only with groups current_user can access

Aims to address the issues here: https://gitlab.com/gitlab-org/gitlab-ce/issues/23004

* Projects can be shared with non-existent groups
* Projects can be shared with groups that the current user does not have access to read

Concerns:

The new implementation of the API endpoint allows projects to be shared with a larger range of groups than can be done via the web UI.

The form for sharing a project with a group uses the following API endpoint to index the available groups: https://gitlab.com/gitlab-org/gitlab-ce/blob/494269fc92f61098ee6bd635a0426129ce2c5456/lib/api/groups.rb#L17. The groups indexed in the web form will only be those groups that the user is currently a member of.

The new implementation allows projects to be shared with any group that the authenticated user has access to view. This widens the range of groups to those that are public and internal.

See merge request !2005
Signed-off-by: Rémy Coutable <remy@rymai.me>

Don't send Private-Token headers to Sentry

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/22537

This bumps 'raven' (the Ruby gem we use to send errors to Sentry) to
version 2.0.2. We need 2.0.0 or newer to be able to sanitize HTTP
headers.

See merge request !2004
Signed-off-by: Rémy Coutable <remy@rymai.me>

Add a new gitlab:users:clear_all_authentication_tokens task

## What are the relevant issue numbers?

Part of #22537.

See merge request !6745
Signed-off-by: Rémy Coutable <remy@rymai.me>

Update mail_room gem

## What does this MR do?

Upgrades the mail_room gem from 0.8.0 to 0.8.1.

## Why was this MR needed?

mail_room throws an exception if the watching script joins a undefined thread. Release 0.8.1 adds a check to ensure that the thread exists before calling join.

Fixes gitlab-org/omnibus-gitlab#1619.

See merge request !6714
Signed-off-by: Rémy Coutable <remy@rymai.me>

Improve issue load time performance by avoiding ORDER BY in find_by call

See merge request !6724
Signed-off-by: Rémy Coutable <remy@rymai.me>

Resolve "NameError: undefined local variable or method `request' for #<Grape::Middleware::Error:0x007fc990..."

## What does this MR do?

Switches from `request` to `env` in an API helper method as the helpers are included in contexts lacking `request`.

## Are there points in the code the reviewer needs to double check?

I couldn't build a reproducer for this.

Closes #22820

See merge request !6615
Signed-off-by: Rémy Coutable <remy@rymai.me>

Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix issues importing services via Import/Export

Prevents errors when initialising services that do not have any properties set yet - case that could happen when importing projects.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/22891

See merge request !6667
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix export project file permissions issue

Fixes security concerns of https://gitlab.com/gitlab-org/gitlab-ce/issues/22757

I have just added the permissions 0700 to the creation of any of the export paths, as @jacobvosmaer suggested in https://gitlab.com/gitlab-org/gitlab-ce/issues/22757#note_16197616

After this has fixed, it could take up to 24 hours in the worse case scenario for old archives to be completely safe - This is the time `ImportExportProjectCleanupWorker` may take to remove the folders. The temporary folders will be 0700 straight away for new installations.

See merge request !2003
Signed-off-by: Rémy Coutable <remy@rymai.me>

Prevent claiming associated model IDs via import

On the import side, we should be careful not to use any IDs as part of the JSON file that could have been manipulated.

Part of https://gitlab.com/gitlab-org/gitlab-ce/issues/20821

Things we already do (__before__ this fix):

1. Remove all primary keys
1. **Always** reassign some of the foreign keys, such as ALL project IDs and user IDs (so it would be difficult to impersonate or try to gain access to another project)
1. Ignore/reject attributes that do not exist in the model
1. If someone reassigns a foreign key `submodel_id`, and that object has another json as the submodel, the new submodel will reassign the `submodel_id` to the newly created submodel ID.

Things we should do:

1. Remove/nullify any other foreign keys that we don't reassign (checked this, and there aren't many, fortunately. In fact, I don't think much harm can be done at all - at the moment).

See merge request !1985
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix project deletion when feature visibility is set to private

Projects that are destroyed are put in the pending_delete state.
The ProjectDestroyWorker checks whether the current user has
access, but since the ProjectFeature class uses the default scope
of the Project, it will not be able to find the right project.

This was a regression in 8.12 that caused the following stack trace:

```
NoMethodError: undefined method `team' for nil:NilClass
  from app/models/project_feature.rb:62:in `get_permission'
  from app/models/project_feature.rb:34:in `feature_available?'
  from app/models/project.rb:21:in `feature_available?'
  from app/policies/project_policy.rb:170:in `disabled_features!'
  from app/policies/project_policy.rb:29:in `rules'
  from app/policies/base_policy.rb:82:in `block in abilities'
  from app/policies/base_policy.rb:113:in `collect_rules'
  from app/policies/base_policy.rb:82:in `abilities'
  from app/policies/base_policy.rb:50:in `abilities'
  from app/models/ability.rb:64:in `uncached_allowed'
  from app/models/ability.rb:58:in `allowed'
  from app/models/ability.rb:49:in `allowed?'
  from app/services/base_service.rb:11:in `can?'
  from lib/gitlab/metrics/instrumentation.rb:155:in `block in can?'
  from lib/gitlab/metrics/method_call.rb:23:in `measure'
  from lib/gitlab/metrics/instrumentation.rb:155:in `can?'
  from app/services/projects/destroy_service.rb:18:in `execute'
```

Closes #22948

See merge request !6688
Signed-off-by: Rémy Coutable <remy@rymai.me>

Restrict failed login attempts from users with 2FA enabled.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/19799.

See merge request !6668
Signed-off-by: Rémy Coutable <remy@rymai.me>

Skip wiki creation when GitHub project has wiki enabled

## What does this MR do?

When importing a repository from if the repository has wiki, we should not create the default wiki. Otherwise the GitHub importer will fail because the wiki repository already exist. This bug was introduced here https://gitlab.com/gitlab-org/gitlab-ce/commit/892dea67717c0efbd6a28f7639f34535ec0a8747

## Are there points in the code the reviewer needs to double check?

No.

## Why was this MR needed?

GitLab fails to import GitHub Wiki.

## Screenshots (if relevant)

## Does this MR meet the acceptance criteria?

- [X] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
  - [X] Added for this feature/bug
  - [ ] All builds are passing
- [X] Conform by the [merge request performance guides](http://docs.gitlab.com/ce/development/merge_request_performance_guidelines.html)
- [X] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [X] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

## What are the relevant issue numbers?

Fixes #22225

See merge request !6665
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix lint-doc error

## What does this MR do?
Removes duplicate changelog versions to fix lint-doc error (https://gitlab.com/gitlab-org/gitlab-ce/builds/4621603)

See merge request !6623
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix bug when trying to cache closed issues from external issue trackers

fixes #22446

See merge request !6619
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix race condition on LFS Token

## What does this MR do?

 Fixes a race condition that can be triggered if the token expires right after we retrieve it, but before we can set the new expiry time.

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6551#note_16190569

Thanks for catching this so quickly @jacobvosmaer-gitlab

cc @DouweM

See merge request !6592
Signed-off-by: Rémy Coutable <remy@rymai.me>

Changed compare dropdowns to dropdowns with search input

## What does this MR do?

This changes the compare dropdowns from text inputs, that when clicked, open a dropdown of branches/tags to dropdowns that have a dropdown toggle and an isolated search input.

## Are there points in the code the reviewer needs to double check?

## Why was this MR needed?

This was needed to fix the poor UX highlighted in #22221, where opening the dropdown showed an initially filtered set of results because the dropdown toggle was the dropdown filter itself. The compare page is always loaded with `master` as each branch/tag selection, so when opening the dropdown, it would only show results matching `master`.

## Screenshots (if relevant)

![2016-09-27_18.28.10](/uploads/0ea1d91cb592c6e140ed62c336e77227/2016-09-27_18.28.10.gif)

## Does this MR meet the acceptance criteria?

- [ ] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [ ] API support added
- Tests
  - [ ] Added for this feature/bug
  - [ ] All builds are passing
- [ ] Conform by the [merge request performance guides](http://docs.gitlab.com/ce/development/merge_request_performance_guidelines.html)
- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

## What are the relevant issue numbers?

Closes #22221

See merge request !6550
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix build sidebar build details padding

## What does this MR do?

Removes a `.block-first` overriding declaration that was added to fix the coverage padding and moved the padding that fixes the coverage block to a `.block.coverage` declaration.

## Are there points in the code the reviewer needs to double check?

## Why was this MR needed?

The build sidebar build details had too much padding as seen in #22529.

## Screenshots (if relevant)

![Screen_Shot_2016-09-24_at_19.38.40](/uploads/2c11a71f3022909fe8a5d1e983445667/Screen_Shot_2016-09-24_at_19.38.40.png)

![Screen_Shot_2016-09-24_at_19.39.06](/uploads/482af29f4bb7e284469466da59901087/Screen_Shot_2016-09-24_at_19.39.06.png)

## Does this MR meet the acceptance criteria?

- [ ] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [ ] API support added
- Tests
  - [ ] Added for this feature/bug
  - [ ] All builds are passing
- [ ] Conform by the [merge request performance guides](http://docs.gitlab.com/ce/development/merge_request_performance_guidelines.html)
- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

## What are the relevant issue numbers?

Closes #22529

See merge request !6506
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix tooltip text when Copy to cliboard is clicked

## What does this MR do?

Fixes #22022 to change tooltip text to "Copied!" when the "Copy to clipboard" button is clicked.

## Are there points in the code the reviewer needs to double check?

Line 29 in app/assets/javascripts/copy_to_clipboard.js

## Why was this MR needed?

There was a regression that prevents tooltip change on button click for Copy to clipboard.

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [x] API support added
- Tests
  - [x] Added for this feature/bug
  - [x] All builds are passing
- [x] Conform by the [merge request performance guides](http://docs.gitlab.com/ce/development/merge_request_performance_guidelines.html)
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

## What are the relevant issue numbers?

Closes #22022

See merge request !6294
Signed-off-by: Rémy Coutable <remy@rymai.me>

Bump Gitlab Shell to support low IO priority for storage moves

## What does this MR do?

Bump Gitlab Shell to support low IO priority for storage moves

## Are there points in the code the reviewer needs to double check?

No

## Why was this MR needed?

We need this shell enhancement to continue to move projects in production 

## Does this MR meet the acceptance criteria?

- [ ] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [ ] API support added
- Tests
  - [ ] Added for this feature/bug
  - [ ] All builds are passing
- [ ] Conform by the [merge request performance guides](http://docs.gitlab.com/ce/development/merge_request_performance_guidelines.html)
- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

## What are the relevant issue numbers?

https://gitlab.com/gitlab-com/infrastructure/issues/443

See merge request !6525

Pass the full project path for resolve buttons

## What does this MR do?

The full project path is passed from the HTML into the JS rather than the resolve button JS generating the URL based on the the namespaces.

@smcgivern @stanhu Do we have anyway of adding tests for installs with relative URLs?

## What are the relevant issue numbers?

Closes #21704

See merge request !6129

Fixed awards dropdown search text from repeating

## What does this MR do?

Thanks @zeiv for this!

>This fixes #20463 by adding the .emoji-search class to the "Search results" h5 element.  AwardsHandler.prototype.setupSearch() was trying to remove the element before creating a new one, but wasn't because the it was attempting to select it with the .emoji-search class, which wasn't being added.

## Are there points in the code the reviewer needs to double check?

>I don't believe so.

## Why was this MR needed?

>See #20463

_(and #22525)_

## Screenshots (if relevant)

![2016-09-23_15.56.04](/uploads/b421400a08b3d9fa32c577e7d8952504/2016-09-23_15.56.04.gif)

## Does this MR meet the acceptance criteria?

- [ ] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [ ] API support added
- Tests
  - [ ] Added for this feature/bug
  - [ ] All builds are passing
- [ ] Conform by the [merge request performance guides](http://docs.gitlab.com/ce/development/merge_request_performance_guidelines.html)
- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

## What are the relevant issue numbers?


Closes #22525

See merge request !6498

Do not regenerate the `lfs_token` every time `git-lfs-authenticate` is called

## What does this MR do?

 Do not regenerate the `lfs_token` every time `git-lfs-authenticate` is called, instead return the saved token if one is present.

This was causing a lot of 401s, leading to 403s, as state in #22527

As it turns out, when pushing a lot of LFS objects, the LFS client was calling `git-lfs-authenticate` in the middle of the request again. This caused the `lfs_token` to be regenerated. The problem lies in that the LFS client was not aware of this change, and was still using the old token. This caused all subsequent requests to fail with a 401 error.

Since HTTP Auth is protected by Rack Attack, this 401s where immediately flagged and resulted in the IP of the user being banned. 

With this change, GitLab returns the value stored in Redis, if one is present, thus if the LFS client calls `git-lfs-authenticate` again during the request, the auth header will remain unchanged, allowing all subsequent requests to continue without issues.

## What are the relevant issue numbers?

Fixes #22527

cc @SeanPackham @jacobvosmaer-gitlab

See merge request !6551

Escape HTML nodes in builds commands in ci linter

This MR removes call to `simple_format` that behaves like `String#html_safe`, thus it passes unescaped HTML tags to the view.

Closes #22541

See merge request !2001

API: disable rails session auth for non-GET/HEAD requests

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/22435

See merge request !1999
Conflicts:
	app/assets/javascripts/labels_select.js

Set a restrictive CORS policy for the API

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/22450

See merge request !1998

Enforce the fork_project permission in Projects::CreateService

Projects::ForkService delegates to this service almost entirely, but needed one small change so it would propagate create errors correctly.

CreateService#execute needs significant refactoring; it is now right at the complexity limit set by Rubocop. I avoided doing so in this commit to keep the diff as small as possible.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/18028

See merge request !1996

Fix duplicate master entries in the mr versions dropdown

## What does this MR do?

Fixes bug when "master" was duplicated per each mr version in the dropdown

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/22519, Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/22715

See merge request !6567

Changed to v-cloak attr to hash rocket and string 'true'

## What does this MR do?

Changed to v-cloak attr to hash rocket and string 'true'

## Are there points in the code the reviewer needs to double check?

## Why was this MR needed?

Maybe it's killing `master`? 😕 

## Screenshots (if relevant)

## Does this MR meet the acceptance criteria?

- [ ] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [ ] API support added
- Tests
  - [ ] Added for this feature/bug
  - [ ] All builds are passing
- [ ] Conform by the [merge request performance guides](http://docs.gitlab.com/ce/development/merge_request_performance_guidelines.html)
- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

## What are the relevant issue numbers?

See merge request !6553

Send ajax request for label update only if they are changed

## What does this MR do?

At the moment we send requests to update the labels on issue even if nothing has been changed. This fixes that and only sends requests when something has actually been changed.

The code gets the initial selected values based on the inputs & then compares that to the selected values when submitting. If they are equal, then no request is sent.

## What are the relevant issue numbers?

Closes #19472 

See merge request !5071

Fixes issue with rails reserved keyword type exporting/importing services.

The attribute `type`in services was being ignored by Import/Export. Added `type` as a method call in the export, as `type` gets ignored invoking `to_json`, manually adding this as a method in `import_export.yml` solves the problem.

On a different note, I found assigning a title directly to `CustomIssueTrackerService` didn't play very well with `prop_accessor`:

```ruby
> CustomIssueTrackerService.new(title: 'asdf')
NoMethodError: undefined method `[]=' for nil:NilClass
> CustomIssueTrackerService.new(title: nil)
NoMethodError: undefined method `[]=' for nil:NilClass
```

This was also causing the Import/Export to failed... So I added a custom setter that fixed the problem.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/22461

See merge request !6499
Conflicts:
	spec/lib/gitlab/import_export/project_tree_restorer_spec.rb

Fix snippets pagination

## What are the relevant issue numbers?

Closes #22500

See merge request !6500