Allow to sort by project size on admin projects page

Fixes #20317

See merge request !5797
(cherry picked from commit dd2088b0)

Set permissions to admin for importing a project via Import/Export

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/20802

In order to import a project, it is now required to be an admin

Moved from https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5766

See merge request !1983
(cherry picked from commit 966b3038)

Fix attribute inclusion import/export config ignored in some cases

In the `import_export.yml` file we define the inclusion of some of the attributes. For some reason, this isn't working in certain cases - very unfortunate this includes `user`. This has been introduced in 8.10.3.

Related https://gitlab.com/gitlab-org/gitlab-ce/issues/20802

See merge request !1982

[ci skip]

Upgrade Rails to 4.2.7.1 for security fixes.

Upgrades Rails from 4.2.7 to 4.2.7.1 for security fixes.

For more information: http://weblog.rubyonrails.org/2016/8/11/Rails-5-0-0-1-4-2-7-2-and-3-2-22-3-have-been-released/

This should be backported to all currently-supported releases.

See merge request !5781

Used cached value of project count to reduce DB load

Following !5746, backport a change from https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/637/diffs#diff-2.

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
  - [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5754
Signed-off-by: Rémy Coutable <remy@rymai.me>

Add a method in Project to return a cached value of total count of projects

This is in preparation to address the DB load caused by the counting
in gitlab-com/infrastructure#303.

See merge request !5746

Revert "Defend against 'Host' header injection"

We reverted this in omnibus but forgot to do it in the source NGINX template.

See merge request !5706
Signed-off-by: Rémy Coutable <remy@rymai.me>

Add a data migration to fix some missing timestamps in the members table (again)

Repeats an earlier migration to fix historic bad data in the members table (missing created_at and updated_at fields)

I'm expecting the WHERE clauses to be fast enough, and to return few enough rows, that the migration doesn't need to use batches, but I'm not too familiar with the size of these tables in the wild, so perhaps that's a poor assumption.

8.10 introduced a dependency on the  `members.created_at` field in the project and namespace member view. If bad data is present, viewing the list of members now results in an NoMethodError and a 500 response from GitLab. Although the previous migration should have fixed all bad rows, we have evidence that it didn't in at least one case, despite the migration claiming to have run in the past.

Closes #20568

See merge request !5670
Signed-off-by: Rémy Coutable <remy@rymai.me>

Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix Import/Export error checking versions

Fixes small bug preventing the correct error message about Import/Export version being displayed.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/20536

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
  - [x] Added for this feature/bug
  - [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5638
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix Import/Export not working in HA mode

Use a shared path instead of `Tempfile` default `/tmp` so the import file is accessible by any GitLab instance.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/20506

See merge request !5618
Signed-off-by: Rémy Coutable <remy@rymai.me>

Signed-off-by: Rémy Coutable <remy@rymai.me>

Don’t close issues on original project

Closes #20527 -> https://gitlab.com/gitlab-org/gitlab-ce/issues/20527

See merge request !1981
Signed-off-by: Rémy Coutable <remy@rymai.me>

Dropdown enter key fix

## What does this MR do?

Fixes issue with dropdowns enter key not working correctly

See merge request !5544

Signed-off-by: Rémy Coutable <remy@rymai.me>

Trim extra displayed carriage returns in diffs and files with CRLFs

## What does this MR do?

This MR strips out extra carriage returns at the end of each tokenized string.

## Are there points in the code the reviewer needs to double check?

Does this remove more than we need?

## Why was this MR needed?

Windows users would see extra lines in their files. Newline characters are already added with every line in the HTML formatter.

## What are the relevant issue numbers?

#20440, gitlab-com/support-forum#892, #20468

## Does this MR meet the acceptance criteria?

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
  - [x] Added for this feature/bug
  - [ ] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5588
Signed-off-by: Rémy Coutable <remy@rymai.me>

Ignore invalid IPs in X-Forwarded-For when trusted proxies are configured.

## What does this MR do?

Catches IPAddr::InvalidAddressError exceptions in `trusted_proxy?` when a) a trusted proxy is set up in the gitlab config and b) an invalid IP address is passed to the method (e.g. one with a port attached). When caught, returns `false` from the method. Prevents a 500 error in this situation.

## What are the relevant issue numbers?

Closes gitlab-org/gitlab-ce#20466.

## Does this MR meet the acceptance criteria?

- [X] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [N/A] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [N/A] API support added
- Tests
  - [X] Added for this feature/bug
  - [X] All builds are passing
- [X] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [X] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [X] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5584
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix attr reader to force the intended values for source and target shas

## What does this MR do?

When importing a pull request from GitHub, the old and new branches may no longer actually exist by those names, but we need to recreate the merge
request diff with the right source and target shas.  We use these `target_branch_sha` and `source_branch_sha` attributes to force  these to the intended values. But the reader methods were always looking up to the target/source branch head instead of check if these values was  previously set, this MR applies a memoization pattern to both of them.

## Are there points in the code the reviewer needs to double check?

This [commit](https://gitlab.com/gitlab-org/gitlab-ce/commit/6ce25e7b4caa9e94de74378729178c7060d640b2) introduced this bug in the importer.

## What are the relevant issue numbers?

gitlab-org/gitlab-ce#20385

## Does this MR meet the acceptance criteria?

- [X] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [X] ~~[Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)~~
- [X] ~~API support added~~
- Tests
  - [X] Added for this feature/bug
  - [ ] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

/cc @akitaonrails @DouweM

See merge request !5573
Signed-off-by: Rémy Coutable <remy@rymai.me>

Properly abort a merge when merge conflicts occur

If somehow a user attempted to accept a merge request that had conflicts (e.g. the "Accept Merge Request" button or the MR itself was not updated), `MergeService` did not properly detect that a conflict
occurred. It would assume that the MR went through without any issues and close the MR as though everything was fine. This could cause data loss if the source branch were removed.

Closes #20425

See merge request !5569
Signed-off-by: Rémy Coutable <remy@rymai.me>

fix repo hooks missing on import

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/19556

Repo hooks are missing from imported projects

- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
  - [x] Added for this feature/bug
  - [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5549
Signed-off-by: Rémy Coutable <remy@rymai.me>

Add a log message when a project is scheduled for destruction for debugging

We have a lot of projects that are in `pending_delete` state. It's not clear whether they were ever scheduled for destruction, or whether Sidekiq just dropped the job due to `MemoryKiller` or some other reason.

Also this will provide a record of which user destroys a project.

#20365

See merge request !5540
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix timing problems running imports on production

Fixes https://gitlab.com/gitlab-com/infrastructure/issues/151

I've found out that in staging, the imported file is not copied fully by the time sidekiq runs the job, this should hopefully fix it. (Tested against staging).

See merge request !5523
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix Importing labels and milestones associations - Import/Export

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/19510 and https://gitlab.com/gitlab-org/gitlab-ce/issues/19447
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
  - [x] Added for this feature/bug
  - [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)

See merge request !5426
Signed-off-by: Rémy Coutable <remy@rymai.me>

[ci skip]
Signed-off-by: Rémy Coutable <remy@rymai.me>

[ci skip]
Signed-off-by: Rémy Coutable <remy@rymai.me>

Signed-off-by: Rémy Coutable <remy@rymai.me>

Update gitlab-shell version to 3.2.1 in the 8.9->8.10 update guide

See merge request !5516
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix missing schema update for 20160722221922

20160722221922 was merged in 8.10.1 but the schema.rb was not updated.

See merge request !5512
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix a bug where forking a project from a repository storage to another would fail

See merge request !5509
Signed-off-by: Rémy Coutable <remy@rymai.me>

Show release notes in tag list

See merge request !5503
Signed-off-by: Rémy Coutable <remy@rymai.me>

Fix expand all diffs button in compare view

## What does this MR do?

Fix the expand all diffs button in compare view.

## Are there points in the code the reviewer needs to double check?

Don't think so.

## Why was this MR needed?

We can't reuse the existing value of the format parameter, because on
the merge request page that's 'json', so the link would go to a JSON
file. We can't set it to HTML, because that adds '.html' to the URL, which
breaks the compare view (and looks bad). Setting it to `nil` uses the
default format, which in all of these cases is HTML anyway.

## What are the relevant issue numbers?

Closes #20241.

See merge request !5500
Signed-off-by: Rémy Coutable <remy@rymai.me>

Rescue Rugged::OSError (lock exists) when creating references.

Rescue an exception on concurrent scenarios. I was able to create a spec for this, because if you create the lock file before trying to create the reference the reference is created without a problem and the lock file is removed. So IMHO there is a race condition where more than one process is trying to create the same reference at the same time raising the exception, so I just added the patch without specs.

```ruby
    it "attempting to call keep_around when exists a lock does not fail" do
      ref = repository.send(:keep_around_ref_name, sample_commit.id)
      path = File.join(repository.path, ref)
      lock_path = "#{path}.lock"

      FileUtils.mkdir_p(path)
      File.open(lock_path, 'w') { |f| f.write('') }

      begin
        # No exception is raised because the lock file is removed at some point in the process
        repository.keep_around(sample_commit.id)
      ensure
        File.delete(path)
        File.delete(lock_path)
      end
    end
```

See merge request !5497
Signed-off-by: Rémy Coutable <remy@rymai.me>