Update doorkeeper to 4.2.0

Changelog: https://git.io/v6PnV

See merge request !5881
(cherry picked from commit c5aa31c83145366d88ce6d8d91e68467cf5baed4)

[ci skip]

Upgrade Rails to 4.2.7.1 for security fixes.

Upgrades Rails from 4.2.7 to 4.2.7.1 for security fixes.

For more information: http://weblog.rubyonrails.org/2016/8/11/Rails-5-0-0-1-4-2-7-2-and-3-2-22-3-have-been-released/

This should be backported to all currently-supported releases.

See merge request !5781

Ensure logged-out users can't see private refs

https://gitlab.com/gitlab-org/gitlab-ce/issues/18033

I'm still not sure what to do about the CHANGELOG on security issues - should I add to a patch release? This issue was assigned to 8.10.

See merge request !1974
(cherry picked from commit 3a6ebb1f)

Fix privilege escalation issue with OAuth external users

Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/19312

This MR fixes a privilege escalation issue, where manually set external users would be reverted back to internal users if they logged in via OAuth and that provider was not in the `external_providers` list.

/cc @douwe

See merge request !1975
(cherry picked from commit 5e6342b7)

Fix visibility of snippets when searching

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/18997

See merge request !1972
(cherry picked from commit 8a197c15)

Update omniauth-saml to 1.6.0 to address a security vulnerability in ruby-saml

Updates `omniauth-saml` to bring in the new `ruby-saml` dependency that addresses [CVE-2016-5697](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-5697)

Fixes #19206

See merge request !4951
(cherry picked from commit c3a8b252)

Only show notes through JSON on confidential issues that the user has access to

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/18535

See merge request !1970

Forbid scripting for wiki files

Wiki files (not pages - files in the repo) are just sent to the browser
with whatever content-type the mime_types gem assigns to them based on
their extension. As this is from the same domain as the GitLab
application, this is an XSS vulnerability.

Set a CSP forbidding all sources for scripting, CSS, XHR, etc. on these
files.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/17298.

See merge request !1969

Remove 'unscoped' from project builds selection

This is a fix for this security bug: https://gitlab.com/gitlab-org/gitlab-ce/issues/18188

/cc @kamil @grzegorz @stanhu

See merge request !1968

Fix UTF-8 handling in incremental trace update API

## What does this MR do?

This MR fixes invalid UTF-8 handling in incremental trace update API (used by GitLab Runner).

## Why was this MR needed?

Current version is using `.length` method to determine current trace size where Runner is using the trace size in bytes. Also this byte size is used in headers and file operations to agree the trace part to send. This is a problem when build trace contains any multi-byte UTF-8 characters. This MR is fixing this situation so all parts are using the same size in bytes.

### Runner -> API communication before fix:
```
Checking for builds... received                     runner=_token_
gitlab-ci-multi-runner 1.3.0~beta.26.gcfd63b9 (cfd63b9)  build=25 runner=_token_
Using Docker executor with image debian:jessie ...  build=25 runner=_token_
Pulling docker image debian:jessie ...              build=25 runner=_token_
25 Submitting build to coordinator... ok            runner=_token_
25 Appending trace to coordinator... ok             RemoteRange=0-158 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=0-158 runner=_token_
25 Appending trace to coordinator... ok             RemoteRange=0-491 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=158-505 runner=_token_
WARNING: 25 Appending trace to coordinator... range missmatch  RemoteRange=0-491 RemoteState= ResponseMessage=416 Requested Range Not Satisfiable ResponseStatusCode=416 SentRange=505-584 runner=_token_
WARNING: 25 Resending trace patch due to range missmatch  runner=_token_
25 Appending trace to coordinator... ok             RemoteRange=0-556 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=491-584 runner=_token_
WARNING: 25 Appending trace to coordinator... range missmatch  RemoteRange=0-556 RemoteState= ResponseMessage=416 Requested Range Not Satisfiable ResponseStatusCode=416 SentRange=584-663 runner=_token_
WARNING: 25 Resending trace patch due to range missmatch  runner=_token_
25 Appending trace to coordinator... ok             RemoteRange=0-621 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=556-663 runner=_token_
Build succeeded                                     build=25 runner=_token_
WARNING: 25 Appending trace to coordinator... range missmatch  RemoteRange=0-621 RemoteState= ResponseMessage=416 Requested Range Not Satisfiable ResponseStatusCode=416 SentRange=663-797 runner=_token_
WARNING: 25 Resending trace patch due to range missmatch  runner=_token_
25 Appending trace to coordinator... ok             RemoteRange=0-741 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=621-797 runner=_token_
25 Submitting build to coordinator... ok            runner=_token_
```

### Runner -> API communication after fix:
```
Checking for builds... received                     runner=_token_
gitlab-ci-multi-runner 1.3.0~beta.26.gcfd63b9 (cfd63b9)  build=26 runner=_token_
Using Docker executor with image debian:jessie ...  build=26 runner=_token_
Pulling docker image debian:jessie ...              build=26 runner=_token_
26 Submitting build to coordinator... ok            runner=_token_
26 Appending trace to coordinator... ok             RemoteRange=0-158 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=0-158 runner=_token_
26 Appending trace to coordinator... ok             RemoteRange=0-505 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=158-505 runner=_token_
26 Appending trace to coordinator... ok             RemoteRange=0-584 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=505-584 runner=_token_
26 Appending trace to coordinator... ok             RemoteRange=0-663 RemoteState=running ResponseMessage=202 Accepted ResponseStatusCode=202 SentRange=584-663 runner=_token_
Build succeeded                                     build=26 runner=_token_
26 Submitting build to coordinator... ok            runner=_token_
```

See merge request !4541

Check if GitHub rate limite API was reached before update Webhooks

## What does this MR do?

Checks if the job needs to sleep, and wait for the rate limit to be reseted before update each Webhook.

## Are there points in the code the reviewer needs to double check?

No.

## Why was this MR needed?

The import process can fail if the API rate limit was reached during the import process.

## What are the relevant issue numbers?

https://gitlab.com/gitlab-org/gitlab-ce/issues/17498

## Screenshots (if relevant)

Not relevant.

See merge request !4509

Adjust the SAML control flow to allow LDAP identities to be added to an existing SAML user.

It correctly lets an existing SAML user to add their LDAP identity automatically at login.

A customer had issues with the `auto_link_ldap_user` feature. The flow was not working if there was an account with a SAML identity, but no LDAP identity. GitLab would pick up the correct LDAP person, but due to the order of the flow, that LDAP person was never associated with the user.

Fixes #17346

/cc @dblessing @balameb @stanhu

See merge request !4498

Ensure we don't show TODOS for projects pending delete

Joins the todos on the projects table in order to run the default scope. Also includes a where clause because the default scope is being removed soon.

An alternative approach, more like the Issues page, would be to filter down the list by passing user.authorized_projects into the where clause.

Or we could just be more defensive in the view when iterating.

Todos page throws 500 error for users with todos in a project pending deletion.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/17813

cc\ @stanhu

See merge request !4300

[ci skip]

Fix 2FA-based login for LDAP users

The OTP input form is shared by both LDAP and standard logins, but when
coming from an LDAP-based form, the form parameters aren't nested in a
Hash based on the `resource_name` value.

Now we check for a nested `remember_me` parameter and use that if it
exists, or fall back to the non-nested parameters if it doesn't.

Somewhat confusingly, the OTP input form _does_ nest parameters under
the `resource_name`, regardless of what type of login we're coming from,
so that allows everything else to work as normal.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/18185

See merge request !4493

Fixes missing number on generated ordered list

Closes #18102

See merge request !4437

Fix serious performance bug with rendering Markdown with InlineDiffFilter

Nokogiri's `node.replace` was being unnecessarily called for every text node in
the document due to a comparison bug. The code previously was comparing the
HTML representation of the full document against the text node, which would
always fail. Fix the comparison to just compare the modified text.

Closes #18011

See merge request !4392

Confidential notes data leak

Fixes part of https://gitlab.com/gitlab-org/gitlab-ee/issues/575

See merge request !1967

Fix wiki project clone address error

_Note: Originally opened at !4407 by @chujinjin._

---

fix wiki project clone address error in Wiki Git Access View, show as below:

![image](/uploads/5e3bf6d1418c42862a885319c31bc3cf/image.png)

Fixes #17643.

See merge request !4429

Use downcased path to container repository as this is expected path by Docker

Docker Engine requires path to be lowercase. This makes all container registry paths to be show and used downcased instead of mixed case.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/17959

See merge request !4420

Use project that belongs to pipeline in view

This MR makes project in pipelines view match the one that pipeline has been created for.

Closes #17943

See merge request !4376

Pass the "Remember me" value to the 2FA token form

Prior, if a user had 2FA enabled and checked the "Remember me" field,
the setting was ignored because the OTP input was on a new form and the
value was never passed.

Closes #18000

See merge request !4369

Add Application Setting to configure Container Registry token expire delay (default 5min)

This adds an option to configure Container Registry token expire delay. The default is set to 5mins (something that is also used by Docker Hub).

What is left:
* [x] Write test to check the expire_delay

Fixes: https://gitlab.com/gitlab-org/gitlab-ce/issues/17890

@stanhu I think that this should land in patch release of 8.8.

See merge request !4364

Merge branch 'make-container-registry-authentication-service-compatible-with-older-docker' into 'master'

Make authentication service for Container Registry to be compatible with < Docker 1.11

This removes the usage of `offline_token` which is only present when using `Docker 1.11.x` instead we relay on `scope`. This should make it compatible with any client starting from 1.6 (I did test only 1.8 and up).

Right now we return 403 if unauthorized user doesn't have access to anything. In all other cases we return token, but with empty `access`, which simply disallow requested action.

See merge request !4363