Backport git access spec changes from EE

https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/645#note_16391185

See merge request !6961