Commit 0d2c0715 authored by Kirill Smelkov's avatar Kirill Smelkov

Teach gitlab-workhorse to serve requests to get raw blobs

Currently GitLab serves requests to get raw blobs via Ruby-on-Rails code and
Unicorn. Because RoR/Unicorn is relatively heavyweight, in environment where
there are a lot of simultaneous requests to get raw blobs, this works very slow
and server overload are frequent.

On the other hand, to get raw blob content, we do not need anything from RoR
framework - we only need to have access to project git repository on filesystem,
and knowing whether request for getting data from there should be granted or
not. That means it is possible to adjust Nginx frontend to route '.../raw/....'
request to more lightweight and performant program which does this particular
task and that will be a net win.

As gitlab-workhorse is written in Go, and Go has good concurrency/parallelism
support and is generally much faster than Ruby, adding raw blob serving task to
it makes sense.

In this patch: we add infrastructure to process GET request for '/raw/...':

- extract project / ref and path from URL
- query auth backend for whether download access should be granted or not
- emit blob content via spawning external `git cat-file`

I've tried to mimic the output to be as close as the one emitted by RoR code,
with the idea that for users the change should be transparent.

As in this patch we do auth backend query for every request to get a blob, RoR
code is still loaded very much, but even this way it is a win:

    TODO timings for wrk

In the next patch we'll cache requests to auth backend and that will improve
performance dramatically.
parent d1b215d5
Pipeline #152 failed with stage