-
Kirill Smelkov authored
Currently GitLab serves requests to get raw blobs via Ruby-on-Rails code and Unicorn. Because RoR/Unicorn is relatively heavyweight, in environment where there are a lot of simultaneous requests to get raw blobs, this works very slow and server is constantly overloaded. On the other hand, to get raw blob content, we do not need anything from RoR framework - we only need to have access to project git repository on filesystem, and knowing whether access for getting data from there should be granted or not. That means it is possible to handle '.../raw/....' request directly in more lightweight and performant gitlab-workhorse. As gitlab-workhorse is written in Go, and Go has good concurrency/parallelism support and is generally much faster than Ruby, moving raw blob serving task to it makes sense and should be a net win. In this patch: we add infrastructure to process GET request for '/raw/...': - extract project / ref and path from URL - query auth backend for whether download access should be granted or not - emit blob content via spawning external `git cat-file` I've tried to mimic the output to be as close as the one emitted by RoR code, with the idea that for users the change should be transparent. As in this patch we do auth backend query for every request to get a blob, RoR code is still loaded very much, so essentially there is no speedup yet: (on a 8-CPU i7-3770S with 16GB of RAM, 2001:67c:1254:e:89::fa34 is on localhost) # without patch: request eventually goes to unicorn (9 unicorn workers) $ ./wrk -c40 -d10 -t1 --latency http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg Running 10s test @ http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg 1 threads and 40 connections Thread Stats Avg Stdev Max +/- Stdev Latency 609.34ms 156.92ms 1.18s 79.60% Req/Sec 64.22 19.90 120.00 67.00% Latency Distribution 50% 596.50ms 75% 678.23ms 90% 805.72ms 99% 1.04s 642 requests in 10.01s, 1.24MB read Requests/sec: 64.16 Transfer/sec: 127.00KB # with this patch: request handled by gitlab-workhorse $ ./wrk -c40 -d10 -t1 --latency http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg Running 10s test @ http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg 1 threads and 40 connections Thread Stats Avg Stdev Max +/- Stdev Latency 622.99ms 200.08ms 1.40s 77.03% Req/Sec 62.65 22.37 120.00 55.00% Latency Distribution 50% 589.51ms 75% 726.88ms 90% 896.09ms 99% 1.18s 626 requests in 10.01s, 1.11MB read Requests/sec: 62.55 Transfer/sec: 113.73KB In the next patch we'll cache requests to auth backend and that will improve performance dramatically.
d763bd17