Commit a913c2e4 authored by Kirill Smelkov's avatar Kirill Smelkov

gitlab: Optimize raw blob downloading

In slapos we do a lot of automated software rebuild constantly, and thus
there is constant flow of requests to get raw blobs from git service,
e.g. like this

    https://lab.nexedi.com/nexedi/slapos/raw/master/software/wendelin/software.cfg

A lot of requests comes to slapos.git repository and currently gitlab,
out of the box, cannot keep up with that load.

I've prepared patches to offload raw blobs download requests handling
from unicorn (ruby) to gitlab-workhorse (go), and that resulted in ~ 17x
speedup - e.g. previously our std shuttle can handle ~ 70 raw-blob
requests/s and with my changes it is now ~ 1200 requests/s.

The patches were sent upstream

    https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/17

and we discussed with GitLab people and made a plan how to proceed
incrementally. It will probably take some time for gitlab team to fully
accept the approach though.

For now we can use our gitlab-workhorse fork. The patches itself are:

    kirr/gitlab-workhorse@1b274d0d
    kirr/gitlab-workhorse@2beb8c95

/cc @kazuhiko, @jerome, @jm
parent 74d4ea62
......@@ -109,9 +109,11 @@ location = ${buildout:parts-directory}/gitlab-shell
[gitlab-workhorse-repository]
<= git-repository
repository = https://gitlab.com/gitlab-org/gitlab-workhorse.git
# 0.4.X + NXD patches
revision = 0.4.1-10-g0d0bd209eea7441a3b4202c2e00a989cf340ee86
#repository = https://gitlab.com/gitlab-org/gitlab-workhorse.git
repository = https://lab.nexedi.com/kirr/gitlab-workhorse.git
# 0.4.X + NXD patches for blob download speedup
# (https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/17)
revision = 0.4.1-23-g2beb8c9539433f072e3db540f91f75894ca6b1b0
location = ${buildout:parts-directory}/gitlab-workhorse
......
......@@ -206,6 +206,13 @@ server {
return 418;
}
# access to raw blobs -> @gitlab-workhorse
location ~ ^/[\w\.-]+/[\w\.-]+/raw/ {
client_max_body_size 0;
error_page 418 = @gitlab-workhorse;
return 418;
}
location @gitlab-workhorse {
client_max_body_size 0;
## If you use HTTPS make sure you disable gzip compression
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment