gitlab: Optimize raw blob downloading
In slapos we do a lot of automated software rebuild constantly, and thus there is constant flow of requests to get raw blobs from git service, e.g. like this https://lab.nexedi.com/nexedi/slapos/raw/master/software/wendelin/software.cfg A lot of requests comes to slapos.git repository and currently gitlab, out of the box, cannot keep up with that load. I've prepared patches to offload raw blobs download requests handling from unicorn (ruby) to gitlab-workhorse (go), and that resulted in ~ 17x speedup - e.g. previously our std shuttle can handle ~ 70 raw-blob requests/s and with my changes it is now ~ 1200 requests/s. The patches were sent upstream https://gitlab.com/gitlab-org/gitlab-workhorse/merge_requests/17 and we discussed with GitLab people and made a plan how to proceed incrementally. It will probably take some time for gitlab team to fully accept the approach though. For now we can use our gitlab-workhorse fork. The patches itself are: gitlab-workhorse@1b274d0d gitlab-workhorse@2beb8c95 /cc @kazuhiko, @jerome, @jm
Showing
Please register or sign in to comment