Commit fb57599d authored by Kirill Smelkov's avatar Kirill Smelkov

blob/auth: Cache auth backend reply for 30s

In previous patch we added code to serve blob content via running `git cat-file
...` directly, but for every such request a request to slow RoR-based auth
backend is made, which is bad for performance.

Let's cache auth backend reply for small period of time, e.g. 30 seconds, which
will change the situation dramatically:

If we have a lot of requests to the same repository, we query auth backend only
for every Nth request and with e.g. 100 raw blob request/s N=3000 which means
that previous load to RoR code essentially goes away.

On the other hand as we query auth backend only once in a while and refresh the
cache, we will not miss potential changes in project settings. I mean potential
e.g. 25 seconds delay for a project to become public, or vise versa to become
private does no real harm.

The cache is done with the idea to allow the read side codepath to execute in
parallel and to be not blocked by eventual cache updates.

Overall this improves performance a lot:

  (on a 8-CPU i7-3770S with 16GB of RAM, 2001:67c:1254:e:89::fa34 is on localhost)

  # request is handled by gitlab-workhorse, but without auth caching
  $ ./wrk -c40 -d10 -t1 --latency http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg
  Running 10s test @ http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg
    1 threads and 40 connections
    Thread Stats   Avg      Stdev     Max   +/- Stdev
      Latency   622.99ms  200.08ms   1.40s    77.03%
      Req/Sec    62.65     22.37   120.00     55.00%
    Latency Distribution
       50%  589.51ms
       75%  726.88ms
       90%  896.09ms
       99%    1.18s
    626 requests in 10.01s, 1.11MB read
  Requests/sec:     62.55
  Transfer/sec:    113.73KB

  # request goes to gitlab-workhorse with auth caching (this patch)
  $ ./wrk -c40 -d10 -t1 --latency http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg
  Running 10s test @ http://[2001:67c:1254:e:89::fa34]:7777/root/slapos/raw/master/software/wendelin/software.cfg
    1 threads and 40 connections
    Thread Stats   Avg      Stdev     Max   +/- Stdev
      Latency    36.62ms   25.39ms 351.14ms   72.02%
      Req/Sec     1.16k    93.73     1.36k    77.00%
    Latency Distribution
       50%   36.30ms
       75%   47.02ms
       90%   66.36ms
       99%  122.46ms
    11580 requests in 10.01s, 20.56MB read
  Requests/sec:   1156.85
  Transfer/sec:      2.05MB

i.e. it is ~ 17x improvement.
parent 277d5067
Pipeline #280 failed with stage