• Kirill Smelkov's avatar
    blob: Cache auth backend reply for 30s · 2beb8c95
    Kirill Smelkov authored
    In previous patch we added code to serve blob content via running `git cat-file
    ...` directly, but for every such request a request to slow RoR-based auth
    backend is made, which is bad for performance.
    
    Let's cache auth backend reply for small period of time, e.g. 30 seconds, which
    will change the situation dramatically:
    
    If we have a lot of requests to the same repository, we query auth backend only
    for every Nth request and with e.g. 100 raw blob request/s N=3000 which means
    that previous load to RoR code essentially goes away.
    
    On the other hand as we query auth backend only once in a while and refresh the
    cache we will not miss potential changes in project settings. I mean potential
    e.g. 25 seconds delay for a project to become public, or vise versa to become
    private does no real harm.
    
    The cache is done with the idea to allow the read side codepath to execute in
    parallel and to be not blocked by eventual cache updates.
    
    Overall this improves performance a lot:
    
      (on a 8-CPU i7-3770S with 16GB of RAM)
    
      # request goes to gitlab-workhorse with the following added to nginx conf
      # location ~ ^/[\w\.-]+/[\w\.-]+/raw/ {
      #   error_page 418 = @gitlab-workhorse;
      #   return 418;
      # }
      # but without auth caching
      $ ./wrk -c40 -d10 -t1 --latency https://[2001:67c:1254:e:8b::c776]:7777/root/slapos/raw/master/software/wendelin/software.cfg
      Running 10s test @ https://[2001:67c:1254:e:8b::c776]:7777/root/slapos/raw/master/software/wendelin/software.cfg
        1 threads and 40 connections
        Thread Stats   Avg      Stdev     Max   +/- Stdev
          Latency   549.37ms  220.53ms   1.69s    84.74%
          Req/Sec    71.01     25.49   160.00     70.71%
        Latency Distribution
           50%  514.66ms
           75%  584.32ms
           90%  767.46ms
           99%    1.37s
        709 requests in 10.01s, 1.26MB read
      Requests/sec:     70.83
      Transfer/sec:    128.79KB
    
      # request goes to gitlab-workhorse with auth caching (this patch)
      $ ./wrk -c40 -d10 -t1 --latency https://[2001:67c:1254:e:8b::c776]:7777/root/slapos/raw/master/software/wendelin/software.cfg
      Running 10s test @ https://[2001:67c:1254:e:8b::c776]:7777/root/slapos/raw/master/software/wendelin/software.cfg
        1 threads and 40 connections
        Thread Stats   Avg      Stdev     Max   +/- Stdev
          Latency    35.18ms   20.78ms 291.34ms   72.79%
          Req/Sec     1.18k   135.34     1.34k    88.00%
        Latency Distribution
           50%   33.96ms
           75%   44.35ms
           90%   58.70ms
           99%  104.76ms
        11704 requests in 10.01s, 20.78MB read
      Requests/sec:   1169.75
      Transfer/sec:      2.08MB
    
    i.e. it is ~ 17x improvement.
    2beb8c95
upstream.go 5.57 KB