Skip to content

G
gitlab-ce
Switch branch/tag
  1. 12 Apr, 2021 2 commits
    • Patrick Steinhardt's avatar
      Use object quarantine directory to enumerate new LFS pointers · e07a1c70
      Patrick Steinhardt authored 
      When accepting pushes, we will check whether pushes contain any new LFS
pointers and, if so, verify that we've got the corresponding LFS object
for each of the poniters in order to ensure consistency. Determining new
LFS pointers is expensive though: we need to perform a complete graph
walk in order to determine which blobs are new and which aren't. The
integrity check's runtime thus scales with repository size and is
frequently seen to take multiple seconds or even time out after 30
seconds. Results are that the push seems to be hanging for quite some
time doing nothing, or that the push is refused altogether in the case
of a timeout.

We can do better though: instead of doing a graph walk, we can just
inspect all pushed objects directly by enumerating all pushed objects.
This is quite trivial to do: when git-receive-pack(1) receives a push,
all pushed objects will be put into a quarantine directory which is then
made available to git hooks via the GIT_OBJECT_DIRECTORY variable, where
the real repository is stored in GIT_ALTERNATIVE_OBJECT_DIRECTORIES.
Instead of doing the graph walk, we can just use git-cat-file(1) with
the `--batch-all-objects` flag and the alternate object directories
unset. The result is a direct enumeration of all pushed objects, which
scales linearly with push size and not with repository size. Doing some
benchmarks for gitlab-org/gitlab showed that these computations are
around 100-200x faster than doing the graph walk, reducing the time from
around 200ms to 2-4ms.

This functionality has recently been implemented in Gitaly via the new
`ListAllLFSPointers()` RPC: given a repository, it will simply list all
reachable or unreachable objects. We can now use above semantics when a
pre-receive hook environment is active and strip the repository's
alternative object directory, which will as a result only list newly
pushed objects.
      e07a1c70
    • Patrick Steinhardt's avatar
      Gemfile: Bump gitaly version to v13.11.0.pre.rc1 · d9f880c7
      Patrick Steinhardt authored 
      Gitaly has recently introduced two new RPCs `ListLFSPointers` and
`ListAllLFSPointers` which are slated to replace the current RPCs
`GetLFSPointers`, `GetNewLFSPointers` and `GetAllLFSPointers`.

As a preparatory step to make use of the replacements, this commit
updates the Gitaly Gem to v13.11.0.pre.rc1.
      d9f880c7
  3. 09 Apr, 2021 20 commits
  5. 08 Apr, 2021 18 commits
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7