Merge branch 'reduce-repo-size' into 'master'

Update reduce repo size docs to use `git filter-repo`

Closes #33472 and #219718

See merge request gitlab-org/gitlab!33576

app/helpers/projects_helper.rb

@@ -284,8 +284,8 @@ module ProjectsHelper
     "xcode://clone?repo=#{CGI.escape(default_url_to_repo(project))}"
   end
 
-  def link_to_bfg
-    link_to 'BFG', 'https://rtyley.github.io/bfg-repo-cleaner/', target: '_blank', rel: 'noopener noreferrer'
+  def link_to_filter_repo
+    link_to 'git filter-repo', 'https://github.com/newren/git-filter-repo', target: '_blank', rel: 'noopener noreferrer'
   end
 
   def explore_projects_tab?

app/views/projects/cleanup/_show.html.haml

@@ -6,7 +6,7 @@
     %button.btn.js-settings-toggle
       = expanded ? _('Collapse') : _('Expand')
     %p
-      = _("Clean up after running %{bfg} on the repository" % { bfg: link_to_bfg }).html_safe
+      = _("Clean up after running %{filter_repo} on the repository" % { filter_repo: link_to_filter_repo }).html_safe
       = link_to icon('question-circle'),
           help_page_path('user/project/repository/reducing_the_repo_size_using_git.md'),
           target: '_blank', rel: 'noopener noreferrer'
@@ -22,7 +22,7 @@
             = _("Choose a file")
           %span.prepend-left-default.js-filename
             = _("No file selected")
-          = f.file_field :bfg_object_map, accept: 'text/plain', class: "hidden js-object-map-input", required: true
+          = f.file_field :bfg_object_map, class: "hidden js-object-map-input", required: true
         .form-text.text-muted
           = _("The maximum file size allowed is %{size}.") % { size: number_to_human_size(Gitlab::CurrentSettings.max_attachment_size.megabytes) }
       = f.submit _('Start cleanup'), class: 'btn btn-success'

changelogs/unreleased/reduce-repo-size.yml

+---
+title: Add support for `git filter-repo` to repository cleanup
+merge_request: 33576
+author:
+type: added

doc/topics/git/lfs/index.md

@@ -97,7 +97,7 @@ Read the documentation on how to [migrate an existing Git repo with Git LFS](mig
 
 To remove objects from LFS:
 
-1. Use [BFG-Cleaner](../../../user/project/repository/reducing_the_repo_size_using_git.md#using-the-bfg-repo-cleaner) or [filter-branch](../../../user/project/repository/reducing_the_repo_size_using_git.md#using-git-filter-branch) to remove the objects from the repository.
+1. Use [`git filter-repo`](../../../user/project/repository/reducing_the_repo_size_using_git.md) to remove the objects from the repository.
 1. Delete the relevant LFS lines for the objects you have removed from your `.gitattributes` file and commit those changes.
 
 ## File Locking

doc/user/project/repository/reducing_the_repo_size_using_git.md

@@ -7,147 +7,239 @@ type: howto
 
 # Reducing the repository size using Git
 
-A GitLab Enterprise Edition administrator can set a [repository size limit](../../admin_area/settings/account_and_limit_settings.md)
-which will prevent you from exceeding it.
-
-When a project has reached its size limit, you will not be able to push to it,
-create a new merge request, or merge existing ones. You will still be able to
-create new issues, and clone the project though. Uploading LFS objects will
-also be denied.
-
-If you exceed the repository size limit, your first thought might be to remove
-some data, make a new commit and push back to the repository. Perhaps you can
-move some blobs to LFS, or remove some old dependency updates from history.
-Unfortunately, it's not so easy and that workflow won't work. Deleting files in
-a commit doesn't actually reduce the size of the repo since the earlier commits
-and blobs are still around. What you need to do is rewrite history with Git's
-[`filter-branch` option](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch),
-or an open source community-maintained tool like the
+When large files are added to a Git repository this makes fetching the
+repository slower, because everyone will need to download the file. These files
+can also take up a large amount of storage space on the server over time.
+
+Rewriting a repository can remove unwanted history to make the repository
+smaller. [`git filter-repo`](https://github.com/newren/git-filter-repo) is a
+tool for quickly rewriting Git repository history, and is recommended over [`git
+filter-branch`](https://git-scm.com/docs/git-filter-branch) and
 [BFG](https://rtyley.github.io/bfg-repo-cleaner/).
 
-Note that even with that method, until `git gc` runs on the GitLab side, the
-"removed" commits and blobs will still be around. You also need to be able to
-push the rewritten history to GitLab, which may be impossible if you've already
-exceeded the maximum size limit.
+DANGER: **Danger:**
+Rewriting repository history is a destructive operation. Make sure to backup
+your repository before you begin. The best way is to [export the
+project](../settings/import_export.html#exporting-a-project-and-its-data).
 
-In order to lift these restrictions, the administrator of the GitLab instance
-needs to increase the limit on the particular project that exceeded it, so it's
-always better to spot that you're approaching the limit and act proactively to
-stay underneath it. If you hit the limit, and your admin can't - or won't -
-temporarily increase it for you, your only option is to prune all the unneeded
-stuff locally, and then create a new project on GitLab and start using that
-instead.
+## Purging files from your repository history
 
-If you can continue to use the original project, we recommend [using
-BFG](#using-the-bfg-repo-cleaner), a tool that's built and
-maintained by the open source community. It's faster and simpler than
-`git filter-branch`, and GitLab can use its account of what has changed to clean
-up its own internal state, maximizing the space saved.
+To make cloning your project faster, rewrite branches and tags to remove
+unwanted files.
 
-CAUTION: **Caution:**
-Make sure to first make a copy of your repository since rewriting history will
-purge the files and information you are about to delete. Also make sure to
-inform any collaborators to not use `pull` after your changes, but use `rebase`.
+1. [Install `git
+   filter-repo`](https://github.com/newren/git-filter-repo/blob/master/INSTALL.md)
+   using a supported package manager, or from source.
 
-CAUTION: **Caution:**
-This process is not suitable for removing sensitive data like password or keys
-from your repository. Information about commits, including file content, is
-cached in the database, and will remain visible even after they have been
-removed from the repository.
-
-## Using the BFG Repo-Cleaner
-
-> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/19376) in GitLab 11.6.
-
-1. [Install BFG](https://rtyley.github.io/bfg-repo-cleaner/) from its open source community repository.
-
-1. Navigate to your repository:
+1. Clone a fresh copy of the repository using `--bare`.
 
    ```shell
-   cd my_repository/
+   git clone --bare https://example.gitlab.com/my/project.git
    ```
 
-1. Change to the branch you want to remove the big file from:
+1. Using `git filter-repo`, purge any files from the history of your repository.
+
+   To purge all large files, the `--strip-blobs-bigger-than` option can be used:
 
    ```shell
-   git checkout master
+   git filter-repo --strip-blobs-bigger-than 10M
    ```
 
-1. Create a commit removing the large file from the branch, if it still exists:
+   To purge specific large files by path, the `--path` and `--invert-paths`
+   options can be combined.
 
    ```shell
-   git rm path/to/big_file.mpg
-   git commit -m 'Remove unneeded large file'
+   git filter-repo --path path/to/big/file.m4v --invert-paths
    ```
 
-1. Rewrite history:
+   See the [`git filter-repo`
+   documentation](https://htmlpreview.github.io/?https://github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html#EXAMPLES)
+   for more examples, and the complete documentation.
+
+1. Force push your changes to overwrite all branches on GitLab.
 
    ```shell
-   bfg --delete-files path/to/big_file.mpg
+   git push origin --force --all
    ```
 
-   An object map file will be written to `object-id-map.old-new.txt`. Keep it
-   around - you'll need it for the final step!
+   [Protected Branches](../protected_branches.md) will cause this to fail. To
+   proceed you will need to remove branch protection, push, and then
+   reconfigure protected branches.
 
-1. Force-push the changes to GitLab:
+1. To remove large files from tagged releases, force push your changes to all
+   tags on GitLab.
 
    ```shell
-   git push --force-with-lease origin master
+   git push origin --force --tags
    ```
 
-   If this step fails, someone has changed the `master` branch while you were
-   rewriting history. You could restore the branch and re-run BFG to preserve
-   their changes, or use `git push --force` to overwrite their changes.
+   [Protected Tags](../protected_tags.md) will cause this to
+   fail. To proceed you will need to remove tag protection, push, and then
+   reconfigure protected tags.
 
-1. Navigate to **Project > Settings > Repository > Repository Cleanup**:
+## Purging files from GitLab storage
 
-   ![Repository settings cleanup form](img/repository_cleanup.png)
+To reduce the size of your repository in GitLab you will need to remove GitLab
+internal refs that reference commits contain large files. Before completing
+these steps, first [purged files from your repository
+history](#purging-files-from-your-repository-history).
 
-   Upload the `object-id-map.old-new.txt` file and press **Start cleanup**.
-   This will remove any internal Git references to the old commits, and run
-   `git gc` against the repository. You will receive an email once it has
-   completed.
+As well as branches and tags, which are a type of Git ref, GitLab automatically
+creates other refs. These refs prevent dead links to commits, or missing diffs
+when viewing merge requests. [Repository cleanup](#repository-cleanup) can be
+used to remove these from GitLab.
 
-NOTE: **Note:**
-This process will remove some copies of the rewritten commits from GitLab's
-cache and database, but there are still numerous gaps in coverage - at present,
-some of the copies may persist indefinitely. [Clearing the instance cache](../../../administration/raketasks/maintenance.md#clear-redis-cache)
-may help to remove some of them, but it should not be depended on for security
-purposes!
+The internal refs for merge requests (`refs/merge-requests/*`),
+[pipelines](../../../ci/pipelines/index.md#troubleshooting-fatal-reference-is-not-a-tree)
+(`refs/pipelines/*`), and environments (`refs/environments/*`) are not
+advertised, which means they are not included when fetching, which makes
+fetching faster. The hidden refs to prevent commits with discussion from being
+deleted (`refs/keep-around/*`) cannot be fetched at all. These refs can,
+however, be accessed from the Git bundle inside the project export.
 
-## Using `git filter-branch`
+1. [Install `git
+   filter-repo`](https://github.com/newren/git-filter-repo/blob/master/INSTALL.md)
+   using a supported package manager, or from source.
 
-1. Navigate to your repository:
+1. Generate a fresh [export the
+   project](../settings/import_export.html#exporting-a-project-and-its-data) and
+   download to your computer.
+
+1. Decompress the backup using `tar`
 
    ```shell
-   cd my_repository/
+   tar xzf project-backup.tar.gz
    ```
 
-1. Change to the branch you want to remove the big file from:
+   This will contain a `project.bundle` file, which was created by [`git
+   bundle`](https://git-scm.com/docs/git-bundle)
+
+1. Clone a fresh copy of the repository from the bundle.
 
    ```shell
-   git checkout master
+   git clone --bare --mirror /path/to/project.bundle
    ```
 
-1. Use `filter-branch` to remove the big file:
+1. Using `git filter-repo`, purge any files from the history of your repository.
+   Because we are trying to remove internal refs, we will rely on the
+   `commit-map` produced by each run to tell us which internal refs to remove.
+
+   NOTE:**Note:**
+   `git filter-repo` creates a new `commit-map` file every run, and overwrite the
+   `commit-map` from the previous run. You will need this file from **every**
+   run. Do the next step every time you run `git filter-repo`.
+
+   To purge all large files, the `--strip-blobs-bigger-than` option can be used:
 
    ```shell
-   git filter-branch --force --tree-filter 'rm -f path/to/big_file.mpg' HEAD
+   git filter-repo --strip-blobs-bigger-than 10M
    ```
 
-1. Instruct Git to purge the unwanted data:
+   To purge specific large files by path, the `--path` and `--invert-paths`
+   options can be combined.
 
    ```shell
-   git reflog expire --expire=now --all && git gc --prune=now --aggressive
+   git filter-repo --path path/to/big/file.m4v --invert-paths
    ```
 
-1. Lastly, force push to the repository:
+   See the [`git filter-repo`
+   documentation](https://htmlpreview.github.io/?https://github.com/newren/git-filter-repo/blob/docs/html/git-filter-repo.html#EXAMPLES)
+   for more examples, and the complete documentation.
+
+1. After running `git filter-repo`, the header and unchanged commits need to be
+   removed from the `commit-map` before uploading to GitLab.
 
    ```shell
-   git push --force origin master
+   tail -n +2 filter-repo/commit-map | grep -E -v '^(\w+) \1$' >> commit-map.txt
    ```
 
-Your repository should now be below the size limit.
+   This command can be run after each run of `git filter-repo` to append the
+   output of the run to `commit-map.txt`
+
+1. Navigate to **Project > Settings > Repository > Repository Cleanup**.
+
+   Upload the `commit-map.txt` file and press **Start cleanup**. This will
+   remove any internal Git references to the old commits, and run `git gc`
+   against the repository. You will receive an email once it has completed.
+
+## Repository cleanup
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/19376) in GitLab 11.6.
+
+Repository cleanup allows you to upload a text file of objects and GitLab will remove
+internal Git references to these objects.
+
+To clean up a repository:
+
+1. Go to the project for the repository.
+1. Navigate to **{settings}** **Settings > Repository**.
+1. Upload a list of objects.
+1. Click **Start cleanup**.
+
+This will remove any internal Git references to old commits, and run `git gc`
+against the repository. You will receive an email once it has completed.
+
+These tools produce suitable output for purging history on the server:
+
+- [`git filter-repo`](https://github.com/newren/git-filter-repo): use the
+  `commit-map` file.
+
+- [BFG](https://rtyley.github.io/bfg-repo-cleaner/): use the
+  `object-id-map.old-new.txt` file.
+
+NOTE: **Note:**
+Housekeeping prunes loose objects older than 2 weeks. This means objects added
+in the last 2 weeks will not be removed immediately. If you have access to the
+Gitaly server, you may run `git gc --prune=now` to prune all loose object
+immediately.
+
+NOTE: **Note:**
+This process will remove some copies of the rewritten commits from GitLab's
+cache and database, but there are still numerous gaps in coverage - at present,
+some of the copies may persist indefinitely. [Clearing the instance
+cache](../../../administration/raketasks/maintenance.md#clear-redis-cache) may
+help to remove some of them, but it should not be depended on for security
+purposes!
+
+## Exceeding storage limit
+
+A GitLab Enterprise Edition administrator can set a [repository size
+limit](../../admin_area/settings/account_and_limit_settings.md) which will
+prevent you from exceeding it.
+
+When a project has reached its size limit, you will not be able to push to it,
+create a new merge request, or merge existing ones. You will still be able to
+create new issues, and clone the project though. Uploading LFS objects will
+also be denied.
+
+If you exceed the repository size limit, your first thought might be to remove
+some data, make a new commit and push back to the repository. Perhaps you can
+move some blobs to LFS, or remove some old dependency updates from history.
+Unfortunately, it's not so easy and that workflow won't work. Deleting files in
+a commit doesn't actually reduce the size of the repo since the earlier commits
+and blobs are still around. What you need to do is rewrite history with Git's
+[`filter-branch` option](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch),
+or an open source community-maintained tool like the
+[`git filter-repo`](https://github.com/newren/git-filter-repo).
+
+Note that even with that method, until `git gc` runs on the GitLab side, the
+"removed" commits and blobs will still be around. You also need to be able to
+push the rewritten history to GitLab, which may be impossible if you've already
+exceeded the maximum size limit.
+
+In order to lift these restrictions, the administrator of the GitLab instance
+needs to increase the limit on the particular project that exceeded it, so it's
+always better to spot that you're approaching the limit and act proactively to
+stay underneath it. If you hit the limit, and your admin can't - or won't -
+temporarily increase it for you, your only option is to prune all the unneeded
+stuff locally, and then create a new project on GitLab and start using that
+instead.
+
+CAUTION: **Caution:**
+This process is not suitable for removing sensitive data like password or keys
+from your repository. Information about commits, including file content, is
+cached in the database, and will remain visible even after they have been
+removed from the repository.
 
 <!-- ## Troubleshooting
 

spec/helpers/projects_helper_spec.rb

@@ -685,11 +685,11 @@ describe ProjectsHelper do
     end
   end
 
-  describe 'link_to_bfg' do
-    subject { helper.link_to_bfg }
+  describe 'link_to_filter_repo' do
+    subject { helper.link_to_filter_repo }
 
-    it 'generates a hardcoded link to the BFG Repo-Cleaner' do
-      result = helper.link_to_bfg
+    it 'generates a hardcoded link to git filter-repo' do
+      result = helper.link_to_filter_repo
       doc = Nokogiri::HTML.fragment(result)
 
       expect(doc.children.size).to eq(1)
@@ -702,8 +702,8 @@ describe ProjectsHelper do
         expect(link.name).to eq('a')
         expect(link[:target]).to eq('_blank')
         expect(link[:rel]).to eq('noopener noreferrer')
-        expect(link[:href]).to eq('https://rtyley.github.io/bfg-repo-cleaner/')
-        expect(link.inner_html).to eq('BFG')
+        expect(link[:href]).to eq('https://github.com/newren/git-filter-repo')
+        expect(link.inner_html).to eq('git filter-repo')
 
         expect(result).to be_html_safe
       end