Merge branch 'docs-verbose-registry-reduction-options' into 'master'

Docs: Container registry storage reduction improvements

See merge request gitlab-org/gitlab!79420

doc/user/packages/container_registry/index.md

@@ -600,93 +600,6 @@ To address the `Blob unknown to registry` error, include the architecture in the
 individual images. For example, use `mygroup/myapp:1.0.0-amd64` and `mygroup/myapp:1.0.0-arm64v8`.
 You can then tag the manifest list with `mygroup/myapp:1.0.0`.
 
-### The cleanup policy doesn't delete any tags
-
-There can be different reasons behind this:
-
-- In GitLab 13.6 and earlier, when you run the cleanup policy you may expect it to delete tags and
-  it does not. This occurs when the cleanup policy is saved without editing the value in the
-  **Remove tags matching** field. This field has a grayed out `.*` value as a placeholder. Unless
-  `.*` (or another regex pattern) is entered explicitly into the field, a `nil` value is submitted.
-  This value prevents the saved cleanup policy from matching any tags. As a workaround, edit the
-  cleanup policy. In the **Remove tags matching** field, enter `.*` and save. This value indicates
-  that all tags should be removed.
-
-- If you are on GitLab self-managed instances and you have 1000+ tags in a container repository, you
-  might run into a [Container Registry token expiration issue](https://gitlab.com/gitlab-org/gitlab/-/issues/288814),
-  with `error authorizing context: invalid token` in the logs.
-
-  To fix this, there are two workarounds:
-
-  - If you are on GitLab 13.9 or later, you can [set limits for the cleanup policy](reduce_container_registry_storage.md#set-cleanup-limits-to-conserve-resources).
-    This limits the cleanup execution in time, and avoids the expired token error.
-
-  - Extend the expiration delay of the Container Registry authentication tokens. This defaults to 5
-    minutes. You can set a custom value by running
-    `ApplicationSetting.last.update(container_registry_token_expire_delay: <integer>)` in the Rails
-    console, where `<integer>` is the desired number of minutes. For reference, 15 minutes is the
-    value currently in use for GitLab.com. Be aware that by extending this value you increase the
-    time required to revoke permissions.
-
-If the previous fixes didn't work or you are on earlier versions of GitLab, you can generate a list
-of the tags that you want to delete, and then use that list to delete the tags. To do this, follow
-these steps:
-
-1. Run the following shell script. The command just before the `for` loop ensures that
-   `list_o_tags.out` is always reinitialized when starting the loop. After running this command, all
-   the tags' names will be in the `list_o_tags.out` file:
-
-   ```shell
-   # Get a list of all tags in a certain container repository while considering [pagination](../../../api/index.md#pagination)
-   echo -n "" > list_o_tags.out; for i in {1..N}; do curl --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags?per_page=100&page=${i}" | jq '.[].name' | sed 's:^.\(.*\).$:\1:' >> list_o_tags.out; done
-   ```
-
-   If you have Rails console access, you can enter the following commands to retrieve a list of tags limited by date:
-
-   ```shell
-   output = File.open( "/tmp/list_o_tags.out","w" )
-   Project.find(<Project_id>).container_repositories.find(<container_repo_id>).tags.each do |tag|
-     output << tag.name + "\n" if tag.created_at < 1.month.ago
-   end;nil
-   output.close
-   ```
-
-   This set of commands creates a `/tmp/list_o_tags.out` file listing all tags with a `created_at` date of older than one month.
-
-1. Remove from the `list_o_tags.out` file any tags that you want to keep. Here are some example
-   `sed` commands for this. Note that these commands are simply examples. You may change them to
-   best suit your needs:
-
-   ```shell
-   # Remove the `latest` tag from the file
-   sed -i '/latest/d' list_o_tags.out
-
-   # Remove the first N tags from the file
-   sed -i '1,Nd' list_o_tags.out
-
-   # Remove the tags starting with `Av` from the file
-   sed -i '/^Av/d' list_o_tags.out
-
-   # Remove the tags ending with `_v3` from the file
-   sed -i '/_v3$/d' list_o_tags.out
-   ```
-
-   If you are running macOS, you must add `.bak` to the commands. For example:
-
-   ```shell
-   sed -i .bak '/latest/d' list_o_tags.out
-   ```
-
-1. Double-check the `list_o_tags.out` file to make sure it contains only the tags that you want to
-   delete.
-
-1. Run this shell script to delete the tags in the `list_o_tags.out` file:
-
-   ```shell
-   # loop over list_o_tags.out to delete a single tag at a time
-   while read -r LINE || [[ -n $LINE ]]; do echo ${LINE}; curl --request DELETE --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags/${LINE}"; sleep 0.1; echo; done < list_o_tags.out > delete.logs
-   ```
-
 ### Troubleshoot as a GitLab server administrator
 
 Troubleshooting the GitLab Container Registry, most of the times, requires

doc/user/packages/container_registry/reduce_container_registry_storage.md

@@ -16,8 +16,9 @@ to automatically manage your container registry usage.
 
 ## Check Container Registry Storage Use
 
-The Usage Quotas page (**Settings > Usage Quotas > Storage**) displays storage usage for Packages, which includes Container Registry,
-however, the storage is not being calculated.
+The Usage Quotas page (**Settings > Usage Quotas > Storage**) displays storage usage for Packages,
+which doesn't include the Container Registry. To track work on this, see the epic
+[Storage management for the Container Registry](https://gitlab.com/groups/gitlab-org/-/epics/7226).
 
 ## Cleanup policy
 
@@ -252,21 +253,108 @@ It is recommended you only enable container cleanup
 policies for projects that were created before GitLab 12.8 if you are confident the number of tags
 being cleaned up is minimal.
 
-## Related topics
+## More Container Registry storage reduction options
 
-- [Delete images](index.md#delete-images)
-- [Delete registry repository](../../../api/container_registry.md#delete-registry-repository)
-- [Delete a registry repository tag](../../../api/container_registry.md#delete-a-registry-repository-tag)
-- [Delete registry repository tags in bulk](../../../api/container_registry.md#delete-registry-repository-tags-in-bulk)
-- [Delete a package](../package_registry/reduce_package_registry_storage.md#delete-a-package)
+Here are some other options to reduce your project's use of Container Registry storage:
 
-## Troubleshooting cleanup policies
+- Use the [GitLab UI](index.md#delete-images)
+  to delete individual image tags or the entire repository containing all the tags.
+- Use the API to [delete individual image tags](../../../api/container_registry.md#delete-a-registry-repository-tag).
+- Use the API to [delete the entire container registry repository containing all the tags](../../../api/container_registry.md#delete-registry-repository).
+- Use the API to [delete registry repository tags in bulk](../../../api/container_registry.md#delete-registry-repository-tags-in-bulk).
 
-If you see the following message:
+## Troubleshooting cleanup policies
 
-"Something went wrong while updating the cleanup policy."
+### `Something went wrong while updating the cleanup policy.`
 
-Check the regex patterns to ensure they are valid.
+If you see this error message, check the regex patterns to ensure they are valid.
 
 GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in the cleanup policy. You can test them with the [regex101 regex tester](https://regex101.com/).
 View some common [regex pattern examples](#regex-pattern-examples).
+
+### The cleanup policy doesn't delete any tags
+
+There can be different reasons behind this:
+
+- In GitLab 13.6 and earlier, when you run the cleanup policy you may expect it to delete tags and
+  it does not. This occurs when the cleanup policy is saved without editing the value in the
+  **Remove tags matching** field. This field has a grayed out `.*` value as a placeholder. Unless
+  `.*` (or another regex pattern) is entered explicitly into the field, a `nil` value is submitted.
+  This value prevents the saved cleanup policy from matching any tags. As a workaround, edit the
+  cleanup policy. In the **Remove tags matching** field, enter `.*` and save. This value indicates
+  that all tags should be removed.
+
+- If you are on GitLab self-managed instances and you have 1000+ tags in a container repository, you
+  might run into a [Container Registry token expiration issue](https://gitlab.com/gitlab-org/gitlab/-/issues/288814),
+  with `error authorizing context: invalid token` in the logs.
+
+  To fix this, there are two workarounds:
+
+  - If you are on GitLab 13.9 or later, you can [set limits for the cleanup policy](reduce_container_registry_storage.md#set-cleanup-limits-to-conserve-resources).
+    This limits the cleanup execution in time, and avoids the expired token error.
+
+  - Extend the expiration delay of the Container Registry authentication tokens. This defaults to 5
+    minutes. You can set a custom value by running
+    `ApplicationSetting.last.update(container_registry_token_expire_delay: <integer>)` in the Rails
+    console, where `<integer>` is the desired number of minutes. For reference, 15 minutes is the
+    value currently in use for GitLab.com. Be aware that by extending this value you increase the
+    time required to revoke permissions.
+
+If the previous fixes didn't work or you are on earlier versions of GitLab, you can generate a list
+of the tags that you want to delete, and then use that list to delete the tags. To do this, follow
+these steps:
+
+1. Run the following shell script. The command just before the `for` loop ensures that
+   `list_o_tags.out` is always reinitialized when starting the loop. After running this command, all
+   the tags' names will be in the `list_o_tags.out` file:
+
+   ```shell
+   # Get a list of all tags in a certain container repository while considering [pagination](../../../api/index.md#pagination)
+   echo -n "" > list_o_tags.out; for i in {1..N}; do curl --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags?per_page=100&page=${i}" | jq '.[].name' | sed 's:^.\(.*\).$:\1:' >> list_o_tags.out; done
+   ```
+
+   If you have Rails console access, you can enter the following commands to retrieve a list of tags limited by date:
+
+   ```shell
+   output = File.open( "/tmp/list_o_tags.out","w" )
+   Project.find(<Project_id>).container_repositories.find(<container_repo_id>).tags.each do |tag|
+     output << tag.name + "\n" if tag.created_at < 1.month.ago
+   end;nil
+   output.close
+   ```
+
+   This set of commands creates a `/tmp/list_o_tags.out` file listing all tags with a `created_at` date of older than one month.
+
+1. Remove from the `list_o_tags.out` file any tags that you want to keep. Here are some example
+   `sed` commands for this. Note that these commands are simply examples. You may change them to
+   best suit your needs:
+
+   ```shell
+   # Remove the `latest` tag from the file
+   sed -i '/latest/d' list_o_tags.out
+
+   # Remove the first N tags from the file
+   sed -i '1,Nd' list_o_tags.out
+
+   # Remove the tags starting with `Av` from the file
+   sed -i '/^Av/d' list_o_tags.out
+
+   # Remove the tags ending with `_v3` from the file
+   sed -i '/_v3$/d' list_o_tags.out
+   ```
+
+   If you are running macOS, you must add `.bak` to the commands. For example:
+
+   ```shell
+   sed -i .bak '/latest/d' list_o_tags.out
+   ```
+
+1. Double-check the `list_o_tags.out` file to make sure it contains only the tags that you want to
+   delete.
+
+1. Run this shell script to delete the tags in the `list_o_tags.out` file:
+
+   ```shell
+   # loop over list_o_tags.out to delete a single tag at a time
+   while read -r LINE || [[ -n $LINE ]]; do echo ${LINE}; curl --request DELETE --header 'PRIVATE-TOKEN: <PAT>' "https://gitlab.example.com/api/v4/projects/<Project_id>/registry/repositories/<container_repo_id>/tags/${LINE}"; sleep 0.1; echo; done < list_o_tags.out > delete.logs
+   ```