Merge branch 'selhorn-kube-bigger-redo' into 'master'

Fairly large reorg of content

See merge request gitlab-org/gitlab!80508

doc/topics/release_your_application.md

@@ -31,7 +31,7 @@ to Kubernetes clusters using the [GitLab Agent](../user/clusters/agent/install/i
 #### GitOps deployments **(PREMIUM)**
 
 With the [GitLab Agent](../user/clusters/agent/install/index.md), you can perform [pull-based
-deployments of Kubernetes manifests](../user/clusters/agent/repository.md#synchronize-manifest-projects). This provides a scalable, secure, and cloud-native
+deployments of Kubernetes manifests](../user/clusters/agent/gitops.md). This provides a scalable, secure, and cloud-native
 approach to manage Kubernetes deployments.
 
 #### Deploy to Kubernetes with the CI/CD Tunnel

doc/user/application_security/cluster_image_scanning/index.md

@@ -292,7 +292,7 @@ scan images from within your Kubernetes cluster and record the vulnerabilities i
 ### Configuration
 
 The Agent runs the cluster image scanning once the `cluster_image_scanning`
-directive is added to your [Agent's configuration repository](../../clusters/agent/repository.md#scan-your-container-images-for-vulnerabilities).
+directive is added to your [Agent's configuration repository](../../clusters/agent/vulnerabilities.md).
 
 ## Security Dashboard
 
@@ -302,7 +302,7 @@ the security vulnerabilities in your groups, projects, and pipelines.
 ## Interacting with the vulnerabilities
 
 After you find a vulnerability, you can address it in the [vulnerability report](../vulnerabilities/index.md)
-or the [GitLab Agent's](../../clusters/agent/install/index.md#view-vulnerabilities-in-cluster-images)
+or the [GitLab Agent's](../../clusters/agent/vulnerabilities.md)
 details section.
 
 ## Troubleshooting

doc/user/clusters/agent/gitops.md

+---
+stage: Configure
+group: Configure
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
+# Using a GitOps workflow for Kubernetes **(PREMIUM)**
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in GitLab 13.7.
+
+With GitOps, you can manage containerized clusters and applications from a Git repository that:
+
+- Is the single source of truth of your system.
+- Is the single place where you operate your system.
+
+By combining GitLab, Kubernetes, and GitOps, you can have:
+
+- GitLab as the GitOps operator.
+- Kubernetes as the automation and convergence system.
+- GitLab CI/CD for Continuous Integration and the agent for Continuous Deployment.
+
+This diagram shows the repositories and main actors in a GitOps deployment:
+
+```mermaid
+sequenceDiagram
+  participant D as Developer
+  participant A as Application code repository
+  participant M as Manifest repository
+  participant K as GitLab agent
+  participant C as Agent configuration repository
+  loop Regularly
+    K-->>C: Grab the configuration
+  end
+  D->>+A: Pushing code changes
+  A->>M: Updating manifest
+  loop Regularly
+    K-->>M: Watching changes
+    M-->>K: Pulling and applying changes
+  end
+```
+
+For details, view the [architecture documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md#high-level-architecture).
+
+## GitOps workflow steps
+
+To update a Kubernetes cluster by using GitOps, complete the following steps.
+
+1. Ensure you have a working Kubernetes cluster, and that the manifests are in a GitLab project.
+1. In the same project, [register and install the GitLab agent](install/index.md).
+1. Configure the agent configuration file so that the agent monitors the project for changes to the Kubernetes manifests.
+   Use the [GitOps configuration reference](#gitops-configuration-reference) for guidance.
+
+Any time you commit updates to your Kubernetes manifests, the agent updates the cluster.
+
+### GitOps configuration reference
+
+The following snippet shows the possible keys and values for the GitOps section of an agent configuration file.
+
+```yaml
+gitops:
+  # The agent watches projects where your manifests are stored.
+  # When a project changes, the agent deploys the changes to the cluster.
+  manifest_projects:
+    # No authentication mechanisms are currently supported.
+    # The `id` is a path to a Git repository that has
+    # Kubernetes manifests in YAML or JSON format.
+  - id: gitlab-org/cluster-integration/gitlab-agent
+    # Namespace to use if not set explicitly in object manifest.
+    # Also used for inventory ConfigMap objects.
+    default_namespace: my-ns
+    # Paths in the repository you want to scan for manifest files.
+    # Directories with names that start with a dot are ignored.
+    paths:
+      # Read all .yaml files from team1/app1 directory.
+      # See https://github.com/bmatcuk/doublestar#about and
+      # https://pkg.go.dev/github.com/bmatcuk/doublestar/v2#Match for globbing rules.
+    - glob: '/team1/app1/*.yaml'
+      # Read all .yaml files from team2/apps and all subdirectories.
+    - glob: '/team2/apps/**/*.yaml'
+      # If 'paths' is not specified or is an empty list, the configuration below is used.
+    - glob: '/**/*.{yaml,yml,json}'
+    # Reconcile timeout defines whether the applier should wait
+    # until all applied resources have been reconciled, and if so,
+    # how long to wait.
+    reconcile_timeout: 3600s # 1 hour by default
+    # Dry run strategy defines whether changes should actually be performed,
+    # or if it is just talk and no action.
+    # https://github.com/kubernetes-sigs/cli-utils/blob/d6968048dcd80b1c7b55d9e4f31fc25f71c9b490/pkg/common/common.go#L68-L89
+    # Can be: none, client, server
+    dry_run_strategy: none # 'none' by default
+    # Prune defines whether pruning of previously applied
+    # objects should happen after apply.
+    prune: true # enabled by default
+    # Prune timeout defines whether we should wait for all resources
+    # to be fully deleted after pruning, and if so, how long we should
+    # wait.
+    prune_timeout: 3600s # 1 hour by default
+    # Prune propagation policy defines the deletion propagation policy
+    # that should be used for pruning.
+    # https://github.com/kubernetes/apimachinery/blob/44113beed5d39f1b261a12ec398a356e02358307/pkg/apis/meta/v1/types.go#L456-L470
+    # Can be: orphan, background, foreground
+    prune_propagation_policy: foreground # 'foreground' by default
+    # Inventory policy defines if an inventory object can take over
+    # objects that belong to another inventory object or don't
+    # belong to any inventory object.
+    # This is done by determining if the apply/prune operation
+    # can go through for a resource based on comparison of
+    # the inventory-id value in the package and the owning-inventory
+    # annotation (config.k8s.io/owning-inventory) in the live object.
+    # https://github.com/kubernetes-sigs/cli-utils/blob/d6968048dcd80b1c7b55d9e4f31fc25f71c9b490/pkg/inventory/policy.go#L12-L66
+    # Can be: must_match, adopt_if_no_inventory, adopt_all
+    inventory_policy: must_match # 'must_match' by default
+```
+
+## Troubleshooting
+
+### Avoiding conflicts when you have multiple projects
+
+The agent watches each glob pattern set under a project's `paths` section independently, and makes updates to the cluster concurrently.
+If changes are found at multiple paths, when the agent attempts to update the cluster,
+a conflict can occur.
+
+To prevent this from happening, consider storing a logical group of manifests in a single place and reference them only once to avoid overlapping globs.
+
+For example, both of these globs match `*.yaml` files in the root directory
+and could cause conflicts:
+
+```yaml
+gitops:
+  manifest_projects:
+  - id: project1
+    paths:
+    - glob: '/**/*.yaml'
+    - glob: '/*.yaml'
+```
+
+Instead, specify a single glob that matches all `*.yaml` files recursively:
+
+```yaml
+gitops:
+  manifest_projects:
+  - id: project1
+    paths:
+    - glob: '/**/*.yaml'
+```
+
+### Use multiple agents or projects
+
+If you store your Kubernetes manifests in separate GitLab projects,
+update your agent configuration file with the location of these projects.
+
+WARNING:
+The project with the agent's
+configuration file can be private or public. Other projects with Kubernetes manifests must be public. Support for private manifest projects is tracked
+in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/283885).

doc/user/clusters/agent/index.md

@@ -16,11 +16,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w
 You can connect your Kubernetes cluster with GitLab to deploy, manage,
 and monitor your cloud-native solutions. You can choose from two primary workflows.
 
-In a **GitOps workflow**, you keep your Kubernetes manifests in GitLab. You install a GitLab agent in your cluster, and
+In a [**GitOps** workflow](gitops.md), you keep your Kubernetes manifests in GitLab. You install a GitLab agent in your cluster, and
 any time you update your manifests, the agent updates the cluster. This workflow is fully driven with Git and is considered pull-based,
 because the cluster is pulling updates from your GitLab repository.
 
-In a **CI/CD** workflow, you use GitLab CI/CD to query and update your cluster by using the Kubernetes API.
+In a [**CI/CD** workflow](ci_cd_tunnel.md), you use GitLab CI/CD to query and update your cluster by using the Kubernetes API.
 This workflow is considered push-based, because GitLab is pushing requests from GitLab CI/CD to your cluster.
 
 Both of these workflows require you to [install an agent in your cluster](install/index.md).
@@ -47,174 +47,15 @@ version. The list of supported versions is based on:
 
 Some GitLab features might work on versions not listed here.
 
-## Using Kubernetes with GitOps **(PREMIUM)**
+## Migrate to the agent from the legacy certificate-based integration
 
-With GitOps, you can manage containerized clusters and applications from a Git repository that:
-
-- Is the single source of truth of your system.
-- Is the single place where you operate your system.
-
-By combining GitLab, Kubernetes, and GitOps, you can have:
-
-- GitLab as the GitOps operator.
-- Kubernetes as the automation and convergence system.
-- GitLab CI/CD for Continuous Integration and the agent for Continuous Deployment.
-
-Beyond that, you can use all the features offered by GitLab as
-the all-in-one DevOps platform for your product and your team.
-
-### GitOps workflow **(PREMIUM)**
-
-The agent uses multiple GitLab projects to provide a flexible workflow
-that can suit various needs. This diagram shows these repositories and the main
-The agent uses multiple GitLab projects to provide a flexible workflow.
-This diagram shows these repositories and the main
-actors involved in a deployment:
-
-```mermaid
-sequenceDiagram
-  participant D as Developer
-  participant A as Application code repository
-  participant M as Manifest repository
-  participant K as GitLab agent
-  participant C as Agent configuration repository
-  loop Regularly
-    K-->>C: Grab the configuration
-  end
-  D->>+A: Pushing code changes
-  A->>M: Updating manifest
-  loop Regularly
-    K-->>M: Watching changes
-    M-->>K: Pulling and applying changes
-  end
-```
-
-For details, view the [architecture documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md#high-level-architecture).
-
-To perform GitOps deployments, you need:
-
-- A properly-configured Kubernetes cluster where the GitLab agent is running.
-- A project that contains the agent's configuration file (`config.yaml`) in the repository.
-  This file tells the agent which repositories to synchronize with the cluster.
-- A project that contains Kubernetes manifests. Any changes to manifests are applied to the cluster.
-
-You can keep the agent's configuration file and Kubernetes manifests in one project, or you can use multiple.
-
-- One GitLab project (recommended): When you use one project for both the Kubernetes manifests
-  and the agent's configuration file, the projects can be either private or public.
-- Two GitLab projects: When you use two different GitLab projects (one for Kubernetes
-  manifests and another for the agent's configuration file), the project with Kubernetes manifests must
-  be public. The project with the agent's configuration file can be either private or public.
-
-Support for separate private projects is tracked in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/283885).
-
-## Remove an agent
-
-You can remove an agent by using the [GitLab UI](#remove-an-agent-through-the-gitlab-ui) or the [GraphQL API](#remove-an-agent-with-the-gitlab-graphql-api).
-
-### Remove an agent through the GitLab UI
-
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/323055) in GitLab 14.7.
-
-To remove an agent from the UI:
-
-1. On the top bar, select **Menu > Projects** and find the project that contains the agent's configuration file.
-1. From the left sidebar, select **Infrastructure > Kubernetes clusters**.
-1. In the table, in the row for your agent, in the **Options** column, select the vertical ellipsis (**{ellipsis_v}**).
-1. Select **Delete agent**.
-
-### Remove an agent with the GitLab GraphQL API
-
-1. Get the `<cluster-agent-token-id>` from a query in the interactive GraphQL explorer.
-   - For GitLab.com, go to <https://gitlab.com/-/graphql-explorer> to open GraphQL Explorer.
-   - For self-managed GitLab, go to `https://gitlab.example.com/-/graphql-explorer`, replacing `gitlab.example.com` with your instance's URL.
-
-   ```graphql
-   query{
-     project(fullPath: "<full-path-to-agent-configuration-project>") {
-       clusterAgent(name: "<agent-name>") {
-         id
-         tokens {
-           edges {
-             node {
-               id
-             }
-           }
-         }
-       }
-     }
-   }
-   ```
-
-1. Remove an agent record with GraphQL by deleting the `clusterAgentToken`.
-
-   ```graphql
-   mutation deleteAgent {
-     clusterAgentDelete(input: { id: "<cluster-agent-id>" } ) {
-       errors
-     }
-   }
-
-   mutation deleteToken {
-     clusterAgentTokenDelete(input: { id: "<cluster-agent-token-id>" }) {
-       errors
-     }
-   }
-   ```
-
-1. Verify whether the removal occurred successfully. If the output in the Pod logs includes `unauthenticated`, it means that the agent was successfully removed:
-
-   ```json
-   {
-       "level": "warn",
-       "time": "2021-04-29T23:44:07.598Z",
-       "msg": "GetConfiguration.Recv failed",
-       "error": "rpc error: code = Unauthenticated desc = unauthenticated"
-   }
-   ```
-
-1. Delete the agent in your cluster:
-
-   ```shell
-   kubectl delete -n gitlab-kubernetes-agent -f ./resources.yml
-   ```
-
-## Migrating to the agent from the legacy certificate-based integration
-
-Find out how to [migrate to the agent for Kubernetes](../../infrastructure/clusters/migrate_to_gitlab_agent.md) from the certificate-based integration.
-
-## Kubernetes network security alerts **(ULTIMATE)**
-
-> [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476) in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477) in GitLab 15.0.
-
-WARNING:
-Cilium integration is in its end-of-life process. It's [deprecated](https://gitlab.com/groups/gitlab-org/-/epics/7476)
-for use in GitLab 14.8, and planned for [removal](https://gitlab.com/groups/gitlab-org/-/epics/7477)
-in GitLab 15.0.
-
-The agent for Kubernetes also provides an integration with Cilium. This integration provides a simple way to
-generate network policy-related alerts and to surface those alerts in GitLab.
-
-Several components work in concert for the agent to generate the alerts:
-
-- A working Kubernetes cluster.
-- Cilium integration through either of these options:
-  - Installation through [cluster management template](../../project/clusters/protect/container_network_security/quick_start_guide.md#use-the-cluster-management-template-to-install-cilium).
-  - Enablement of [hubble-relay](https://docs.cilium.io/en/v1.8/concepts/overview/#hubble) on an
-    existing installation.
-- One or more network policies through any of these options:
-  - Use the [Container Network Policy editor](../../application_security/policies/index.md#container-network-policy-editor) to create and manage policies.
-  - Use an [AutoDevOps](../../application_security/policies/index.md#container-network-policy) configuration.
-  - Add the required labels and annotations to existing network policies.
-- A configuration repository with [Cilium configured in `config.yaml`](repository.md#surface-network-security-alerts-from-cluster-to-gitlab)
-
-The setup process follows the same [agent's installation steps](install/index.md),
-with the following differences:
-
-- When you define a configuration repository, you must do so with [Cilium settings](repository.md#surface-network-security-alerts-from-cluster-to-gitlab).
-- You do not need to specify the `gitops` configuration section.
+Read about how to [migrate to the agent for Kubernetes](../../infrastructure/clusters/migrate_to_gitlab_agent.md) from the certificate-based integration.
 
 ## Related topics
 
+- [GitOps workflow](gitops.md)
+- [GitLab CI/CD workflow](ci_cd_tunnel.md)
+- [Install the agent](install/index.md)
+- [Work with the agent](repository.md)
 - [Troubleshooting](troubleshooting.md)
-- [Contribute to the GitLab agent's development](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/doc)
+- [Contribute to the agent's development](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/doc)

doc/user/clusters/agent/troubleshooting.md

@@ -188,6 +188,5 @@ Alternatively, you can mount the certificate file at a different location and in
 }
 ```
 
-This error is shown if the manifest project is not public. To fix it,
-[make sure your manifest project is public](repository.md#synchronize-manifest-projects) or your manifest files
+This error is shown if the manifest project is not public. To fix it, make sure your manifest project is public or your manifest files
 are stored in the Agent's configuration repository.

doc/user/clusters/agent/vulnerabilities.md

+---
+stage: Configure
+group: Configure
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
+# Container vulnerability scanning **(ULTIMATE)**
+
+> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6346) in GitLab 14.8 [with a flag](../../../administration/feature_flags.md) named `cluster_vulnerabilities`. Enabled by default.
+
+To view cluster vulnerabilities, you can view the [vulnerability report](../../application_security/vulnerabilities/index.md).
+You can also configure your agent so the vulnerabilities are displayed with other agent information in GitLab.
+
+## View cluster vulnerabilities
+
+Prerequisite:
+
+- You must have at least the Developer role.
+- [Cluster image scanning](../../application_security/cluster_image_scanning/index.md)
+  must be part of your build process.
+
+To view vulnerability information in GitLab:
+
+1. On the top bar, select **Menu > Projects** and find the project that contains the agent configuration file.
+1. On the left sidebar, select **Infrastructure > Kubernetes clusters**.
+1. Select the **Agent** tab.
+1. Select the agent you want to see the vulnerabilities for.
+
+![Cluster agent security tab UI](../img/cluster_agent_security_tab_v14_8.png)
+
+## Enable cluster vulnerability scanning **(ULTIMATE)**
+
+You can use [cluster image scanning](../../application_security/cluster_image_scanning/index.md)
+to scan container images in your cluster for security vulnerabilities.
+
+To begin scanning all resources in your cluster, add a `starboard`
+configuration block to your agent configuration file with no `filters`:
+
+```yaml
+starboard:
+  vulnerability_report:
+    filters: []
+```
+
+The namespaces that are able to be scanned depend on the [Starboard Operator install mode](https://aquasecurity.github.io/starboard/latest/operator/configuration/#install-modes).
+By default, the Starboard Operator only scans resources in the `default` namespace. To change this
+behavior, edit the `STARBOARD_OPERATOR` environment variable in the `starboard-operator` deployment
+definition.
+
+By adding filters, you can limit scans by:
+
+- Resource name
+- Kind
+- Container name
+- Namespace
+
+```yaml
+starboard:
+  vulnerability_report:
+    filters:
+      - namespaces:
+        - staging
+        - production
+        kinds:
+        - Deployment
+        - DaemonSet
+        containers:
+        - ruby
+        - postgres
+        - nginx
+        resources:
+        - my-app-name
+        - postgres
+        - ingress-nginx
+```
+
+A resource is scanned if the resource matches any of the given names and all of the given filter
+types (`namespaces`, `kinds`, `containers`, `resources`). If a filter type is omitted, then all
+names are scanned. In this example, a resource isn't scanned unless it has a container named `ruby`,
+`postgres`, or `nginx`, and it's a `Deployment`:
+
+```yaml
+starboard:
+  vulnerability_report:
+    filters:
+      - kinds:
+        - Deployment
+        containers:
+        - ruby
+        - postgres
+        - nginx
+```
+
+There is also a global `namespaces` field that applies to all filters:
+
+```yaml
+starboard:
+  vulnerability_report:
+    namespaces:
+    - production
+    filters:
+    - kinds:
+      - Deployment
+    - kinds:
+      - DaemonSet
+      resources:
+      - log-collector
+```
+
+In this example, the following resources are scanned:
+
+- All deployments (`Deployment`) in the `production` namespace.
+- All daemon sets (`DaemonSet`) named `log-collector` in the `production` namespace.

doc/user/clusters/management_project_template.md

@@ -69,7 +69,7 @@ We assume that you already have a cluster connected through the Agent and
 
 1. [Create a new project from the Cluster Management Project Template](#create-a-new-project-based-on-the-cluster-management-template).
 This new project is "project B".
-1. In your "project A", [grant the Agent access to the new project (B) through the CI/CD Tunnel](agent/repository.md#authorize-projects-to-use-an-agent).
+1. In your "project A", [grant the Agent access to the new project (B) through the CI/CD Tunnel](agent/ci_cd_tunnel.md#authorize-the-agent).
 1. From the "project's B" settings, add a [new environment variable](../../ci/variables/index.md#add-a-cicd-variable-to-a-project) `$KUBE_CONTEXT` and set it to `path/to/agent-configuration-project:your-agent-name`.
 1. In "project B", [configure the components](#configure-the-available-components) inherited from the template.
 

doc/user/infrastructure/clusters/index.md

@@ -68,5 +68,5 @@ The concept of [project-level](../../project/clusters/index.md),
 extinct in the new model, although the functionality remains to some extent.
 
 The Agent is always configured in a single GitLab project, but you can use the CI/CD Tunnel to
-[authorize other projects and groups to use the same Agent](../../clusters/agent/repository.md#authorize-projects-and-groups-to-use-an-agent).
+authorize other projects and groups to use the same Agent.
 By doing so, you are granting these projects and groups access to the same cluster, which is similar to group-level clusters' use case.

doc/user/infrastructure/clusters/migrate_to_gitlab_agent.md

@@ -34,7 +34,7 @@ all the features available to clusters connected through certificates.
 With GitLab-managed clusters, GitLab creates separate service accounts and namespaces
 for every branch and deploys using these resources.
 
-To achieve a similar result with the GitLab Agent, you can use [impersonation](../../clusters/agent/repository.md#use-impersonation-to-restrict-project-and-group-access)
+To achieve a similar result with the GitLab Agent, you can use [impersonation](../../clusters/agent/ci_cd_tunnel.md#use-impersonation-to-restrict-project-and-group-access)
 strategies to deploy to your cluster with restricted account access. To do so:
 
 1. Choose the impersonation strategy that suits your needs.