Automatic merge of gitlab-org/gitlab master

changelogs/unreleased/323059-batch-load-findings-by-uuid.yml

+---
+title: Batch-load vulnerability findings by UUID
+merge_request: 55642
+author:
+type: performance

changelogs/unreleased/323175-database-timeout-in-pages-migration-task.yml

+---
+title: Add index for pages migration
+merge_request: 55757
+author:
+type: added

db/migrate/20210302103851_add_deployed_deployment_id_index_to_project_pages_metadata.rb

+# frozen_string_literal: true
+
+class AddDeployedDeploymentIdIndexToProjectPagesMetadata < ActiveRecord::Migration[6.0]
+  include Gitlab::Database::MigrationHelpers
+
+  DOWNTIME = false
+
+  disable_ddl_transaction!
+
+  INDEX_NAME = 'index_on_pages_metadata_not_migrated'
+
+  def up
+    add_concurrent_index :project_pages_metadata, :project_id, where: "deployed = TRUE AND pages_deployment_id is NULL", name: INDEX_NAME
+  end
+
+  def down
+    remove_concurrent_index_by_name :project_pages_metadata, INDEX_NAME
+  end
+end

db/schema_migrations/20210302103851

+b2dad27276941e17248f86764196525bd91b088eed78ad7aa6ae2e5a2c9e82bd
\ No newline at end of file

db/structure.sql

@@ -22909,6 +22909,8 @@ CREATE INDEX index_on_namespaces_lower_name ON namespaces USING btree (lower((na
 
 CREATE INDEX index_on_namespaces_lower_path ON namespaces USING btree (lower((path)::text));
 
+CREATE INDEX index_on_pages_metadata_not_migrated ON project_pages_metadata USING btree (project_id) WHERE ((deployed = true) AND (pages_deployment_id IS NULL));
+
 CREATE INDEX index_on_projects_lower_path ON projects USING btree (lower((path)::text));
 
 CREATE INDEX index_on_routes_lower_path ON routes USING btree (lower((path)::text));

doc/install/index.md

@@ -9,110 +9,49 @@ type: index
 
 # Installation **(FREE SELF)**
 
-GitLab can be installed in most GNU/Linux distributions and with several
+GitLab can be installed in most GNU/Linux distributions, and with several
 cloud providers. To get the best experience from GitLab, you must balance
 performance, reliability, ease of administration (backups, upgrades, and
 troubleshooting), and the cost of hosting.
 
-Depending on your platform, select from the following available methods to
-install GitLab:
-
-- [_Omnibus GitLab_](#installing-gitlab-on-linux-using-the-omnibus-gitlab-package-recommended):
-  The official deb/rpm packages that contain a bundle of GitLab and the
-  components it depends on, including PostgreSQL, Redis, and Sidekiq.
-- [_GitLab Helm chart_](#installing-gitlab-on-kubernetes-via-the-gitlab-helm-charts):
-  The cloud native Helm chart for installing GitLab and all of its components
-  on Kubernetes.
-- [_Docker_](#installing-gitlab-with-docker): The Omnibus GitLab packages,
-  Dockerized.
-- [_Source_](#installing-gitlab-from-source): Install GitLab and all of its
-  components from scratch.
-- [_Cloud provider_](#installing-gitlab-on-cloud-providers): Install directly
-  from platforms like AWS, Azure, and GCP.
-
-If you're not sure which installation method to use, we recommend you use
-Omnibus GitLab. The Omnibus GitLab packages are mature,
-[scalable](../administration/reference_architectures/index.md), and are used
-today on GitLab.com. The Helm charts are recommended for those who are familiar
-with Kubernetes.
-
 ## Requirements
 
 Before you install GitLab, be sure to review the [system requirements](requirements.md).
 The system requirements include details about the minimum hardware, software,
 database, and additional requirements to support GitLab.
 
-## Installing GitLab on Linux using the Omnibus GitLab package (recommended)
-
-The Omnibus GitLab package uses our official deb/rpm repositories, and is
-recommended for most users.
-
-If you need additional scale or resilience, we recommend deploying
-GitLab as described in our [reference architecture documentation](../administration/reference_architectures/index.md).
-
-[**> Install GitLab using the Omnibus GitLab package.**](https://about.gitlab.com/install/)
-
-### GitLab Environment Toolkit (GET)
-
-The [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit) (GET) provides a set of automation tools to easily deploy a [reference architecture](../administration/reference_architectures/index.md) on most major cloud providers.
-
-It is currently in beta, and is not yet recommended for production use.
-
-[**> Install a GitLab reference architecture using the GitLab Environment Toolkit.**](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit#documentation)
+## Choose the installation method
 
-## Installing GitLab on Kubernetes via the GitLab Helm charts
-
-When installing GitLab on Kubernetes, there are some trade-offs that you
-need to be aware of:
-
-- Administration and troubleshooting requires Kubernetes knowledge.
-- It can be more expensive for smaller installations. The default installation
-  requires more resources than a single node Omnibus deployment, as most services
-  are deployed in a redundant fashion.
-- There are some feature [limitations to be aware of](https://docs.gitlab.com/charts/#limitations).
-
-Due to these trade-offs, having Kubernetes experience is a requirement for
-using this method. We recommend being familiar with Kubernetes before using it
-to deploy GitLab in production. The methods for management, observability, and
-some concepts are different than traditional deployments.
-
-[**> Install GitLab on Kubernetes using the GitLab Helm charts.**](https://docs.gitlab.com/charts/)
-
-## Installing GitLab with Docker
-
-GitLab maintains a set of official Docker images based on the Omnibus GitLab
-package.
-
-[**> Install GitLab using the official GitLab Docker images.**](docker.md)
-
-## Installing GitLab from source
-
-If the Omnibus GitLab package isn't available for your distribution, you can
-install GitLab from source. This can be useful with unsupported systems, like
-\*BSD. For an overview of the directory structure, see the
-[structure documentation](installation.md#gitlab-directory-structure).
+Depending on your platform, select from the following available methods to
+install GitLab:
 
-[**> Install GitLab from source.**](installation.md)
+| Installation method                                            | Description | When to choose |
+|----------------------------------------------------------------|-------------|----------------|
+| [Linux package](https://docs.gitlab.com/omnibus/installation/) | The official deb/rpm packages (also known as Omnibus GitLab) that contains a bundle of GitLab and the components it depends on, including PostgreSQL, Redis, and Sidekiq. | This is the recommended method for getting started. The Linux packages are mature, scalable, and are used today on GitLab.com. If you need additional flexibility and resilience, we recommend deploying GitLab as described in the [reference architecture documentation](../administration/reference_architectures/index.md). |
+| [Helm charts](https://docs.gitlab.com/charts/)                 | The cloud native Helm chart for installing GitLab and all of its components on Kubernetes. | When installing GitLab on Kubernetes, there are some trade-offs that you need to be aware of: <br/>- Administration and troubleshooting requires Kubernetes knowledge.<br/>- It can be more expensive for smaller installations. The default installation requires more resources than a single node Linux package deployment, as most services are deployed in a redundant fashion.<br/>- There are some feature [limitations to be aware of](https://docs.gitlab.com/charts/#limitations).<br/><br/> Use this method if your infrastructure is built on Kubernetes and you're familiar with how it works. The methods for management, observability, and some concepts are different than traditional deployments. |
+| [Docker](https://docs.gitlab.com/omnibus/docker/)              | The GitLab packages, Dockerized. | Use this method if you're familiar with Docker. |
+| [Source](installation.md)                                      | Install GitLab and all of its components from scratch. | Use this method if none of the previous methods are available for your platform. Useful for unsupported systems like \*BSD.|
+| [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit#documentation) | The GitLab Environment toolkit provides a set of automation tools to deploy a [reference architecture](../administration/reference_architectures/index.md) on most major cloud providers. | Since GET is in beta and not yet recommended for production use, use this method if you want to test deploying GitLab in scalable environment. |
 
-## Installing GitLab on cloud providers
+## Install GitLab on cloud providers
 
-GitLab can be installed on a variety of cloud providers by using any of
-the above methods, provided the cloud provider supports it.
+Regardless of the installation method, you can install GitLab on several cloud
+providers, assuming the cloud provider supports it. Here are several possible installation
+methods, the majority which use the Linux packages:
 
-- [Install on AWS](aws/index.md): Install Omnibus GitLab on AWS using the community AMIs that GitLab provides.
-- [Install GitLab on Google Cloud Platform](google_cloud_platform/index.md): Install Omnibus GitLab on a VM in GCP.
-- [Install GitLab on Azure](azure/index.md): Install Omnibus GitLab from Azure Marketplace.
-- [Install GitLab on OpenShift](https://docs.gitlab.com/charts/installation/cloud/openshift.html): Install GitLab on OpenShift by using the GitLab Helm charts.
-- [Install GitLab on DigitalOcean](https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-gitlab-on-ubuntu-18-04): Install Omnibus GitLab on DigitalOcean.
-- _Testing only!_ [DigitalOcean and Docker Machine](digitaloceandocker.md):
-  Quickly test any version of GitLab on DigitalOcean using Docker Machine.
+| Cloud provider                                                | Description |
+|---------------------------------------------------------------|-------------|
+| [AWS (HA)](aws/index.md)                                      | Install GitLab on AWS using the community AMIs provided by GitLab. |
+| [Google Cloud Platform (GCP)](google_cloud_platform/index.md) | Install GitLab on a VM in GCP. |
+| [Azure](azure/index.md)                                       | Install GitLab from Azure Marketplace. |
+| [DigitalOcean](https://about.gitlab.com/blog/2016/04/27/getting-started-with-gitlab-and-digitalocean/) | Install GitLab on DigitalOcean. You can also [test GitLab on DigitalOcean using Docker Machine](digitaloceandocker.md). |
 
 ## Next steps
 
 Here are a few resources you might want to check out after completing the
 installation:
 
-- [Upload a license](../user/admin_area/license.md)  or [start a free trial](https://about.gitlab.com/free-trial/):
+- [Upload a license](../user/admin_area/license.md) or [start a free trial](https://about.gitlab.com/free-trial/):
   Activate all GitLab Enterprise Edition functionality with a license.
 - [Set up runners](https://docs.gitlab.com/runner/): Set up one or more GitLab
   Runners, the agents that are responsible for all of the GitLab CI/CD features.

doc/user/application_security/secret_detection/index.md

@@ -102,8 +102,7 @@ as shown in the following table:
 Secret Detection is performed by a [specific analyzer](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/Secret-Detection.gitlab-ci.yml)
 during the `secret-detection` job. It runs regardless of your app's programming language.
 
-The Secret Detection analyzer includes [Gitleaks](https://github.com/zricethezav/gitleaks) and
-[TruffleHog](https://github.com/dxa4481/truffleHog) checks.
+The Secret Detection analyzer includes [Gitleaks](https://github.com/zricethezav/gitleaks) checks.
 
 Note that the Secret Detection analyzer ignores Password-in-URL vulnerabilities if the password
 begins with a dollar sign (`$`), as this likely indicates the password is an environment variable.
@@ -200,7 +199,7 @@ Secret Detection can be customized by defining available CI/CD variables:
 > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211387) in GitLab 13.5.
 
 You can customize the default secret detection rules provided with GitLab.
-Customization allows you to exclude rules and add new rules.
+Customization allows replace the default secret detection rules with rules that you define.
 
 To create a custom ruleset:
 

ee/app/services/security/store_report_service.rb

@@ -33,7 +33,15 @@ module Security
     end
 
     def create_all_vulnerabilities!
-      @report.findings.map { |finding| create_vulnerability_finding(finding)&.id }.compact.uniq
+      # Look for existing Findings using UUID
+      finding_uuids = @report.findings.map(&:uuid)
+      vulnerability_findings_by_uuid = project.vulnerability_findings
+        .where(uuid: finding_uuids) # rubocop: disable CodeReuse/ActiveRecord
+        .to_h { |vf| [vf.uuid, vf] }
+
+      @report.findings.map do |finding|
+        create_vulnerability_finding(vulnerability_findings_by_uuid, finding)&.id
+      end.compact.uniq
     end
 
     def mark_as_resolved_except(vulnerability_ids)
@@ -43,7 +51,7 @@ module Security
              .update_all(resolved_on_default_branch: true)
     end
 
-    def create_vulnerability_finding(finding)
+    def create_vulnerability_finding(vulnerability_findings_by_uuid, finding)
       unless finding.valid?
         put_warning_for(finding)
         return
@@ -51,7 +59,9 @@ module Security
 
       vulnerability_params = finding.to_hash.except(:compare_key, :identifiers, :location, :scanner, :scan, :links)
       entity_params = Gitlab::Json.parse(vulnerability_params&.dig(:raw_metadata)).slice('description', 'message', 'solution', 'cve', 'location')
-      vulnerability_finding = create_or_find_vulnerability_finding(finding, vulnerability_params.merge(entity_params))
+      # Vulnerabilities::Finding (`vulnerability_occurrences`)
+      vulnerability_finding = vulnerability_findings_by_uuid[finding.uuid] ||
+        create_new_vulnerability_finding(finding, vulnerability_params.merge(entity_params))
 
       update_vulnerability_scanner(finding)
 
@@ -73,7 +83,7 @@ module Security
     end
 
     # rubocop: disable CodeReuse/ActiveRecord
-    def create_or_find_vulnerability_finding(finding, create_params)
+    def create_new_vulnerability_finding(finding, create_params)
       find_params = {
         scanner: scanners_objects[finding.scanner.key],
         primary_identifier: identifiers_objects[finding.primary_identifier.key],
@@ -81,21 +91,15 @@ module Security
       }
 
       begin
-        # Look for existing Findings using UUID
-        vulnerability_finding = project.vulnerability_findings.find_by(uuid: finding.uuid)
-
         # If there's no Finding then we're dealing with one of two cases:
         # 1. The Finding is a new one
         # 2. The Finding is already saved but has UUIDv4
-        unless vulnerability_finding
-          vulnerability_finding = project.vulnerability_findings
-            .create_with(create_params)
-            .find_or_initialize_by(find_params)
-          vulnerability_finding.uuid = finding.uuid
+        project.vulnerability_findings
+          .create_with(create_params)
+          .find_or_initialize_by(find_params).tap do |f|
+          f.uuid = finding.uuid
+          f.save!
         end
-
-        vulnerability_finding.save!
-        vulnerability_finding
       rescue ActiveRecord::RecordNotUnique => e
         # This might happen if we're processing another report in parallel and it finds the same Finding
         # faster. In that case we need to perform the lookup again