Move remaining secure analyzers to core

Why:

* https://gitlab.com/groups/gitlab-org/-/epics/2098

This change addresses the need by:

* Remove required check for sast feature in CI template
* Remove required check for secret feature in CI template
* Document when analyzers were moved to core

changelogs/unreleased/sa-to-core.yml

+---
+title: Bring SAST to Core - bandit, Flawfinder, Gitleaks, Gosec, Kubesec, NodeJsScan, phpcs-security-audit, PMD, Security Code Scan, Sobelow, SpotBugs
+merge_request: 37648
+author:
+type: changed

doc/user/application_security/secret_detection/index.md

@@ -19,7 +19,7 @@ malicious users to gain access to resources like deployment environments.
 GitLab 11.9 includes a new check called Secret Detection. It scans the content of the repository
 to find API keys and other information that should not be there.
 
-GitLab displays identified secrets as part of the SAST reports visibly in a few places:
+GitLab displays identified secrets visibly in a few places:
 
 - [Security Dashboard](../security_dashboard/)
 - Pipelines' **Security** tab
@@ -46,6 +46,25 @@ CAUTION: **Caution:**
 If you use your own Runners, make sure the Docker version installed
 is **not** `19.03.0`. See [troubleshooting information](../sast#error-response-from-daemon-error-processing-tar-file-docker-tar-relocation-error) for details.
 
+### Making Secret Detection available to all GitLab tiers
+
+To make Secret Detection available to as many customers as possible, we have enabled it for all GitLab tiers.
+However not all features are available on every tier. See the breakdown below for more details.
+
+#### Summary of features per tier
+
+Different features are available in different [GitLab tiers](https://about.gitlab.com/pricing/),
+as shown in the following table:
+
+| Capability                                                                | In Core             | In Ultimate        |
+|:--------------------------------------------------------------------------|:--------------------|:-------------------|
+| [Configure Secret Detection Scanners](#configuration)                                 | **{check-circle}**  | **{check-circle}** |
+| [Customize Secret Detection Settings](#customizing-settings)                 | **{check-circle}**  | **{check-circle}** |
+| View [JSON Report](../sast/index.md#reports-json-format)                                  | **{check-circle}**  | **{check-circle}** |
+| [Presentation of JSON Report in Merge Request](#overview)                 | **{dotted-circle}** | **{check-circle}** |
+| [Interaction with Vulnerabilities](../vulnerabilities/index.md) | **{dotted-circle}** | **{check-circle}** |
+| [Access to Security Dashboard](../security_dashboard/index.md)                       | **{dotted-circle}** | **{check-circle}** |
+
 ## Configuration
 
 NOTE: **Note:**

ee/spec/lib/gitlab/ci/templates/sast_gitlab_ci_yaml_spec.rb

@@ -21,28 +21,6 @@ RSpec.describe 'SAST.gitlab-ci.yml' do
     end
 
     context 'when project has no license' do
-      it 'includes no jobs' do
-        expect { pipeline }.to raise_error(Ci::CreatePipelineService::CreateError)
-      end
-    end
-
-    context 'when project has Ultimate license' do
-      let(:license) { create(:license, plan: License::ULTIMATE_PLAN) }
-
-      before do
-        allow(License).to receive(:current).and_return(license)
-      end
-
-      context 'when SAST_DISABLE_DIND=false' do
-        before do
-          create(:ci_variable, project: project, key: 'SAST_DISABLE_DIND', value: 'false')
-        end
-
-        it 'includes orchestrator job' do
-          expect(build_names).to match_array(%w[sast])
-        end
-      end
-
       context 'when SAST_DISABLED=1' do
         before do
           create(:ci_variable, project: project, key: 'SAST_DISABLED', value: '1')
@@ -96,5 +74,23 @@ RSpec.describe 'SAST.gitlab-ci.yml' do
         end
       end
     end
+
+    context 'when project has Ultimate license' do
+      let(:license) { create(:license, plan: License::ULTIMATE_PLAN) }
+
+      before do
+        allow(License).to receive(:current).and_return(license)
+      end
+
+      context 'when SAST_DISABLE_DIND=false' do
+        before do
+          create(:ci_variable, project: project, key: 'SAST_DISABLE_DIND', value: 'false')
+        end
+
+        it 'includes orchestrator job' do
+          expect(build_names).to match_array(%w[sast])
+        end
+      end
+    end
   end
 end

lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml

@@ -52,8 +52,7 @@ sast:
   rules:
     - if: $SAST_DISABLED || $SAST_DISABLE_DIND == 'false'
       when: never
-    - if: $CI_COMMIT_BRANCH &&
-          $GITLAB_FEATURES =~ /\bsast\b/
+    - if: $CI_COMMIT_BRANCH
   script:
     - /analyzer run
 
@@ -65,7 +64,6 @@ bandit-sast:
     - if: $SAST_DISABLED || $SAST_DISABLE_DIND == 'false'
       when: never
     - if: $CI_COMMIT_BRANCH &&
-          $GITLAB_FEATURES =~ /\bsast\b/ &&
           $SAST_DEFAULT_ANALYZERS =~ /bandit/
       exists:
         - '**/*.py'
@@ -106,7 +104,6 @@ flawfinder-sast:
     - if: $SAST_DISABLED || $SAST_DISABLE_DIND == 'false'
       when: never
     - if: $CI_COMMIT_BRANCH &&
-          $GITLAB_FEATURES =~ /\bsast\b/ &&
           $SAST_DEFAULT_ANALYZERS =~ /flawfinder/
       exists:
         - '**/*.c'
@@ -120,7 +117,6 @@ kubesec-sast:
     - if: $SAST_DISABLED || $SAST_DISABLE_DIND == 'false'
       when: never
     - if: $CI_COMMIT_BRANCH &&
-          $GITLAB_FEATURES =~ /\bsast\b/ &&
           $SAST_DEFAULT_ANALYZERS =~ /kubesec/ &&
           $SCAN_KUBERNETES_MANIFESTS == 'true'
 
@@ -132,7 +128,6 @@ gosec-sast:
     - if: $SAST_DISABLED || $SAST_DISABLE_DIND == 'false'
       when: never
     - if: $CI_COMMIT_BRANCH &&
-          $GITLAB_FEATURES =~ /\bsast\b/ &&
           $SAST_DEFAULT_ANALYZERS =~ /gosec/
       exists:
         - '**/*.go'
@@ -145,7 +140,6 @@ nodejs-scan-sast:
     - if: $SAST_DISABLED || $SAST_DISABLE_DIND == 'false'
       when: never
     - if: $CI_COMMIT_BRANCH &&
-          $GITLAB_FEATURES =~ /\bsast\b/ &&
           $SAST_DEFAULT_ANALYZERS =~ /nodejs-scan/
       exists:
         - 'package.json'
@@ -158,7 +152,6 @@ phpcs-security-audit-sast:
     - if: $SAST_DISABLED || $SAST_DISABLE_DIND == 'false'
       when: never
     - if: $CI_COMMIT_BRANCH &&
-          $GITLAB_FEATURES =~ /\bsast\b/ &&
           $SAST_DEFAULT_ANALYZERS =~ /phpcs-security-audit/
       exists:
         - '**/*.php'
@@ -171,7 +164,6 @@ pmd-apex-sast:
     - if: $SAST_DISABLED || $SAST_DISABLE_DIND == 'false'
       when: never
     - if: $CI_COMMIT_BRANCH &&
-          $GITLAB_FEATURES =~ /\bsast\b/ &&
           $SAST_DEFAULT_ANALYZERS =~ /pmd-apex/
       exists:
         - '**/*.cls'
@@ -184,7 +176,6 @@ secrets-sast:
     - if: $SAST_DISABLED || $SAST_DISABLE_DIND == 'false'
       when: never
     - if: $CI_COMMIT_BRANCH &&
-          $GITLAB_FEATURES =~ /\bsast\b/ &&
           $SAST_DEFAULT_ANALYZERS =~ /secrets/
 
 security-code-scan-sast:
@@ -195,7 +186,6 @@ security-code-scan-sast:
     - if: $SAST_DISABLED || $SAST_DISABLE_DIND == 'false'
       when: never
     - if: $CI_COMMIT_BRANCH &&
-          $GITLAB_FEATURES =~ /\bsast\b/ &&
           $SAST_DEFAULT_ANALYZERS =~ /security-code-scan/
       exists:
         - '**/*.csproj'
@@ -209,7 +199,6 @@ sobelow-sast:
     - if: $SAST_DISABLED || $SAST_DISABLE_DIND == 'false'
       when: never
     - if: $CI_COMMIT_BRANCH &&
-          $GITLAB_FEATURES =~ /\bsast\b/ &&
           $SAST_DEFAULT_ANALYZERS =~ /sobelow/
       exists:
         - 'mix.exs'
@@ -222,7 +211,6 @@ spotbugs-sast:
     - if: $SAST_DISABLED || $SAST_DISABLE_DIND == 'false'
       when: never
     - if: $CI_COMMIT_BRANCH &&
-          $GITLAB_FEATURES =~ /\bsast\b/ &&
           $SAST_DEFAULT_ANALYZERS =~ /spotbugs/
       exists:
         - '**/*.groovy'

lib/gitlab/ci/templates/Security/Secret-Detection.gitlab-ci.yml

@@ -21,8 +21,7 @@ secret_detection_default_branch:
   rules:
     - if: $SECRET_DETECTION_DISABLED
       when: never
-    - if: $CI_DEFAULT_BRANCH == $CI_COMMIT_BRANCH &&
-          $GITLAB_FEATURES =~ /\bsecret_detection\b/
+    - if: $CI_DEFAULT_BRANCH == $CI_COMMIT_BRANCH
   script:
     - /analyzer run
 
@@ -31,8 +30,7 @@ secret_detection:
   rules:
     - if: $SECRET_DETECTION_DISABLED
       when: never
-    - if: $CI_COMMIT_BRANCH && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH &&
-          $GITLAB_FEATURES =~ /\bsecret_detection\b/
+    - if: $CI_COMMIT_BRANCH && $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH
   script:
     - git fetch origin $CI_DEFAULT_BRANCH $CI_BUILD_REF_NAME
     - export SECRET_DETECTION_COMMIT_TO=$(git log --left-right --cherry-pick --pretty=format:"%H" refs/remotes/origin/$CI_DEFAULT_BRANCH...refs/remotes/origin/$CI_BUILD_REF_NAME | tail -n 1)

spec/services/ci/create_pipeline_service_spec.rb

@@ -512,7 +512,7 @@ RSpec.describe Ci::CreatePipelineService do
         it 'pull it from Auto-DevOps' do
           pipeline = execute_service
           expect(pipeline).to be_auto_devops_source
-          expect(pipeline.builds.map(&:name)).to match_array(%w[build code_quality eslint-sast test])
+          expect(pipeline.builds.map(&:name)).to match_array(%w[build code_quality eslint-sast secret_detection_default_branch secrets-sast test])
         end
       end