Allow to quarantine end-to-end tests per environment

This builds upon https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35668
and allows to quarantine end-to-end tests only when they run against
specific environments.
Signed-off-by: Rémy Coutable <remy@rymai.me>

doc/development/testing_guide/end_to_end/environment_selection.md

@@ -33,7 +33,7 @@ RSpec.describe 'Area' do
   it 'runs in any environment' do; end
 
   it 'runs only in production', only: :production do; end
-  
+
   it 'runs only in staging', only: { subdomain: :staging } do; end
 
   it 'runs in dev', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end
@@ -45,3 +45,10 @@ end
 NOTE: **Note:**
 If the test has a `before` or `after`, you must add the `only` metadata
 to the outer `RSpec.describe`.
+
+## Quarantining a test for a specific environment
+
+Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a
+test only when it runs against a specific environment. The syntax is exactly the same, except that the `only: { ... }`
+hash is nested in the [`quarantine: { ... }`](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests) hash.
+For instance, `quarantine: { only: { subdomain: :staging } }` will only quarantine the test when run against staging.

doc/development/testing_guide/end_to_end/rspec_metadata_tests.md

@@ -10,7 +10,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec
 | `:elasticsearch`  | The test requires an Elasticsearch service. It is used by the [instance-level scenario](https://gitlab.com/gitlab-org/gitlab-qa#definitions) [`Test::Integration::Elasticsearch`](https://gitlab.com/gitlab-org/gitlab/-/blob/72b62b51bdf513e2936301cb6c7c91ec27c35b4d/qa/qa/ee/scenario/test/integration/elasticsearch.rb) to include only tests that require Elasticsearch. |
 | `:kubernetes`     | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. *This tag is often be paired with `:orchestrated`.* |
 | `:orchestrated`   | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify GitLab's configuration (for example, Staging). |
-| `:quarantine`     | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. |
+| `:quarantine`     | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). |
 | `:reliable`       | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. |
 | `:requires_admin` | The test requires an admin account. Tests with the tag are excluded when run against Canary and Production environments. |
 | `:runner`         | The test depends on and will set up a GitLab Runner instance, typically to run a pipeline. |

qa/qa/specs/helpers/quarantine.rb

@@ -20,7 +20,7 @@ module QA
               Quarantine.skip_or_run_quarantined_tests_or_contexts(config.inclusion_filter.rules, example)
 
               if example.metadata.key?(:only)
-                skip('Test is not compatible with this environment') unless Runtime::Env.dot_com?(example.metadata[:only])
+                skip('Test is not compatible with this environment') unless Runtime::Env.address_matches?(example.metadata[:only])
               end
             end
           end
@@ -50,21 +50,15 @@ module QA
             skip("Only running tests tagged with :quarantine and any of #{included_filters.keys}") if should_skip_when_focused?(example.metadata, included_filters)
           else
             if example.metadata.key?(:quarantine)
-              quarantine_message = %w(In quarantine)
               quarantine_tag = example.metadata[:quarantine]
 
-              if !!quarantine_tag
-                quarantine_message << case quarantine_tag
-                                      when String
-                                        ": #{quarantine_tag}"
-                                      when Hash
-                                        ": #{quarantine_tag[:issue]}"
-                                      else
-                                        ''
-                                      end
+              if quarantine_tag&.is_a?(Hash) && quarantine_tag&.key?(:only)
+                # If the :quarantine hash contains :only, we respect that.
+                # For instance `quarantine: { only: { subdomain: :staging } }` will only quarantine the test when it runs against staging.
+                return unless Runtime::Env.address_matches?(quarantine_tag[:only])
               end
 
-              skip(quarantine_message.join(' ').strip)
+              skip(quarantine_message(quarantine_tag))
             end
           end
         end
@@ -73,6 +67,20 @@ module QA
           filter.reject { |key, _| key == :quarantine }
         end
 
+        def quarantine_message(quarantine_tag)
+          quarantine_message = %w(In quarantine)
+          quarantine_message << case quarantine_tag
+                                when String
+                                  ": #{quarantine_tag}"
+                                when Hash
+                                  quarantine_tag.key?(:issue) ? ": #{quarantine_tag[:issue]}" : ''
+                                else
+                                  ''
+                                end
+
+          quarantine_message.join(' ').strip
+        end
+
         # Checks if a test or context should be skipped.
         #
         # Returns true if

qa/spec/specs/helpers/quarantine_spec.rb

@@ -124,7 +124,7 @@ describe QA::Specs::Helpers::Quarantine do
     end
   end
 
-  describe '.skip_or_run_quarantined_tests' do
+  describe '.skip_or_run_quarantined_tests_or_contexts' do
     context 'with no tag focused' do
       before do
         described_class.configure_rspec
@@ -148,6 +148,37 @@ describe QA::Specs::Helpers::Quarantine do
         expect(group.examples.first.execution_result.status).to eq(:passed)
       end
 
+      context 'with environment set' do
+        before do
+          QA::Runtime::Scenario.define(:gitlab_address, 'https://staging.gitlab.com')
+          described_class.configure_rspec
+        end
+
+        it 'is skipped when set on contexts or descriptions' do
+          group = describe_successfully 'Quarantined in staging', quarantine: { only: { subdomain: :staging } } do
+            it('runs in staging') {}
+          end
+
+          expect(group.examples.first.execution_result.status).to eq(:pending)
+          expect(group.examples.first.execution_result.pending_message)
+            .to eq('In quarantine')
+        end
+
+        it 'is skipped only in staging' do
+          group = describe_successfully do
+            it('skipped in staging', quarantine: { only: { subdomain: :staging } }) {}
+            it('runs in staging', quarantine: { only: :production }) {}
+            it('skipped in staging also', quarantine: { only: { subdomain: %i[release staging] } }) {}
+            it('runs in any env') {}
+          end
+
+          expect(group.examples[0].execution_result.status).to eq(:pending)
+          expect(group.examples[1].execution_result.status).to eq(:passed)
+          expect(group.examples[2].execution_result.status).to eq(:pending)
+          expect(group.examples[3].execution_result.status).to eq(:passed)
+        end
+      end
+
       context 'quarantine message' do
         shared_examples 'test with quarantine message' do |quarantine_tag|
           it 'outputs the quarantine message' do