@@ -36,9 +36,16 @@ Personas are described at https://about.gitlab.com/handbook/marketing/product-ma
Add all known Documentation Requirements here, per https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html#documentation-requirements
If this feature requires changing permissions, this document https://docs.gitlab.com/ee/user/permissions.html must be updated accordingly. -->
### Testing
### Availability & Testing
<!-- What risks does this change pose? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? See the test engineering process for further help: https://about.gitlab.com/handbook/engineering/quality/test-engineering/ -->
<!-- What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing?
Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Plese use the list below as guidance.
* Unit test changes
* Integration test changes
* End-to-end test change
See the test engineering planning process and reach out to your counterpart Software Engineer in Test for assistance : https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-planning -->
### What does success look like, and how can we measure that?
%p.mb-2=s_('%{usage_ping_link_start}Learn more%{usage_ping_link_end} about what information is shared with GitLab Inc.').html_safe%{usage_ping_link_start: usage_ping_link_start,usage_ping_link_end: '</a>'.html_safe}
=s_('ClusterIntegration|A cluster management project can be used to run deployment jobs with Kubernetes <code>cluster-admin</code> privileges.').html_safe
=s_('ClusterIntegration|A cluster management project can be used to run deployment jobs with Kubernetes <code>cluster-admin</code> privileges.').html_safe
=s_('User Cohorts are only shown when the %{usage_ping_link_start}usage ping%{usage_ping_link_end} is enabled.').html_safe%{usage_ping_link_start: usage_ping_link_start,usage_ping_link_end: '</a>'.html_safe}
=s_('To enable it and see User Cohorts, visit %{application_settings_link_start}application settings%{application_settings_link_end}.').html_safe%{application_settings_link_start: application_settings_link_start,application_settings_link_end: '</a>'.html_safe}
=s_('To help improve GitLab, we would like to periodically collect usage information. This can be changed at any time in %{settings_link_start}Settings%{link_end}. %{info_link_start}More Information%{link_end}').html_safe%{settings_link_start: settings_link_start,info_link_start: info_link_start,link_end: '</a>'.html_safe}
propagated upstream, through Omnibus, back to the CE / EE merge request.
Please note, we plan to [add more specific information](https://gitlab.com/gitlab-org/quality/team-tasks/issues/156)
about the tests included in each job/scenario that runs in `gitlab-qa`.
##### Running custom tests
The [existing scenarios](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md)
that run in the downstream `gitlab-qa` pipeline include many tests, but there are times when you might want to run a
test or a group of tests that are different than the groups in any of the existing scenarios.
For example, when we [dequarantine](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#dequarantining-tests)
a flaky test we first want to make sure that it's no longer flaky.
We can do that using the `ce:custom-parallel` and `ee:custom-parallel` jobs.
Both are manual jobs that you can configure using custom variables.
When you click the name (not the play icon) of one of the parallel jobs,
you'll be prompted to enter variables. You can use any of [the variables
that can be used with `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables)
as well as these:
| Variable | Description |
|-|-|
| `QA_SCENARIO` | The scenario to run (default `Test::Instance::Image`) |
| `QA_TESTS` | The test(s) to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests via RSpec, e.g., `qa/specs/features/ee/browser_ui` would include all the `EE` UI tests. |
| `QA_RSPEC_TAGS` | The RSpec tags to add (no default) |
For now [manual jobs with custom variables will not use the same variable
when retried](https://gitlab.com/gitlab-org/gitlab/issues/31367), so if you want to run the same test(s) multiple times,
specify the same variables in each `custom-parallel` job (up to as
many of the 10 available jobs that you want to run).
#### Using the `review-qa-all` jobs
On every pipeline during the `test` stage, the `review-qa-smoke` job is
...
...
@@ -105,8 +134,9 @@ See [Review Apps][review-apps] for more details about Review Apps.
## How do I run the tests?
There are two main options for running the tests. If you simply want to run the
existing tests against a live GitLab instance or against a pre-built docker image
If you are not [testing code in a merge request](#testing-code-in-merge-requests),
there are two main options for running the tests. If you simply want to run
the existing tests against a live GitLab instance or against a pre-built docker image
you can use the [GitLab QA orchestrator][gitlab-qa-readme]. See also [examples
of the test scenarios you can run via the orchestrator](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#examples).
