Commit 4c15508b authored by Marcia Ramos's avatar Marcia Ramos

Merge branch 'alberts-document-order-dependent-flaky-test' into 'master'

Document order-dependent flaky tests

See merge request gitlab-org/gitlab!59369
parents b9470d9e 6775d035
......@@ -76,6 +76,33 @@ For instance `RETRIES=1 bin/rspec ...` would retry the failing examples once.
- [Replace FFaker factory data with sequences](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29643): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10184>
- [Transient failure in spec/finders/issues_finder_spec.rb](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30211#note_26707685): <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10404>
### Order-dependent flaky tests
These flaky tests can fail depending on the order they run with other tests. For example:
- <https://gitlab.com/gitlab-org/gitlab/-/issues/327668>
To identify the tests that lead to such failure, we can use `rspec --bisect`,
which would give us the minimal test combination to reproduce the failure:
```shell
rspec --bisect ee/spec/services/ee/merge_requests/update_service_spec.rb ee/spec/services/ee/notes/quick_actions_service_spec.rb ee/spec/services/epic_links/create_service_spec.rb ee/spec/services/ee/issuable/bulk_update_service_spec.rb
Bisect started using options: "ee/spec/services/ee/merge_requests/update_service_spec.rb ee/spec/services/ee/notes/quick_actions_service_spec.rb ee/spec/services/epic_links/create_service_spec.rb ee/spec/services/ee/issuable/bulk_update_service_spec.rb"
Running suite to find failures... (2 minutes 18.4 seconds)
Starting bisect with 3 failing examples and 144 non-failing examples.
Checking that failure(s) are order-dependent... failure appears to be order-dependent
Round 1: bisecting over non-failing examples 1-144 . ignoring examples 1-72 (1 minute 11.33 seconds)
...
Round 7: bisecting over non-failing examples 132-133 . ignoring example 132 (43.78 seconds)
Bisect complete! Reduced necessary non-failing examples from 144 to 1 in 8 minutes 31 seconds.
The minimal reproduction command is:
rspec ./ee/spec/services/ee/issuable/bulk_update_service_spec.rb[1:2:1:1:1:1,1:2:1:2:1:1,1:2:1:3:1] ./ee/spec/services/epic_links/create_service_spec.rb[1:1:2:2:6:4]
```
We can reproduce the test failure with the reproduction command above. If we change the order of the tests, the test would pass.
### Time-sensitive flaky tests
- <https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10046>
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment