Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
Léo-Paul Géneau
gitlab-ce
Commits
e6dea313
Commit
e6dea313
authored
Jan 17, 2019
by
Rémy Coutable
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Document better the process around quarantined tests
Signed-off-by:
Rémy Coutable
<
remy@rymai.me
>
parent
f598daf2
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
27 additions
and
6 deletions
+27
-6
doc/development/testing_guide/flaky_tests.md
doc/development/testing_guide/flaky_tests.md
+27
-6
No files found.
doc/development/testing_guide/flaky_tests.md
View file @
e6dea313
...
...
@@ -7,16 +7,37 @@ eventually.
## Quarantined tests
Tests can be put in quarantine by assigning
`:quarantine`
metadata. This means
they will be skipped unless run with
`--tag quarantine`
. This can be used for
tests that are expected to fail while a fix is in progress (similar to how
[
`skip` or `pending`
](
https://relishapp.com/rspec/rspec-core/v/3-8/docs/pending-and-skipped-examples
)
can be used).
When a test frequently fails in
`master`
,
[
a ~"broken master" issue
](
https://about.gitlab.com/handbook/engineering/workflow/#broken-master
)
should be created.
If the test cannot be fixed in a timely fashion, there is an impact on the
productivity of all the developers, so you can decide to put it in quarantine by
assigning the
`:quarantine`
metadata.
```
This means it will be skipped unless run with
`--tag quarantine`
:
```
shell
bin/rspec
--tag
quarantine
```
**
Before putting a test in quarantine, you should make sure that a
~"broken master" issue exists for it so it won't stay in quarantine forever.
**
Once a test is in quarantine, there are 3 choices:
-
Should the test be fixed (i.e. get rid of its flakiness)?
-
Should the test be moved to a lower level of testing?
-
Should the test be removed entirely (e.g. because there's already a
lower-level test, or it's duplicating another same-level test, or it's testing
too much etc.)?
### Quarantine tests on the CI
Quarantined tests are run on the CI in dedicated jobs that are allowed to fail:
-
`rspec-pg-quarantine`
and
`rspec-mysql-quarantine`
(CE & EE)
-
`rspec-pg-quarantine-ee`
and
`rspec-mysql-quarantine-ee`
(EE only)
## Automatic retries and flaky tests detection
On our CI, we use [rspec-retry] to automatically retry a failing example a few
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment