Merge branch 'rspec_bisect_flaky' into 'master'

New script to find minimal order-dependent flaky specs

See merge request gitlab-org/gitlab!75457

doc/development/testing_guide/flaky_tests.md

@@ -82,26 +82,19 @@ These flaky tests can fail depending on the order they run with other tests. For
 
 - <https://gitlab.com/gitlab-org/gitlab/-/issues/327668>
 
-To identify the tests that lead to such failure, we can use `rspec --bisect`,
+To identify the tests that lead to such failure, we can use `scripts/rspec_bisect_flaky`,
 which would give us the minimal test combination to reproduce the failure:
 
-```shell
-rspec --bisect ee/spec/services/ee/merge_requests/update_service_spec.rb ee/spec/services/ee/notes/quick_actions_service_spec.rb ee/spec/services/epic_links/create_service_spec.rb  ee/spec/services/ee/issuable/bulk_update_service_spec.rb
-Bisect started using options: "ee/spec/services/ee/merge_requests/update_service_spec.rb ee/spec/services/ee/notes/quick_actions_service_spec.rb ee/spec/services/epic_links/create_service_spec.rb ee/spec/services/ee/issuable/bulk_update_service_spec.rb"
-Running suite to find failures... (2 minutes 18.4 seconds)
-Starting bisect with 3 failing examples and 144 non-failing examples.
-Checking that failure(s) are order-dependent... failure appears to be order-dependent
-
-Round 1: bisecting over non-failing examples 1-144 . ignoring examples 1-72 (1 minute 11.33 seconds)
-...
-Round 7: bisecting over non-failing examples 132-133 . ignoring example 132 (43.78 seconds)
-Bisect complete! Reduced necessary non-failing examples from 144 to 1 in 8 minutes 31 seconds.
-
-The minimal reproduction command is:
-  rspec ./ee/spec/services/ee/issuable/bulk_update_service_spec.rb[1:2:1:1:1:1,1:2:1:2:1:1,1:2:1:3:1] ./ee/spec/services/epic_links/create_service_spec.rb[1:1:2:2:6:4]
-```
+1. First obtain the list of specs that ran before the flaky test. You can search
+   for the list under `Knapsack node specs:` in the CI job output log.
+1. Save the list of specs as a file, and run:
+
+    ```shell
+    cat knapsack_specs.txt | xargs scripts/rspec_bisect_flaky
+    ```
 
-We can reproduce the test failure with the reproduction command above. If we change the order of the tests, the test would pass.
+If there is an order-dependency issue, the script above will print the minimal
+reproduction.
 
 ### Time-sensitive flaky tests
 

scripts/rspec_bisect_flaky

+#!/usr/bin/env bash
+
+## Usage: scripts/rspec_bisect_flaky <files...>
+#
+# The files should be listed in order, with the last file being the file where
+# the flaky spec lives.
+
+if [ $# -eq 0 ]; then
+  echo "Usage: scripts/rspec_bisect_flaky <files...>"
+  exit
+fi
+
+files=( $@ )
+len=${#files[@]}
+target=${files[$len-1]}
+
+# Trap interrupts and exit instead of continuing the loop
+trap "echo Exited!; exit 2;" SIGINT SIGTERM
+
+# Show which set of specs are running
+set -x
+
+# Do the speedy case first, run each spec with our failing spec
+for file in "${files[@]}"; do
+  bin/rspec $file $target
+done
+
+# Do a full bisect given we did not find candidates with speedy cases 
+bin/rspec --bisect=verbose $@