if you have any custom console outputs as part of your rspec
config or setup, the json parser will fail to parse that output
and fail the build. Adding SUPPRESS_OUTPUT=1 so we can check for
that env var and suppress any unnecessary console outputs.
flaky_jobs_to_sentry fails if the specs are not in the
spec/ dir. Modified the regex to not specifically look
for spec/ but rather just remove the ./ at the beginning
and [] at the end.
The rerun command for the flaky tests can be pretty big. This becomes
a bigger problem when there are more flaky tests appearing.
We allow users to skip the rerun command from the summry report by
setting the env variable `RSPECQ_REPORTER_RERUN_COMMAND_SKIP`. The location
and worker are still repoted.
The sentry payload remains the same.
Originally, a flaky test would appear in the report by its just example id
which is not helpful at all.
Then we merged 9f6816e which tried to replace the flaky example id with a
reproduction command. The command did not work however because the premise
was wrong. You see, to correctly reproduce a flaky test you need more than
the seed and the test file. You also need all the files the preceded the
flaky test, up to the point in time when you reset rspec and the databases.
For rspecq, this point in time is the start of a worker.
Both of the above issues are mitigiated:
1. Flaky tests are reported by their example location, which by itself does
not offer anything, but at least it shows the developer which test exactly
is flaky.
2. Provides a reproduction command for each flaky test, which is the gist.
The reproduction (or rerun) command, consists of the seed and the files originally tested by the worker when the test failed.
The new argument modifies how the rspecq worker publishes the files and
examples to the queue.
By detault it first merges individual examples into files and then
depending on whether the `update-timings` is passed, it will either
publish based on the timings or randomly shuffly them.
If the `reproduction` argument is passed, the worker will ignore timings and
randomness and will publish the files exactly as given.
This feature is very important in order for a developer to be able to
successfully reproduce a flaky test as reported by the reporter in a previous
run.
Cody Cutrer