ensure we pick up the other location style when queueing up reruns
and when deciding not to re-run changed files
test plan:
1. see examples that were rerun in selenium build on PS 2
2. see modified examples that were not rerun in rspec build on PS 2
Change-Id: I31e8f4b8cc0ef8a0cc6cb2d5e8b66d69d1e931c0
Reviewed-on: https://gerrit.instructure.com/109288
Tested-by: Jenkins
Reviewed-by: Landon Wilkins <lwilkins@instructure.com>
Product-Review: Landon Wilkins <lwilkins@instructure.com>
QA-Review: Landon Wilkins <lwilkins@instructure.com>
don't re-run all shared_examples when just one fails, and don't rerun
otherwise colliding example locations
also ensure the various formatters output a unique/correct rerun argument
for any such failing example
test plan:
1. see intentional patchset 1 failures and corresponding output and error
pages
2. see rerun failures on this patchset (if any)
Change-Id: I3f40b576f9f78bd9541cf9934ef348724fa815e6
Reviewed-on: https://gerrit.instructure.com/109235
Tested-by: Jenkins
Reviewed-by: Landon Wilkins <lwilkins@instructure.com>
Product-Review: Landon Wilkins <lwilkins@instructure.com>
QA-Review: Landon Wilkins <lwilkins@instructure.com>
include the line number of the not-rerunnable spec for easier tracking in
janky
test plan:
see intentional failures and rerun behavior in patchset 2
Change-Id: I3c381f94c71dc082a83f0f025794dc135b0336b4
Reviewed-on: https://gerrit.instructure.com/107760
Tested-by: Jenkins
Reviewed-by: Landon Wilkins <lwilkins@instructure.com>
Product-Review: Landon Wilkins <lwilkins@instructure.com>
QA-Review: Landon Wilkins <lwilkins@instructure.com>
ensure we discard and stop recording for skipped/passed examples. also
don't even start recording once we hit the threshold
in the process, refactor a bit so it's more clear what's going on
Change-Id: Iea7ad53b2e74d76bdfe26dee73ef57bc4d956a8d
Reviewed-on: https://gerrit.instructure.com/102539
Tested-by: Jenkins
Reviewed-by: Landon Wilkins <lwilkins@instructure.com>
Product-Review: Landon Wilkins <lwilkins@instructure.com>
QA-Review: Landon Wilkins <lwilkins@instructure.com>
reimplement it using formatters. notable changes:
1. the index page has separate sections for "Initial", "Rerun 1", etc.
2. you now get error pages for before(:all) failures
3. you now get error pages for non-selenium failures
4. you now get links to pages for failures prior to the current one
5. in addition to not capturing screenshots/videos on too-many-failures,
we now stop making error pages if there are too many, so that we don't
fill up the workspace on mass badness
test plan:
1. see build in repo-who-shall-not-be-named
Change-Id: I54f065f1515f2b639b3f90b471a19e434a17aece
Reviewed-on: https://gerrit.instructure.com/102002
Tested-by: Jenkins
Reviewed-by: Landon Wilkins <lwilkins@instructure.com>
Product-Review: Landon Wilkins <lwilkins@instructure.com>
QA-Review: Landon Wilkins <lwilkins@instructure.com>
the idea is to prevent people from making specs slower, while
grandfathering in existing specs
basically:
1. if it's a spec we've seen before, don't let it run much slower than
usual ... basically however long it took in the post-merge, with a
very generous fudge factor
2. if it's a new spec in this commit, enforce an aggressive timeout (10
sec, but configurable via env var)
3. regardless, don't let any spec take more than 60 seconds (not new, but
now configurable via env var) ... since local testing typically won't
have stats available, this will come into play
Change-Id: I4d8de97ccbef324ffde5b04de78db39e915b7e8a
Reviewed-on: https://gerrit.instructure.com/100379
Tested-by: Jenkins
Reviewed-by: Landon Wilkins <lwilkins@instructure.com>
Product-Review: Landon Wilkins <lwilkins@instructure.com>
QA-Review: Landon Wilkins <lwilkins@instructure.com>
Rob Orton