[lldb] Add more documentation on test variants

This formalizes some of the discussion on D121631.

Differential Revision: https://reviews.llvm.org/D121900
This commit is contained in:
Pavel Labath 2022-03-17 11:41:10 +01:00
parent 0f362f9d25
commit c7fc7205bb
1 changed files with 16 additions and 4 deletions

View File

@ -185,10 +185,22 @@ of variants. It's very tempting to add more variants because it's an easy way
to increase test coverage. It doesn't scale. It's easy to set up, but increases
the runtime of the tests and has a large ongoing cost.
The key take away is that the different variants don't obviate the need for
focused tests. So relying on it to test say DWARF5 is a really bad idea.
Instead you should write tests that check the specific DWARF5 feature, and have
the variant as a nice-to-have.
The test variants are most useful when developing a larger feature (e.g. support
for a new DWARF version). The test suite contains a large number of fairly
generic tests, so running the test suite with the feature enabled is a good way
to gain confidence that you haven't missed an important aspect. However, this
genericness makes them poor regression tests. Because it's not clear what a
specific test covers, a random modification to the test case can make it start
(or stop) testing a completely different part of your feature. And since these
tests tend to look very similar, it's easy for a simple bug to cause hundreds of
tests to fail in the same way.
For this reason, we recommend using test variants only while developing a new
feature. This can often be done by running the test suite with different
arguments -- without any modifications to the code. You can create a focused
test for any bug found that way. Often, there will be many tests failing, but a
lot of then will have the same root cause. These tests will be easier to debug
and will not put undue burden on all other bots and developers.
In conclusion, you'll want to opt for an API test to test the API itself or
when you need the expressivity, either for the test case itself or for the