Right now if the test suite encounters a cleanup error it just prints "CLEANUP
ERROR:" but not any additional information.
This patch just prints the exception that caused the cleanup error. This should
make debugging the failing tests for D83865 easier (and seems in general nice to
have).
Reviewed By: labath
Differential Revision: https://reviews.llvm.org/D83874
After more investigation, I realised this part of the code is totally
unused. It was used for communicating the test results from the
"inferior" dotest process to the main "dosep" process running
everything. Now that everything is being orchestrated through lit, this
is not used for anything.
There already are decorators and "--excluded" option to mark test-cases/files
as expected to fail. However, when a new test file is added and it which relates
to a feature that a target doesn't support, this requires either adding decorators
to that file or modifying the file provided as "--excluded" option value.
The purpose of this patch is to avoid any modifications in such cases.
E.g. if a target doesn't support "watchpoints" and passes "--xfail-category watchpoint"
to dotest, a testing job will not fail after a new watchpoint-related test file is added.
Differential Revision: https://reviews.llvm.org/D71906
This is needed to not re-write parent's categories by categories of a nested folder,
e.g. commands/expression/completion specify "cmdline" category, however it still belongs
to parent's "expression" category.
The sentinel ".categories" in the test-suite root directory is no longer needed.
Differential Revision: https://reviews.llvm.org/D71905
Summary:
A lot of tests do this trick but the vast majority of them don't even call `print()`.
Most of this patch was generated by a script that just looks at all the files and deletes the line if there is no `print (` or `print(` anywhere else in the file.
I checked the remaining tests manually and deleted the import if we never call print (but instead do stuff like `expr print(...)` and similar false-positives).
I also corrected the additional empty lines after the import in the files that I manually edited.
Reviewers: JDevlieghere, labath, jfb
Reviewed By: labath
Subscribers: dexonsmith, wuzish, nemanjai, kbarton, christof, arphaman, abidh, lldb-commits
Tags: #lldb
Differential Revision: https://reviews.llvm.org/D71452
Summary:
When creating a test with `lldbinline.MakeInlineTest()`, the reported `inspect.getfile(test.__class__)` is `lldbtest.pyc`, meaning any `.categories` file will be ineffective for those tests. Check for the test_filename first, which inline tests will set.
Additionally, raise an error with the starting dir if `.categories` is not found. This makes the problem more obvious when it occurs: when the test is separated from the test framework tree.
Reviewers: labath, JDevlieghere
Subscribers: lldb-commits
Tags: #lldb
Differential Revision: https://reviews.llvm.org/D71099
This patch removes the -q (quiet) flag and changing the default
behavior. Currently the flag serves two purposes that are somewhat
contradictory, as illustrated by the difference between the argument
name (quiet) and the configuration flag (parsable). On the one hand it
reduces output, but on the other hand it prints more output, like the
result of individual tests. My proposal is to guard the extra output
behind the verbose flag and always print the individual test results.
Differential revision: https://reviews.llvm.org/D66837
llvm-svn: 370226
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
Summary:
This used to be important when all tests were run in a single process,
but that has no longer been the case for a while. Furthermore, this hook fails
to build on new mac versions for several people, and it's not clear
whether fixing it is worth the effort.
Reviewers: jingham, clayborg, davide
Subscribers: lldb-commits
Differential Revision: https://reviews.llvm.org/D41871
llvm-svn: 322167
Summary:
This mechanism was mostly redundant with the file-based .categories
mechanism, and it was interfering with it, as any test which implemented
a getCategories method would not inherit the filesystem categories.
This patch removes it. The existing categories are preserved either by
adding a .categories file, or using the @add_test_categories decorator.
Reviewers: jingham, clayborg, zturner
Subscribers: lldb-commits
Differential Revision: https://reviews.llvm.org/D39515
llvm-svn: 317277
Summary:
This patch is necessary because individual test cases are not required
to have unique names. Therefore, test cases must now
be specified explicitly in the form <TestCase>.<TestMethod>.
Because it works by regex matching, passing just <TestCase> will
still disable an entire file.
This also allows for multiple exclusion files to be specified.
Reviewers: zturner, labath, jingham, tfiala
Subscribers: lldb-commits, sas
Differential Revision: https://reviews.llvm.org/D24988
llvm-svn: 283238
Summary:
The current implementation of the test suite allows the user to run
a certain subset of tests using '-p', but does not allow the inverse,
where a user wants to run all but some number of known failing tests.
Implement this functionality.
Reviewers: labath, zturner, tfiala
Subscribers: jingham, sas, lldb-commits
Differential Revision: https://reviews.llvm.org/D24629
llvm-svn: 282298
*** to conform to clang-format’s LLVM style. This kind of mass change has
*** two obvious implications:
Firstly, merging this particular commit into a downstream fork may be a huge
effort. Alternatively, it may be worth merging all changes up to this commit,
performing the same reformatting operation locally, and then discarding the
merge for this particular commit. The commands used to accomplish this
reformatting were as follows (with current working directory as the root of
the repository):
find . \( -iname "*.c" -or -iname "*.cpp" -or -iname "*.h" -or -iname "*.mm" \) -exec clang-format -i {} +
find . -iname "*.py" -exec autopep8 --in-place --aggressive --aggressive {} + ;
The version of clang-format used was 3.9.0, and autopep8 was 1.2.4.
Secondly, “blame” style tools will generally point to this commit instead of
a meaningful prior commit. There are alternatives available that will attempt
to look through this change and find the appropriate prior commit. YMMV.
llvm-svn: 280751
Summary:
LLDBTestResult.hardMarkAsSkipped marked the whole class as skipped when the first class in the
test failed the category check. This meant that subsequent tests in the same class did not run
even if they were passing the category filter. Fix that.
Reviewers: tfiala
Subscribers: lldb-commits
Differential Revision: http://reviews.llvm.org/D22213
llvm-svn: 275173
Summary:
print build errors nicely in test output
This test infrastructure change adds a new Python exception
for test subject builds that fail. The output of the build
command is captured and propagated to both the textual test
output display code and to the test event system.
The ResultsFormatter objects have been modified to do something
more useful with this information. The xUnit formatter
now replaces the non-informative Python build error stacktrace
with the build error content. The curses ResultsFormatter
prints a 'B' for build errors rather than 'E'.
The xUnit output, in particular, makes it much easier for
developers to track down test subject build errors that cause
test failures when reports come in from CI.
Reviewers: granata.enrico
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D20252
llvm-svn: 269525
This change moves all the test event handling and its related
ResultsFormatter classes out of the packages/Python/lldbsuite/test dir
into a packages/Python/lldbsuite/test_event package. Formatters are
moved into a sub-package under that.
I am limiting the scope of this change to just the motion and a few
minor issues caught by a static Python checker (e.g. removing unused
import statements).
This is a pre-step for adding package-level tests to the test event
system. I also intend to simplify test event results formatter selection
after I make sure this doesn't break anybody.
See:
http://reviews.llvm.org/D19288
Reviewed by:
Pavel Labath
llvm-svn: 266885
Summary:
Previously the add_test_categories would simply overwrite the current set of categories for a
method. This change makes the decorator truly "add" categories, by extending the current set of
categories instead of replacing it.
To do this, I have:
- replaced the getCategories() property on a method (which was itself a method), with a simple
list property "categories". This makes add_test_categories easier to implement, and test
categories isn't something which should change between calls anyway.
- rewritten the getCategoriesForTest function to merge method categories with the categories of
the test case. Previously, it would just use the method categories if they were present. I have
also greatly simplified this method. Originally, it would use a lot of introspection to enable
it being called on various types of objects. Based on my tests, it was only ever being called
on a test case. The new function uses much less introspection then the preivous one, so we
should easily catch any stray uses, if there are any, as they will generate exceptions now.
Reviewers: zturner, tfiala, tberghammer
Subscribers: lldb-commits
Differential Revision: http://reviews.llvm.org/D15451
llvm-svn: 255493
This will be used in a future change to support rerunning flakey tests
that hit a test result isue in a low-load, single worker test runner phase.
This is implemented as an additive-style event rather than being
evaluated and added to the start_test event because the decorator code
only runs after the start_test event is created and sent. i.e.
LLDBTestResult.startTest() runs before the test method decorators run.
llvm-svn: 255351
This was an option to display a graphical progress bar. Nobody
is using this, and it doesn't work correctly anyway with the new
result formatter.
llvm-svn: 255153
Raphael Isemann