This allows bots which haven't updated to pass in --filecheck to
dotest.py to run more tests. FileCheck-dependent tests will continue to
fail.
Differential Revision: https://reviews.llvm.org/D53175
llvm-svn: 344401
Add a "filecheck" method to the LLDB test base. This allows test authors
to pattern match command output using FileCheck, making it possible to
write stricter tests than what `self.expect` allows.
For context (motivation, examples of stricter checking, etc), see the
lldb-dev thread: "Using FileCheck in lldb inline tests".
Differential Revision: https://reviews.llvm.org/D50751
llvm-svn: 342508
The single-process test runner is invoked in a number of different
scenarios, including when multiple test dirs are specified or (afaict)
when lit is used to drive the test suite.
Unfortunately the --test-subdir option did not work with the single
process test runner, breaking an important use case (using lit to run
swift-lldb Linux tests):
Failure URL: https://ci.swift.org/job/swift-PR-Linux/6841
We won't be able to run lldb tests within swift PR testing without
filtering down the set of tests.
This change makes --test-subdir work with the single-process runner.
llvm-svn: 339929
This patch adds a new lldb-vscode tool that speaks the Microsoft Visual Studio Code debug adaptor protocol. It has full unit tests that test all packets.
This tool can be easily packaged up into a native extension and used with Visual Studio Code, and it can also be used by Nuclide
Differential Revision: https://reviews.llvm.org/D50365
llvm-svn: 339911
On systems where it's not supported.
As far as I understand Linux is the only systems which now ships
with libstdcxx (maybe NetBSD?, but I'm not entirely sure of the
state of lldb on the platform).
We could make this more fine grained looking for the header as
we do for libcxx. This is a little tricky as there's no such
thing as /usr/include/c++/v1, but libstdcxx encodes the version
number in the path (i.e. /usr/include/c++/5.4). I guess we might
match a regex, but it seems fragile to me.
Differential Revision: https://reviews.llvm.org/D49110
llvm-svn: 336724
Summary:
1) In TestLinuxCore rather than skipping the tests on Windows, mark them as expected failures and add a bug reference
2) In dotest.py replace the undefined property in the exceptions with the actual property causing the exception
Reviewers: asmith, labath, zturner
Reviewed By: labath, zturner
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D46590
llvm-svn: 331886
Summary:
In the magic test duplicator, we were making the decision whether to
create a test variant based on the compiler and the target platform.
This meant that the set of known tests was different for each test
configuration.
This patch makes the set of generated test variants static and handles
the skipping via runtime checks instead. This is more consistent with
how we do other test-skipping decision (e.g. for libc++ tests), and
makes it easier to expose the full set of tests to lit, which now does
not need to know anything about what things can potentially cause tests
to appear or disappear.
Reviewers: JDevlieghere, aprantl
Subscribers: eraman, lldb-commits
Differential Revision: https://reviews.llvm.org/D45949
llvm-svn: 330708
Summary:
With the upstream implementation of dsymutil containing almost all
functionality from the one shipped with Xcode, we want to use the
in-tree version for running the test suite.
This will also allow us to re-enable TestUnicodeSymbols which was
failing because of the discrepancy in how Unicode symbols were hashed in
lldb and older versions of dsymutil.
Reviewers: aprantl, davide, jingham, labath
Subscribers: mgorny, llvm-commits, lldb-commits
Differential Revision: https://reviews.llvm.org/D45518
llvm-svn: 329889
The test "test_fp_special_purpose_register_read" in TestRegisters.py
fails on Darwin machines configured to use an out-of-tree debugserver.
The error message is: 'register read ftag' returns expected result, got
'ftag = 0x80'. This indicates that the debugserver in use is too old.
This commit introduces a decorator which can be used to skip tests which
rely on having a just-built debugserver. This resolves the issue:
$ ./bin/llvm-dotest -p TestRegisters.py -v
1 out of 617 test suites processed - TestRegisters.py
Test Methods: 7
Success: 6
Skip: 1
...
llvm-svn: 327052
LLDB queries Spotlight to locate .dSYM bundles based on the UUID
embedded in a binary, and because the UUID is a hash of filename and
.text section, there *will* be conflicts inside $BUILD_DIR.
This should fix the broken green dragon bots.
llvm-svn: 324019
This patch is the result of a discussion on lldb-dev, see
http://lists.llvm.org/pipermail/lldb-dev/2018-January/013111.html for
background.
For each test (should be eventually: each test configuration) a
separate build directory is created and we execute
make VPATH=$srcdir/path/to/test -C $builddir/path/to/test -f $srcdir/path/to/test/Makefile -I $srcdir/path/to/test
In order to make this work all LLDB tests need to be updated to find
the executable in the test build directory, since CWD still points at
the test's source directory, which is a requirement for unittest2.
Although we have done extensive testing, I'm expecting that this first
attempt will break a few bots. Please DO NOT HESITATE TO REVERT this
patch in order to get the bots green again. We will likely have to
iterate on this some more.
Differential Revision: https://reviews.llvm.org/D42281
llvm-svn: 323803
Now the function returns `False`, and the caller can take the
appropriate action (in this case, for --executable, reporting
that the file doesn't exist).
llvm-svn: 323566
Summary:
This used to be important when all tests were run in a single process,
but that has no longer been the case for a while. Furthermore, this hook fails
to build on new mac versions for several people, and it's not clear
whether fixing it is worth the effort.
Reviewers: jingham, clayborg, davide
Subscribers: lldb-commits
Differential Revision: https://reviews.llvm.org/D41871
llvm-svn: 322167
The commit breaks the case where you specify just a filename to the
compiler. Previously, it would look up the compiler in your path, now it
complains that the compiler is not found. One of the lldb buildbots is
depending on this. It seems like a nice feature to have, as it means
less typing and being able to avoid hard-coding the system compiler path
in the bot config.
This reverts commit r316393.
llvm-svn: 316451
If you pass an invalid compiler/debugger path on the cmdline to `dotest.py` this is what you get.
Traceback (most recent call last):
[...]
File "dotest.py", line 7, in <module>
lldbsuite.test.run_suite()
[...]
And with the patch applied:
/home/davide/work/build-lldb/bin/clandasfasg is not a valid path, exiting
Differential Revision: https://reviews.llvm.org/D39199
llvm-svn: 316393
Summary:
This aims to replace the different decorators we've had on each libc++
test with a single solution. Each libc++ will be assigned to the
"libc++" category and a single central piece of code will decide whether
we are actually able to run libc++ test in the given configuration by
enabling or disabling the category (while giving the user the
opportunity to override this).
I started this effort because I wanted to get libc++ tests running on
android, and none of the existing decorators worked for this use case:
- skipIfGcc - incorrect, we can build libc++ executables on android
with gcc (in fact, after this, we can now do it on linux as well)
- lldbutil.skip_if_library_missing - this checks whether libc++.so is
loaded in the proces, which fails in case of a statically linked
libc++ (this makes copying executables to the remote target easier to
manage).
To make this work I needed to split out the pseudo_barrier code from the
force-included file, as libc++'s atomic does not play well with gcc on
linux, and this made every test fail, even though we need the code only
in the threading tests.
So far, I am only annotating one of the tests with this category. If
this does not break anything, I'll proceed to update the rest.
Reviewers: jingham, zturner, EricWF
Subscribers: srhines, lldb-commits
Differential Revision: https://reviews.llvm.org/D30984
llvm-svn: 299028
r298203 make SBPlatform::MakeDirectory less recursive, which breaks the
test suite creation of test directory hierarchy creation on the remote
target. Since the function was never fully recursive, and the name does
not imply recursiveness, I fix the problem by modifying the test runner
to do the recursion manually.
I also make the runner complain more loudly when it fails to create the
directory -- previously it just printed the error to stdout and caused
most of the tests to hang, which is not very helpful in diagnosing the
problem.
llvm-svn: 298261
dotest.py script doesn't validate executables passed on the command line
before spawning dozens of subprocesses, all of which fail silently,
leaving an empty results file.
We should validate the lldb and compiler executables on
configuration, aborting when given invalid paths, to prevent numerous,
cryptic, and spurious failures.
<rdar://problem/31117272>
llvm-svn: 298111
Summary:
This has been broken at least since the new test result framework was
added, which was over a year ago. It looks like nobody has missed it
since.
Removing this makes the gmodules handling code saner, as it already did
not know how to handle the multiple-compilers case.
My motivation for this is libc++ data formatters support on android -- I
am trying make a central way of determining whether libc++ tests can be
run, and without this, I would have to resort to similar hacks as the
gmodules code.
Reviewers: jingham, zturner
Subscribers: danalbert, tfiala, lldb-commits
Differential Revision: https://reviews.llvm.org/D30779
llvm-svn: 297811
This patch adds support to the test suite for overriding the path to debugserver, and uses the override to point to the build tree's debugserver on Darwin.
llvm-svn: 297776
Summary:
dotest.py has a framework option that is not respected. This patch makes the framework path properly configurable via the --framework option.
This patch also adds a function to the lldbtest.Base class named "hasDarwinFramework" which allows us to not rely on the host platform to determine if a framework is present. If running on Darwin, and not building a framework, this will follow the *nix code paths which are appropriate for Darwin.
Reviewers: tfiala
Subscribers: lldb-commits, mgorny
Differential Revision: https://reviews.llvm.org/D25886
llvm-svn: 285541
Summary: Not everyone names their code sign identity "lldb_codesign", so it is nice to allow this to be overridden.
Reviewers: zturner, tfiala
Subscribers: labath, mgorny, lldb-commits
Differential Revision: https://reviews.llvm.org/D25714
llvm-svn: 284893
Summary:
The test suite calls realpath on the lldb executable then append "-mi" to it to find the path of the lldb-mi executable. This does not work when using CMake builds on *nix platforms. On *nix platforms when a version number is set on executables CMake generates the binary as ${name}-${version} with a symlink named ${name} pointing to it.
This results in the lldb executable being named lldb-4.0.0, and since lldb-4.0.0-mi doesn't ever match the lldb-mi executable these tests are always disabled.
This patch looks for lldb-mi in the same directory as lldb.
Reviewers: zturner, tfiala
Subscribers: ki.stfu, enlight, lldb-commits
Differential Revision: https://reviews.llvm.org/D25486
llvm-svn: 284041
Summary:
This patch is necessary because individual test cases are not required
to have unique names. Therefore, test cases must now
be specified explicitly in the form <TestCase>.<TestMethod>.
Because it works by regex matching, passing just <TestCase> will
still disable an entire file.
This also allows for multiple exclusion files to be specified.
Reviewers: zturner, labath, jingham, tfiala
Subscribers: lldb-commits, sas
Differential Revision: https://reviews.llvm.org/D24988
llvm-svn: 283238
Summary:
The current implementation of the test suite allows the user to run
a certain subset of tests using '-p', but does not allow the inverse,
where a user wants to run all but some number of known failing tests.
Implement this functionality.
Reviewers: labath, zturner, tfiala
Subscribers: jingham, sas, lldb-commits
Differential Revision: https://reviews.llvm.org/D24629
llvm-svn: 282298
*** to conform to clang-format’s LLVM style. This kind of mass change has
*** two obvious implications:
Firstly, merging this particular commit into a downstream fork may be a huge
effort. Alternatively, it may be worth merging all changes up to this commit,
performing the same reformatting operation locally, and then discarding the
merge for this particular commit. The commands used to accomplish this
reformatting were as follows (with current working directory as the root of
the repository):
find . \( -iname "*.c" -or -iname "*.cpp" -or -iname "*.h" -or -iname "*.mm" \) -exec clang-format -i {} +
find . -iname "*.py" -exec autopep8 --in-place --aggressive --aggressive {} + ;
The version of clang-format used was 3.9.0, and autopep8 was 1.2.4.
Secondly, “blame” style tools will generally point to this commit instead of
a meaningful prior commit. There are alternatives available that will attempt
to look through this change and find the appropriate prior commit. YMMV.
llvm-svn: 280751
The variables referenced in the print message are not defined. Simply state
that the requisite script is not found. Correct grammar to indicate that the
tests are rather likely to fail rather than unlikely to fail.
llvm-svn: 269628
The parameter here is a list, not a string. Ensure that the we splat the list
into arguments prior to invoke os.path.join. This would previously fail with a
`startswith` is not a member of `list`.
llvm-svn: 269627
Summary:
This change enhances the LLDB test infrastructure to convert
load-time exceptions in a given Python test module into errors.
Before this change, specifying a non-existent test decorator,
or otherwise having some load-time error in a python test module,
would not get flagged as an error.
With this change, typos and other load-time errors in a python
test file get converted to errors and reported by the
test runner.
This change also includes test infrastructure tests that include
covering the new work here. I'm going to wait until we have
these infrastructure tests runnable on the main platforms before
I try to work that into all the normal testing workflows.
The test infrastructure tests can be run by using the standard python module testing practice of doing the following:
cd packages/Python/lldbsuite/test_event
python -m unittest discover -s test/src -p 'Test*.py'
Those tests run the dotest inferior with a known broken test and verify that the errors are caught. These tests did not pass until I modified dotest.py to capture them properly.
@zturner, if you have the chance, if you could try those steps above (the python -m unittest ... line) on Windows, that would be great if we can address any python2/3/Windows bits there. I don't think there's anything fancy, but I didn't want to hook it into test flow until I know it works there.
I'll be slowly adding more tests that cover some of the other breakage I've occasionally seen that didn't get collected as part of the summarization. This is the biggest one I'm aware of.
Reviewers: zturner, labath
Subscribers: zturner, lldb-commits
Differential Revision: http://reviews.llvm.org/D20193
llvm-svn: 269489
Summary:
lldb-server tests are currently being skipped on the
check-lldb target. This is because we get the path of
lldb-server by modifying the path to the lldb executable.
However, by this point, we've changed directories, and a
relative path to the build/bin directory will no longer point
to the location of lldb-server.
Storing an absolute path solves this issue.
Reviewers: vharron, zturner, tfiala, labath
Subscribers: labath, lldb-commits, sas
Differential Revision: http://reviews.llvm.org/D19082
llvm-svn: 267463
This change moves all the test event handling and its related
ResultsFormatter classes out of the packages/Python/lldbsuite/test dir
into a packages/Python/lldbsuite/test_event package. Formatters are
moved into a sub-package under that.
I am limiting the scope of this change to just the motion and a few
minor issues caught by a static Python checker (e.g. removing unused
import statements).
This is a pre-step for adding package-level tests to the test event
system. I also intend to simplify test event results formatter selection
after I make sure this doesn't break anybody.
See:
http://reviews.llvm.org/D19288
Reviewed by:
Pavel Labath
llvm-svn: 266885
Win32 API calls that are Unicode aware require wide character
strings, but LLDB uses UTF8 everywhere. This patch does conversions
wherever necessary when passing strings into and out of Win32 API
calls.
Patch by Cameron
Differential Revision: http://reviews.llvm.org/D17107
Reviewed By: zturner, amccarth
llvm-svn: 264074
Nobody seems to know what purpose these files serve, yet they were accumulating by the thousands in the test traces directory. I'm proposing we delete them.
Creating these files accounted for about 2.5% of the time to run ninja check-lldb on my machine, which isn't a lot, but it's something.
llvm-svn: 263122
This is meant to reduce the typing that one needs to do to get from the test subdirectory to actual test cases. Now one can just do
$ ./dotest.py ./testcases/<yaddayaddayadda>
llvm-svn: 255741
Use of --rerun-all-issues will enable any test method failure, not just
test methods marked with the flakey decorator, to rerun.
Currently this does not change the flakey logic's immediate rerun
attempt. I want to make sure this doesn't cause any significant issues
before changing that part.
The rerun reporting is only known to work properly with the
default (new) BasicResultsFormatter reporting. Once we work out
any issues, I'll go back and make sure the curses output handles
it properly as well.
llvm-svn: 255543
Also adds full path info for exceptional exits and timeouts when
no test method is currently running.
Adds --rerun-all-issues command line arg. If specified, all
test issues are eligible for rerun. If not specified, only tests
marked flakey are eligible for rerun.
The actual rerunning will occur in an upcoming change. This
change just handles tha accounting of what should be rerun.
llvm-svn: 255438
Vedant Kumar