Commit Graph

274 Commits

Author SHA1 Message Date
Adrian McCarthy 9e3bbb903d Add check for __main__ in dotest.py to ensure problems with multiprocessing on some Windows machines.
llvm-svn: 251549
2015-10-28 19:59:18 +00:00
Zachary Turner 06daa70385 Try to fix the linux buildbot.
It's complaining that it doesn't under the "import" command, so
I guess I need this hashbang at the beginning so that it knows
it's a Python script.

llvm-svn: 251544
2015-10-28 19:18:53 +00:00
Zachary Turner c432c8f856 Move lldb/test to lldb/packages/Python/lldbsuite/test.
This is the conclusion of an effort to get LLDB's Python code
structured into a bona-fide Python package.  This has a number
of benefits, but most notably the ability to more easily share
Python code between different but related pieces of LLDB's Python
infrastructure (for example, `scripts` can now share code with
`test`).

llvm-svn: 251532
2015-10-28 17:43:26 +00:00
Zachary Turner af383ff70c Preparation for turning lldbsuite into a Python package.
The idea behind this patch is to expose the meat of
LLDB's Python infrastructure (test suite, scripts, etc)
as a single package.  This makes reusability and code
sharing among sub-packages easy.

Differential Revision: http://reviews.llvm.org/D14131

llvm-svn: 251460
2015-10-27 22:33:47 +00:00
Zachary Turner 0a0490b152 Rename `lldb_shared` to `use_lldb_suite`.
llvm-svn: 251444
2015-10-27 20:12:05 +00:00
Pavel Labath e272b77120 Deprecate -m/+m dotest options in favor of test categories
Summary:
This change deprecates -m/+m dotest options (the options are still recognized but they print an
error message pointing to the new options) and adds a new lldb-mi test category instead. To just
run lldb-mi tests, use '-G lldb-mi'. To skip lldb-mi tests, use '--skip-category lldb-mi'. All
lldb-mi tests are marked as such using the getCategories method on the base MiTestCaseBase class
and the @lldbmi_test decorator is not needed. In case one still needs to annotate a specific test
function as an lldb-mi test, one can use the @add_test_categories(['lldb-mi']) decorator to
achieve that.

Reviewers: tfiala, dawn, ki.stfu, abidh

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D14060

llvm-svn: 251400
2015-10-27 09:34:34 +00:00
Zachary Turner cd236b8cc6 Python 3: Convert uses of `callable(x)` to `six.callable(x)`.
llvm-svn: 251329
2015-10-26 18:48:24 +00:00
Pavel Labath dc8b2d3d3a Port the python api decorator to use test categories
Summary:
Per discussions on the mailing list, I have implemented a decorator which annotates individual
test methods with categories. I have used this framework to replace the '-a' and '+a'
command-line switches (now '-G pyapi' and '--skip-category pyapi') and the @python_api_test
decorator (now @add_test_categories('pyapi')). The test suite now gives an error message
suggesting the new options if the user specifies the deprecated +/-a switches. If the general
direction is good, I will follow this up with other switches.

Reviewers: tberghammer, tfiala, granata.enrico, zturner

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D14020

llvm-svn: 251277
2015-10-26 09:28:32 +00:00
Zachary Turner 606e1e33ef Python3 - Wrap more statements in calls to list()
llvm-svn: 251129
2015-10-23 17:53:51 +00:00
Zachary Turner 35d017f0fc Add from __future__ import print_function everywhere.
Apparently there were tons of instances I missed last time, I
guess I accidentally ran 2to3 non-recursively.  This should be
every occurrence of a print statement fixed to use a print function
as well as from __future__ import print_function being added to
every file.

After this patch print statements will stop working everywhere in
the test suite, and the print function should be used instead.

llvm-svn: 251121
2015-10-23 17:04:29 +00:00
Zachary Turner 1c4059a0c4 Python 3 porting - Wrap returns from map() in list()
Under Python 2 this has no effect, since map() returns a list.
In Python 3 map() returns an iterable, so wrapping in a list is
necessary to keep the same semantics.

llvm-svn: 251060
2015-10-22 20:39:59 +00:00
Zachary Turner 7b24ff1509 Remove special case logic for finding 3rd party libs
llvm-svn: 251047
2015-10-22 19:55:18 +00:00
Zachary Turner 43a01e45f9 Use six to portably assign metaclasses in Python 2 and 3.
llvm-svn: 250859
2015-10-20 21:06:05 +00:00
Zachary Turner 8741e31876 Introduce a mechanism for reusing Python modules out of tree.
Right now our Python code does not all share a common root.  Tests and
scripts both contain python code that cannot take advantage of reusability
since they are unrelated siblings of each other.

In particular, this presents a problem for wanting to use third party
packages from both sides, since it does not make sense to copy the module
into both places.

This patch solves this by introducing a script lldb_shared.py which is a
very lightweight script that just searches up the tree until it finds a
root, and then imports a module from there.  That module knows how to
find all of the shared code that LLDB uses, and adjusts sys.path
accordingly to make them all visible.

llvm-svn: 250858
2015-10-20 21:05:57 +00:00
Zachary Turner ff890daf12 Convert print statements to print function calls.
This patch was generating by running `2to3` on the files in the
lldb/test directory.  This patch should be NFC, but it does
introduce the `from __future__ import print_function` line, which
will break future uses of the print statement.

llvm-svn: 250763
2015-10-19 23:45:41 +00:00
Enrico Granata a512103bdf Add a flakey category for flakey tests
llvm-svn: 250750
2015-10-19 22:53:34 +00:00
Pavel Labath 6204506fa6 dotest.py: Fail if we detect multiple tests with the same name
Summary:
Log files produced by dotest have names derived from the test name, and this produces errors in
case we have multiple tests with the same name. Additionally, it's good if the test name explains
what the test is testing (which it clearly doesn't do well if there are multiple tests with
identical names). This commit makes the presence of such tests a hard error.

Reviewers: tberghammer, zturner

Subscribers: iancottrell, lldb-commits

Differential Revision: http://reviews.llvm.org/D13588

llvm-svn: 249828
2015-10-09 12:48:17 +00:00
Tamas Berghammer 6f5d647c17 Fix the way dwo tests are skipped on darwin
We want to skip these tests when the target platform is darwin, not
when the host because they have to be enabled in case of darwin ->
android

llvm-svn: 249825
2015-10-09 12:06:07 +00:00
Pavel Labath 8112dd9951 Fix disabling of dwo tests
without this, dwo tests would be run even if they were meant to be disabled.

llvm-svn: 249819
2015-10-09 10:09:36 +00:00
Greg Clayton db2a576a77 Don't do DWO tests on darwin as they aren't supported. If this is an error and they are supported, we need to add support _only_ if the clang compiler that is being used supports the compiler option for DWO stuff as right now I just get an error saying "warning: unused option -gsplit-dwarf". So unless we can successfully test if "-gsplit-dwarf" is truly supported, I am going to disable it on darwin for now so we don't get test suite failures.
llvm-svn: 249625
2015-10-07 22:06:08 +00:00
Tamas Berghammer 4c0c7a7f52 Run tests with dwo symbol files
dwo symbol files are generated when code compiled with the "-gsplit-dwarf"
command option (https://gcc.gnu.org/wiki/DebugFission). This CL modifies
the test system to run tests with inferiors compile with the "-gsplit-dwarf"

Differential revision: http://reviews.llvm.org/D13300

llvm-svn: 249530
2015-10-07 10:02:17 +00:00
Todd Fiala ea594abc78 test framework: fixed issue when using results formatter with no formatter options
I broke the formatter options-passing parsing when no formatter
options are provided.

llvm-svn: 248397
2015-09-23 17:20:09 +00:00
Todd Fiala ea73624e5f Cleaned up results formatter options hand-off.
* --results-formatter-options renamed to --results-formatter-option,
  with short version of -O

* Multiple --results-formatter-option=OPTION can be specified.  The
  comma-separating mechanism has been removed.

* XunitFormatter options modified: -n and -r are now short forms of
  --ignore-skip-name and --ignore-skip-reason.  Those long option
  names were tweaked lightly.  They also can be specified multiple
  times on the command line.  The comma-separating, multiple-pattern-
  per-option mechanism has been removed.

One can now specify:

  dotest.py --results-file stdout -O-ndsym -O-nlldb-mi

for example, to ignore reporting skips for dsym-related or lldb-mi-related
tests in the xUnit report.

llvm-svn: 248384
2015-09-23 15:21:28 +00:00
Todd Fiala de9a44e3e9 test framework: parallel test runner sends terminate to formatter before printing to stdout
The parallel test runner now sends the terminate event to the formatter
(if there is one) after the parallel test runs but before dumping anything
to stdout/stderr at the end of the run.  This allows the existing
stdout/stderr summary reporting to co-exist nicely with a formatter like
the test_results.Curses that otherwise clobbers the screen.

llvm-svn: 248228
2015-09-22 00:15:50 +00:00
Todd Fiala 40b180e7ee test events: added optional value type to extra event key/val pairs
The test events had worker indexes coming across as strings.  I
want them to be ints.  worker_index now comes across as an int in
the dicationary.

The optional type can be specified with:
--event-add-entries key=val[:type][,key2=val2[:type2]...]
The type piece may be 'int' at this time.  That is all.  Otherwise
it will be a string.

llvm-svn: 248066
2015-09-18 23:46:30 +00:00
Todd Fiala e83f140833 test events: announce worker count in new initialize event
See the following for details:
http://reviews.llvm.org/D12987

llvm-svn: 248059
2015-09-18 22:45:31 +00:00
Todd Fiala 33896a9872 Adds parallel work queue index to test events, stdout/stderr results support.
See http://reviews.llvm.org/D12983 for details.

llvm-svn: 248036
2015-09-18 21:01:13 +00:00
Todd Fiala 68615ce606 Add JUnit/XUnit-formatted output to the lldb test run system
Also introduces the test event system into our test runner framework.
See the following for details:
http://reviews.llvm.org/D12831

llvm-svn: 247722
2015-09-15 21:38:04 +00:00
Ilia K dae2e564d8 Fix an AttributeError in dotest.py if --executable points to a wrong place
This patch fixes the following case:
```
  $ ./dotest.py --executable=~/p/llvm/build_ninja/bin/lldb tools/lldb-mi/
  '~/p/llvm/build_ninja/bin/lldb' is not a path to a valid executable
  Traceback (most recent call last):
    File "./dotest.py", line 1306, in <module>
      setupSysPath()
    File "./dotest.py", line 1004, in setupSysPath
      if not lldbtest_config.lldbExec:
  AttributeError: 'module' object has no attribute 'lldbExec'
```

And with this fix:
```
  $ ./dotest.py --executable=~/p/llvm/build_ninja/bin/lldb tools/lldb-mi/
  '~/p/llvm/build_ninja/bin/lldb' is not a path to a valid executable
  The 'lldb' executable cannot be located.  Some of the tests may not be run as a result.
```

llvm-svn: 247256
2015-09-10 09:24:43 +00:00
Todd Fiala a195adde00 Fix regressions in dotest.py when passing filters or directories.
See https://llvm.org/bugs/show_bug.cgi?id=24708 for details.
Using '-f FILTER' or unnamed arguments (directories) to dotest.py
will now force no-multiprocessing mode.  This stops a bombardment
of test output spam when following the instructions we provide
in a test session trace for rerunning the test.

llvm-svn: 247101
2015-09-08 23:22:19 +00:00
Todd Fiala 8cbeed3248 dotest.py ctrl-c support, addition of --test-runner-name option.
See http://reviews.llvm.org/D12651 for more details.

For the parallel test runner, -v now also implies --output-on-success.

llvm-svn: 247084
2015-09-08 22:22:33 +00:00
Zachary Turner b8dec76f26 Protect dotest.py with an if __name__ == "__main__"
llvm-svn: 246816
2015-09-03 23:03:16 +00:00
Todd Fiala fed95660f3 Roll dosep.py parallel test runner into dotest.py command line
See the following for details:
http://reviews.llvm.org/D12587

llvm-svn: 246794
2015-09-03 18:58:44 +00:00
Zachary Turner b1490b6172 Don't throw an exception when module cleanup fails.
Just because one test fails to clean up correctly, it should not
prevent other tests from attempting to run.

llvm-svn: 246063
2015-08-26 19:44:56 +00:00
Chaoren Lin bb2cc9cf56 Unique file names for TestStarted and TestFinished.
Summary: This fixes the issue of multiple test suites opening the same file for writing.

Reviewers: clayborg, zturner

Subscribers: lldb-commits

Differential Revision: http://reviews.llvm.org/D10284

llvm-svn: 239188
2015-06-05 21:08:24 +00:00
Vince Harron 8994fedc2b dotest.py - fixed a bug displaying usage
./dotest.py --help

llvm-svn: 238043
2015-05-22 19:49:23 +00:00
Vince Harron 47fc26f9fe Fix lldb executable discovery on OS X
It was broken by r237632

Differential Revision: http://reviews.llvm.org/D9846

llvm-svn: 237928
2015-05-21 18:21:07 +00:00
Vince Harron a7959f6585 trivial log print fix
llvm-svn: 237878
2015-05-21 05:57:28 +00:00
Vince Harron d347f27f45 dotest - minor doc fix
llvm-svn: 237665
2015-05-19 05:10:22 +00:00
Vince Harron 0c6160156a Make it possible to run dotest on Linux without any parameters
dotest will select clang-3.5 by default and fall back on clang/gcc
dotest will look for the lldb executable in the typical cmake
output locations:
{lldb}/../../../build/bin (next to llvm directory)
{lldb}/../../../build/host/bin (next to llvm directory)
{lldb}/../build/bin (next to lldb directory)
{lldb}/../build/host/bin (next to lldb directory)

llvm-svn: 237632
2015-05-18 23:07:18 +00:00
Vince Harron f8b9a1da2b Refactored dotest arg parser so that it's accessible from dosep
This allows dosep to understand an act on dotest arguments

Differential Revision: http://reviews.llvm.org/D9820

llvm-svn: 237602
2015-05-18 19:40:54 +00:00
Vince Harron f1518f5981 dotest - Refactored some code into loops
Renamed 'base' to 'lldbRootDirectory'
Renamed 'lldbPath' to 'lldbPythonDir'
Added cmake output directories to lldb executable search locations
Fixed invocation of lldb -P to ignore 'Cannot read termcap database'

Differential Revision: http://reviews.llvm.org/D9818

llvm-svn: 237601
2015-05-18 19:40:00 +00:00
Vince Harron 790d95cbec Refactored lldb executable name discovery
The lldb executable was referenced through the code by 7 different
(effectively) global variables.

global lldbExecutablePath
global lldbExecutable
os.environ['LLDB_EXEC']
os.environ['LLDB_TEST']
dotest.lldbExec
dotest.lldbHere
lldbtest.lldbExec

This change uses one global variable lldbtest_config.lldbExec to
replace them all.

Differential Revision: http://reviews.llvm.org/D9817

llvm-svn: 237600
2015-05-18 19:39:03 +00:00
Zachary Turner fb69dd8b64 Don't import lock on Windows.
lock imports fcntl, which doesn't exist on Windows.  If we need
to use this class on Windows, we will need to implement something
based on the CreateMutex API.

llvm-svn: 237292
2015-05-13 20:21:33 +00:00
Vince Harron 05df698d2b Darwin - fix intermitent crashes in import crashinfo
Summary:
"import crashinfo" was probably failing because multiple dotest invocations
are all trying to compile crashinfo.so at the same time.
    
I've put a mutex around the compile step to prevent this.
   
Reviewers: clayborg, chying
    
Subscribers: lldb-commits
    
Differential Revision: http://reviews.llvm.org/D9732

llvm-svn: 237230
2015-05-13 05:00:23 +00:00
Vince Harron d5fa102d44 We can't use sys.path[0] to determine the script directory because it doesn't work under a debugger
Test Plan: run tests with/without python debugger
Tested on OSX & Linux with PyCharm

Reviewers: chying, clayborg

Differential Revision: http://reviews.llvm.org/D9593

llvm-svn: 236957
2015-05-10 15:24:12 +00:00
Vince Harron 9753dd98b3 Add support for ./dotest.py --channel and --log-success
Summary:
New dotest options that allow arbitrary log channels and
categories to be enabled.  Also enables logging for locally run
debug servers.
    
Log messages are separated into separate files per test case.
(this makes it possible to log in dosep runs)
    
These new log files are stored side-by-side with trace files in the
session directory.
    
These files are deleted by default if the test run is successful.
    
If --log-success is specified, even successful logs are retained.
    
--log-success is useful for creating reference log files.
    
Test Plan:
add '--channel "lldb all" --channel "gdb-remote packets" --log-success'
to your dotest options

Tested on OSX and Linux
    
Differential Revision: http://reviews.llvm.org/D9594

llvm-svn: 236956
2015-05-10 15:22:09 +00:00
Adrian McCarthy c99d2fadb5 Don't attempt DSym tests on Windows.
llvm-svn: 236620
2015-05-06 18:19:51 +00:00
Chaoren Lin f62b7fc317 Fix typo. platfrom -> platform
llvm-svn: 236543
2015-05-05 20:34:36 +00:00
Robert Flack 068898cb83 Fix @skipUnlessPlatform and @skipIfPlatform decorators to work on classes.
The decorators to skip if or unless certain platforms relied on calling
skipTest, which only worked if they were decorating individual test cases.
However for decorating entire classes it needs different behavior. This behavior
is already encapsulated in unittest2.skipIf and unittest2.skipUnless so this
modifies the platform decorators to use these unittest skip decorators.

Test Plan:
./dotest.py -t -p TestObjCMethods2.py
See skipped tests on linux, and see tests run on macosx.

Differential Revision: http://reviews.llvm.org/D8903

llvm-svn: 234514
2015-04-09 18:07:58 +00:00