Commit Graph

48 Commits

Author SHA1 Message Date
Jonas Devlieghere f6c870a794 [test] Fix --framework argument passed to dotest.
The framework argument was broken when I removed the generator
expressions upstream.  I replaced $<TARGET_FILE_DIR:liblldb> with
${LLVM_LIBRARY_OUTPUT_INTDIR}) which is not correct.

rdar://40534649

llvm-svn: 333412
2018-05-29 12:30:27 +00:00
Jonas Devlieghere 52721286b2 [CMake] Unify and relayer testing
This patch restructures part of LLDB's testing configuration:

1. I moved the test dependencies up the chain so every dotest dependency
   becomes a lit dependency as well. It wouldn't make sense for dotest to
   have other dependencies when it's being run by lit. Lit on the other
   hand can still specify extra dependencies.

2. I replaced as much generator expressions with variables as possible.
   This is consistent with the rest of LLVM and doesn't break generators
   that support multiple targets (MSVC, Xcode). This wasn't a problem
   before, but now we need to expand the dotest arguments in the lit
   configuration and there's only one test suite even with multiple
   targets.

3. I moved lldb-dotest into it's own directory under utils since there's
   no need anymore for it to located under `test/`.

Differential revision: https://reviews.llvm.org/D46334

llvm-svn: 331463
2018-05-03 16:54:10 +00:00
Jonas Devlieghere 5a3c9ee93d Revert "[lit] Replace generator expressions in lit.site.cfg"
Using GENERATE breaks generators that support multiple configurations,
e.g. MSVC. Reverting for now until we find a better solution.

llvm-svn: 331285
2018-05-01 17:08:09 +00:00
Jonas Devlieghere 42b8d26414 [lit] Replace generator expressions in lit.site.cfg
The lit site configuration for the test suite can contain generator
expressions such as $<TARGET_FILE:debugserver> that need to be
substituted.

llvm-svn: 331277
2018-05-01 16:19:48 +00:00
Jonas Devlieghere 0d5c5936cb [CMake] Gate 'dsymutil' dependency
Only add `dsymutil` as a test dependency when this is not a stand-alone
build (LLDB_BUILT_STANDALONE).

llvm-svn: 330568
2018-04-23 08:44:06 +00:00
Jonas Devlieghere 49f4aee0e3 [lit] Generate a single lit cfg file for tests that require dotest.py
The current way that the lit configuration is generated for the LLDB
tests that run using dotest causes cmake to fail when using a generator
which supports multiple configurations (such as Visual Studio). The
failure is because file GENERATE will create a file *per possible
configuration* resulting in the same lit configuration file being
overwritten multiple times.

To fix the issue, we need to create a single lit file that is agnostic
of the configurations and can be used for any configuration.

Patch by: Stella Stamenova

Differential revision: https://reviews.llvm.org/D45918

llvm-svn: 330518
2018-04-21 18:23:04 +00:00
Jonas Devlieghere 1a928f3fc7 [LIT] Have lit run the lldb test suite
This is the first in what will hopefully become a series of patches to
replace the driver logic in dotest.py with LIT. The motivation for this
change is that there's no point in maintaining two driver
implementations. Since all of the LLVM projects are using lit, this is
the obvious choice.

Obviously the goal is maintain full compatibility with the functionality
offered by dotest. As such we won't be removing anything until that
point has been reached.

This patch is the initial attempt (referred to as v1) to run the lldb
test suite with lit. To do so we introduced a custom LLDB test format
that invokes dotest.py with a single test file.

Differential revision: https://reviews.llvm.org/D45333

llvm-svn: 330275
2018-04-18 17:08:49 +00:00
Jonas Devlieghere 1bf22e7722 [dotest] Use in-tree dsymutil on Darwin
Summary:
With the upstream implementation of dsymutil containing almost all
functionality from the one shipped with Xcode, we want to use the
in-tree version for running the test suite.

This will also allow us to re-enable TestUnicodeSymbols which was
failing because of the discrepancy in how Unicode symbols were hashed in
lldb and older versions of dsymutil.

Reviewers: aprantl, davide, jingham, labath

Subscribers: mgorny, llvm-commits, lldb-commits

Differential Revision: https://reviews.llvm.org/D45518

llvm-svn: 329889
2018-04-12 09:25:32 +00:00
Jonas Devlieghere ecc8aee652 [dotest] Use subprocess.call to forward arguments in wrapper
As suggested by Pavel on lldb-commits. Originally I picked os.system
because it was so much more simple than the subprocess module, but that
no longer holds true after yesterday's hack in r328020. This is what it
should've been in the first place.

Differential revision: https://reviews.llvm.org/D44728

llvm-svn: 328089
2018-03-21 11:13:56 +00:00
Pavel Labath 63047d05c0 [dotest] Clean up test folder clean-up
Summary:
This patch implements a unified way of cleaning the build folder of each
test. This is done by completely removing the build folder before each
test, in the respective setUp() method. Previously, we were using a
combination of several methods, each with it's own drawbacks:
- nuking the entire build tree before running dotest: the issue here is
  that this did not take place if you ran dotest manually
- running "make clean" before the main "make" target: this relied on the
  clean command being correctly implemented. This was usually true, but
  not always.
- for files which were not produced by make, each python file was
  responsible for ensuring their deleting, using a variety of methods.

With this approach, the previous methods become redundant. I remove the
first two, since they are centralized. For the other various bits of
clean-up code in python files, I indend to delete it when I come
across it.

Reviewers: aprantl

Subscribers: emaste, ki.stfu, mgorny, eraman, lldb-commits

Differential Revision: https://reviews.llvm.org/D44526

llvm-svn: 327703
2018-03-16 12:04:46 +00:00
Vedant Kumar 9a5466fd9a [test] cmake: Ensure liblldb builds before tests run
Without liblldb as a test dependency, tests which link it in from an
lldb framework (via Base.buildDriver()) won't work.

llvm-svn: 327595
2018-03-15 01:09:13 +00:00
Jonas Devlieghere 22af9c63f3 [dotest] Rename llvm-dotest -> lldb-dotest and make it a custom target
This renames llvm-dotest to lldb-dotest and makes it a custom target so
you can run `ninja lldb-dotest` to rebuild whatever is necessary before
rerunning the tests.

Differential revision: https://reviews.llvm.org/D44473

llvm-svn: 327519
2018-03-14 15:36:32 +00:00
Vedant Kumar 45ae11cd80 [test] Skip a test when using an out-of-tree debugserver
The test "test_fp_special_purpose_register_read" in TestRegisters.py
fails on Darwin machines configured to use an out-of-tree debugserver.

The error message is: 'register read ftag' returns expected result, got
'ftag = 0x80'. This indicates that the debugserver in use is too old.

This commit introduces a decorator which can be used to skip tests which
rely on having a just-built debugserver. This resolves the issue:

$ ./bin/llvm-dotest -p TestRegisters.py -v
  1 out of 617 test suites processed - TestRegisters.py
  Test Methods:          7
  Success:               6
  Skip:                  1
...

llvm-svn: 327052
2018-03-08 19:46:39 +00:00
Jonas Devlieghere a52cb80db6 [test] Add dotest wrapper
This adds a wrapper around dotest, similar to llvm-lit in llvm. The
wrapper is created in the binary directory, next to LLDB and allows you
to invoke dotest without having to pass any of the configuration
arguments yourself. I think this could also be useful for re-running a
particular test case when it fails, as an alternative to "Command
Invoked".

The motivation for this is that I'd like to replace the driver part of
dotest with lit. As a first step, I'd like to have lit invoke dotest,
which would just run the complete test suite, completely identical to
what the CMake target does today. Once this is in place, we can have lit
run dotest for the different test directories, and ultimately once per
python file. Along the way we can strip out driver functionality from
dotest where appropriate.

https://reviews.llvm.org/D44002

llvm-svn: 326687
2018-03-05 10:03:44 +00:00
Davide Italiano fb92f4e4c4 [testsuite] Run lit tests as part of `check-lldb`.
Also, fix a missing dependency, as lit requires llvm-config
to run. This is becoming more and more important as we
write more FileCheck style tests.

Differential Revision:  https://reviews.llvm.org/D43591

llvm-svn: 325719
2018-02-21 21:10:44 +00:00
Vedant Kumar 361b5e1169 [cmake] Darwin: Copy in the system debugserver if needed
This makes the built debugger functional on Darwin when compiling
without code signing (as documented in docs/code-signing.txt).

llvm-svn: 325068
2018-02-13 23:05:46 +00:00
Adrian Prantl 74bd5eb067 CMake: fix build directory name in clean rule
llvm-svn: 324281
2018-02-05 22:35:46 +00:00
Adrian Prantl 34769d80dd Use an alternative approach to prevent Spotlight from indexing the build directory.
llvm-svn: 324115
2018-02-02 18:32:29 +00:00
Adrian Prantl 5ec76fe720 Compile the LLDB tests out-of-tree.
This patch is the result of a discussion on lldb-dev, see
http://lists.llvm.org/pipermail/lldb-dev/2018-January/013111.html for
background.

For each test (should be eventually: each test configuration) a
separate build directory is created and we execute

  make VPATH=$srcdir/path/to/test -C $builddir/path/to/test -f $srcdir/path/to/test/Makefile -I $srcdir/path/to/test

In order to make this work all LLDB tests need to be updated to find
the executable in the test build directory, since CWD still points at
the test's source directory, which is a requirement for unittest2.

Although we have done extensive testing, I'm expecting that this first
attempt will break a few bots. Please DO NOT HESITATE TO REVERT this
patch in order to get the bots green again. We will likely have to
iterate on this some more.

Differential Revision: https://reviews.llvm.org/D42281

llvm-svn: 323803
2018-01-30 18:29:16 +00:00
Pavel Labath 1902ffd9a4 [lldb] Generic base for testing gdb-remote behavior
Summary:
Adds new utilities that make it easier to write test cases for lldb acting as a client over a gdb-remote connection.

- A GDBRemoteTestBase class that starts a mock GDB server and provides an easy way to check client packets
- A MockGDBServer that, via MockGDBServerResponder, can be made to issue server responses that test client behavior.
- Utility functions for handling common data encoding/decoding
- Utility functions for creating dummy targets from YAML files

----

Split from the review at https://reviews.llvm.org/D42145, which was a new feature that necessitated the new testing capabilities.

Reviewers: clayborg, labath

Reviewed By: clayborg, labath

Subscribers: hintonda, davide, jingham, krytarowski, mgorny, lldb-commits

Differential Revision: https://reviews.llvm.org/D42195
Patch by Owen Shaw <llvm@owenpshaw.net>

llvm-svn: 323636
2018-01-29 10:02:40 +00:00
Vedant Kumar 3c3345be1b [CMake] Make check-lldb work with LLDB_CODESIGN_IDENTITY=''
On Darwin, if a test machine isn't set up for code-signing (see
docs/code-signing.txt), running check-lldb should use the system
debugserver instead of the unsigned one built in-tree. This makes it
possible to run lldb's test suite without having code-signing set up,
which is really convenient.

Differential Revision: https://reviews.llvm.org/D42215

llvm-svn: 322803
2018-01-18 01:16:30 +00:00
Davide Italiano 864a6edf8b [CMake] darwin-debug is an hard dependency for tests on macOS.
Fixes a few failured on the testsuite with CMake.

llvm-svn: 320891
2017-12-15 23:27:10 +00:00
Stephane Sezer 972d513a5c Add a dependency from check-lldb on lld
Summary:
This is required when using the in-tree clang for building tests,
because -fuse-ld=lld is used by default.

Subscribers: mgorny

Differential Revision: https://reviews.llvm.org/D39689

llvm-svn: 317501
2017-11-06 19:25:33 +00:00
Davide Italiano 4ab13cda81 [CMake] Build clang as dependency when using in-tree clang for tests.
Discussed with Zachary Turner and Pavel Labath on lldb-dev.
Let's hope this doesn't break anything :)

llvm-svn: 316800
2017-10-27 21:22:57 +00:00
Pavel Labath 7e17fc5e09 Default to using in-tree clang for building test executables
Summary:
Using the in-tree clang should be the default test configuration as that
is the one compiler that we can be sure everyone has (better
reproducibility of test results). Also, it should hopefully reduce the
impact of pr35040.

This also reduces the number of settings which control the compiler
used. LLDB_TEST_C_COMPILER is used for C files and
LLDB_TEST_CXX_COMPILER for C++ files. Both of the settings default to
the in-tree clang.

Reviewers: zturner

Subscribers: mgorny, davide, lldb-commits

Differential Revision: https://reviews.llvm.org/D39215

llvm-svn: 316728
2017-10-27 02:24:04 +00:00
Tim Hammerquist c635cb223e cmake build needs to run tests AND collect results
CMake target "check-lldb" runs the lldb dotest.py suite, but doesn't
collect the results in a usable format. In adding the arguments
necessary to collect these results, I found some minor bugs in CMake
that prevented dotest overrides from being used. This patch fixes them.

<rdar://problem/33389717> cmake build needs to run tests AND collect results

llvm-svn: 308393
2017-07-18 23:45:06 +00:00
Chris Bieneman 265ca535ab [CMake] Override debugserver to use the build tree on Darwin
This patch adds support to the test suite for overriding the path to debugserver, and uses the override to point to the build tree's debugserver on Darwin.

llvm-svn: 297776
2017-03-14 20:04:46 +00:00
Pavel Labath 197b65c7b8 test: pass correct objcopy and ar paths to the test runner
Summary:
The test runner has code to autodetect this, but it's not very smart --
in particular, it fails in the case where we build the test executables
with the just-built clang. Since cmake already has the knowledge about
the right toolchain, we can just have it pass the appropriate flags to
the test runner.

This also removes the "temporary" cache-scrubbing hack added a couple
months ago.

Reviewers: zturner, beanz

Subscribers: mgorny, lldb-commits

Differential Revision: https://reviews.llvm.org/D30453

llvm-svn: 296593
2017-03-01 10:08:44 +00:00
Chris Bieneman 679d02f2a1 [CMake] Only support LLDB_BUILD_FRAMEWORK on CMake 3.7 and later
CMake's framework target generation was unable to generate POST_BUILD steps (see: https://gitlab.kitware.com/cmake/cmake/issues/16363).

It turns out working around this is really not reasonable. The more reasonable solution to me is just to not support LLDB.framework unless you are on CMake 3.7 or newer.

Since CMake 3.7.1 is released that's how I'm going to handle this.

llvm-svn: 289841
2016-12-15 18:18:47 +00:00
Chris Bieneman 668fe2f92b [CMake] Support LLDB_TEST_CLANG in check-lldb target(s)
This just hooks up the in-tree compiler to be optionally used when running the test suite.

llvm-svn: 285839
2016-11-02 17:19:26 +00:00
Chris Bieneman b9fc4727cc [CMake] Populate the build directory's framework
This ensures that the Resources and clang headers are properly symlinked in LLDB's framework. This should fix the modules-related tests when building on Darwin with CMake if you are building a framework.

I have another fix coming which gets them working on Darwin if you're building liblldb instead of a framework.

llvm-svn: 285651
2016-10-31 22:31:07 +00:00
Chris Bieneman bd6d69987c [Test Suite] Properly respect --framework option
Summary:
dotest.py has a framework option that is not respected. This patch makes the framework path properly configurable via the --framework option.

This patch also adds a function to the lldbtest.Base class named "hasDarwinFramework" which allows us to not rely on the host platform to determine if a framework is present. If running on Darwin, and not building a framework, this will follow the *nix code paths which are appropriate for Darwin.

Reviewers: tfiala

Subscribers: lldb-commits, mgorny

Differential Revision: https://reviews.llvm.org/D25886

llvm-svn: 285541
2016-10-31 04:48:10 +00:00
Chris Bieneman 7ba5581b84 [Test Suite] Allow overriding codesign identity
Summary: Not everyone names their code sign identity "lldb_codesign", so it is nice to allow this to be overridden.

Reviewers: zturner, tfiala

Subscribers: labath, mgorny, lldb-commits

Differential Revision: https://reviews.llvm.org/D25714

llvm-svn: 284893
2016-10-21 22:13:55 +00:00
Chris Bieneman f50761f128 [CMake] Don't include LLDB_TEST_COMPILER in cached variable
Summary:
CMake has no builtin mechanism for cache invalidation. As a general convention you want to not expand user-specified variables in other cached variables because they will not get updated when the user changes their specified value.

This patch moves the "-C" option for dotest.py into the LLDB_TEST_COMMON_ARGS and out of the CMake cache. In order to prevent issues with out-of-date cache files on builders I've added code to scrub "-C ${LLDB_TEST_COMPILER}" out of the CMake caches, by Force writing the variable. This code can be removed in a few days once the change has trickled through CI systems.

Reviewers: tfiala, labath, zturner

Subscribers: lldb-commits, mgorny

Differential Revision: https://reviews.llvm.org/D25751

llvm-svn: 284551
2016-10-18 23:54:28 +00:00
Chris Bieneman e549894b7d [CMake] Cleanup check-lldb targets
Summary:
This patch adds the following fixes to the check-lldb targets:

* Adds missing dependencies on lldb tools so they get built before tests execute
* Adds Ninja USES_TERMINAL to the target so that the output streams to stdout as it executes
* Uses a generator expression to find the lldb executable, this is more robust than constructing the path manually

Reviewers: tfiala, zturner

Subscribers: mgorny, lldb-commits

Differential Revision: https://reviews.llvm.org/D25490

llvm-svn: 284046
2016-10-12 20:26:13 +00:00
Zachary Turner fa7f948237 Always rerun all tests on Windows.
There is flakiness somewhere in the core infrastructure on Windows,
so to get the buildbot reliably green we need to mark all tests
as flaky.

llvm-svn: 270460
2016-05-23 17:32:04 +00:00
Zachary Turner 8d4d151bb2 Allow custom formatting of session log file names.
Differential Revision: http://reviews.llvm.org/D20306

llvm-svn: 269793
2016-05-17 18:02:34 +00:00
Todd Fiala fed95660f3 Roll dosep.py parallel test runner into dotest.py command line
See the following for details:
http://reviews.llvm.org/D12587

llvm-svn: 246794
2015-09-03 18:58:44 +00:00
Zachary Turner 555a7a6ad2 Add a method to disable the Windows crash / assert dialogs.
When running the test suite on Windows, we can't have Windows popping
up dialogs when LLDB crashes in native code because it will hang
the test suite.  This patch silences those dialogs by checking an
environment variable at startup and configuring Windows based on
its value.

This patch also adds an environment variable to force inferiors to
never spawn in their own console window.  This is useful to prevent
new window spawm when running the test suite.

Reviewed by: Scott Graham

Differential Revision: http://reviews.llvm.org/D6628

llvm-svn: 224137
2014-12-12 18:10:52 +00:00
Ed Maste 098a40180c Fix test execution: the cmake keyword is CACHE, not CACHED
llvm-svn: 214522
2014-08-01 16:57:46 +00:00
Zachary Turner 7c1bc2b8ae Make CMake choose the target architecture according to the build.
Previously, CMake was invoking the test runner and not specifying
what architecture to use when building test executables.  The
Makefiles for the test executables then had logic to choose x64
by default.  This doesn't work on Windows because the test compiler
would then try to link against the 64-bit MSVCRT and not find them
since only the 32-bit MSVCRT was in the path.

This patch addresses this by figuring out, at CMake time, whether
or not you are building LLDB with a 64 or 32-bit toolchain.  Then,
it explicitly passes this value to the test runner, causing the
test runner to build tests whose architecture matches that of LLDB
itself.  This can still be overridden by setting the CMake variable
LLDB_TEST_EXECUTABLE_ARCH=(x64|x86)

llvm-svn: 214443
2014-07-31 21:07:41 +00:00
Zachary Turner 07b21d7a69 Rename dosep.ty to dosep.py
llvm-svn: 213555
2014-07-21 16:16:31 +00:00
Zachary Turner 2f12195b1d Allow the user to override the LLDB_TEST_COMPILER at CMake level.
llvm-svn: 213421
2014-07-18 22:46:22 +00:00
Todd Fiala a4ec2fcff9 Add executable extension to debugger name, run dotest via binary.
See http://reviews.llvm.org/D3904 for details.

Change by Scott Graham.

llvm-svn: 210036
2014-06-02 17:49:35 +00:00
Daniel Malea b218b7f22a Unset CFLAGS/CXXFLAGS when running tests via cmake/make check targets
- should fix automatic tests set up on http://llvm-jenkins.debian.net
- needed in order to make Debian package builds depend on passing test suite

llvm-svn: 181736
2013-05-13 21:41:35 +00:00
Daniel Malea 3c216ace9b Fix running of LLDB tests with cmake+ninja+ccache
- Avoid passing 'ccache' as the test compiler. Instead, use first arg after ccache.

llvm-svn: 176596
2013-03-07 00:50:54 +00:00
Daniel Malea ab5c2012d3 Fix Ninja LLDB build on Linux (via cmake -G Ninja)
- resolved circular dependency issue by making liblldb depend directly on LLDBWrapPython.cpp
- removed use of '..' for relative directories -- ninja doesn't like this
- fixed build-order problem

llvm-svn: 176517
2013-03-05 21:59:12 +00:00
Daniel Malea 931b17c705 Finish up CMake support for LLDB (tested on Linux)
- add missing scripts (driver, tests, etc...)
- enable running of tests from cmake with "make check-lldb" target
- fix up problem with clang dependencies (this enables parallel builds)
- implement platform-specific FIXMEs in source/CMakeLists.txt

llvm-svn: 176306
2013-02-28 23:11:46 +00:00