Commit Graph

951 Commits

Author SHA1 Message Date
serge-sans-paille 2706942a22 [lit] Ship and bundle license for lit package
lit is bundled independently from LLVM in at least one location
(https://pypi.org/project/lit/), it's easier to state its license locally.

Differential Revision: https://reviews.llvm.org/D90136
2020-10-30 15:34:00 +01:00
Ben Dunbobbin 6a5c19dd39 [LIT] fixed up accidentally committed EOF problems. 2020-10-21 13:29:58 +01:00
Ben Dunbobbin 81c0d36a18 [LIT] error if directly named test won't be run indirectly
Currently, a LIT test named directly (on the command line) will
be run even if the name of the test file does not meet the rules
to be considered a test in the LIT test configuration files for
its test suite. For example, if the test does not have a
recognised file extension.

This makes it relatively easy to write a LIT test that won't
actually be run. I did in: https://reviews.llvm.org/D82567

This patch adds an error to avoid users doing that. There is a
small performance overhead for this check. A command line option
has been added so that users can opt into the old behaviour.

Differential Revision: https://reviews.llvm.org/D83069
2020-10-21 13:21:29 +01:00
Alex Richardson 9a6ae91128 [lit] Avoid calling realpath() for every printed message
I did some profiling of lit while trying to optimize the libc++ test
startup for remote hosts and it turns out that there is a realpath() call
for every message printed and this shows up in the profile.
The inspect.getframeinfo() function calls realpath() internally and
moreover we don't need most of the other information returned from it.
This patch uses inspect.getsourcefile() and os.path.abspath() to remove
../ from the path instead. Not resolving symlinks reduces the startup time
for running a single test with lit by about 50ms for me.

Reviewed By: ldionne, yln
Differential Revision: https://reviews.llvm.org/D89186
2020-10-19 16:21:34 +01:00
Raphael Isemann 777df5c93d [lldb][NFC] Fix some invalid escapes sequences in Python strings
I recently had to run the test suite with a debug Python which got started
warning about some invalid escape sequences in LLDB's Python code. They all
attempt to add a backslash by doing a single backslash instead of a double
backslash in a normal string. This seems to work fine for now, but Python says
this behaviour is deprecated, so this patch turns all those strings into raw
strings (where a single backslash is actually a single backslash)

Reviewed By: JDevlieghere

Differential Revision: https://reviews.llvm.org/D88289
2020-10-13 16:04:24 +02:00
Dan Liew 295d4e420f [lit] Try to remove the flakeyness of `shtest-timeout.py` and `googletest-timeout.py`.
The tests previously relied on the `short.py` and `FirstTest.subTestA`
script being executed on a machine within a short time window (1 or 2
seconds). While this "seems to work" it can fail on resource constrained
machines. We could bump the timeout a little bit (bumping it too
much would mean the test would take a long time to execute) but it wouldn't
really solve the problem of the test being prone to failures.

This patch tries to remove this flakeyness by separating testing into
two separate parts:

1. Testing if a test can hit a timeout.
2. Testing if a test can run to completion in the presence of a
timeout.

This way we can give (1.) a really short timeout (to make the test run
as fast as possible) and (2.) a really long timeout. This means for (2.)
we are no longer trying to rely on the "short" test executing within
some short time window. Instead the window is now 3600 seconds which
should be long enough even for a heavily resource constrained machine to
execute the "short" test.

Thanks to Julian Lettner for suggesting this approach. This superseeds
my original approach in https://reviews.llvm.org/D88807.

This patch also changes the command line override test to run the quick
test rather than the slow one to make the test run faster.

Differential Revision: https://reviews.llvm.org/D89020
2020-10-08 10:46:18 -07:00
Alexandre Ganea d3d790fc98 Revert [lit] Support running tests on Windows without GnuWin32
This reverts b3418cb4eb and d12ae042e1

This breaks some external bots, see discussion in https://reviews.llvm.org/D84380

In the meanwhile, please use `cmake -DLLVM_LIT_TOOLS_DIR="C:/Program Files/Git/usr/bin"` or add it to %PATH%.
2020-10-06 15:38:18 -04:00
Joseph Huber 1dce692de1 Revert "[OpenMP] Add Error Handling for Conflicting Pointer Sizes for Target Offload"
Reverting because detecting architecture size doesn't work on all
platforms.

This reverts commit eaf73293cb.
2020-10-05 12:35:39 -04:00
Joseph Huber eaf73293cb [OpenMP] Add Error Handling for Conflicting Pointer Sizes for Target Offload
Summary:
This patch adds an error to Clang that detects if OpenMP offloading is
used between two architectures with incompatible pointer sizes. This
ensures that the data mapping can be done correctly and solves an issue
in code generation generating the wrong size pointer. This patch adds a
new lit substitution, %omp_powerpc_triple that, if the system is 32-bit or
64-bit, sets the powerpc triple accordingly. This was required to fix
some OpenMP tests that automatically populated the target architecture.

Reviewers: jdoerfert

Subscribers: cfe-commits guansong sstefan1 yaxunl delcypher

Tags: OpenMP clang LLVM

Differential Revision: https://reviews.llvm.org/D88594
2020-10-05 11:02:13 -04:00
Reid Kleckner d12ae042e1 [lit] Fix Python 2/3 compat in new winreg search code
This should fix the test failures on the clang win64 bot:
http://lab.llvm.org:8011/builders/clang-x64-windows-msvc/builds/18830
It has been red since Sept 23-ish.

This was subtle to debug. Windows has 'find' and 'sort' utilities in
C:\Windows\system32, but they don't support all the same flags as the
coreutils programs. I configured the buildbot above with Python 2.7
64-bit (hey, it was set up in 2016). When I installed git for Windows, I
opted to add all the Unix utilities that come with git to the system
PATH. This is *almost* enough to make the LLVM tests pass, but not
quite, because if you use the system PATH, the Windows version of find
and sort come first, but the tests that use diff, cmp, etc, will all
pass. So only a handful of tests will fail, and with cryptic error
messages.

The code changed in this CL doesn't work with Python 2. Before
Python 3.2, the winreg.OpenKey function did not accept the `access=`
keyword argument, the caller was required to pass an unused `reserved`
positional argument of 0. The try/except/pass around the OpenKey
operation masked this usage error in Python 2.

Further, the result of the registry operation has to be converted from
unicode to add it to the environment, but that was incidental.
2020-10-01 12:22:28 -07:00
Zachary Turner b3418cb4eb [lit] Support running tests on Windows without GnuWin32
Historically, we have told contributors that GnuWin32 is a pre-requisite
because our tests depend on utilities such as sed, grep, diff, and more.
However, Git on Windows includes versions of these utilities in its
installation.  Furthermore, GnuWin32 has not been updated in many years.
For these reasons, it makes sense to have the ability to run llvm tests
in a way that is both:
  a) Easier on the user (less stuff to install)
  b) More up-to-date (The verions that ship with git are at least as
     new, if not newer, than the versions in GnuWin32.

We add support for this here by attempting to detect where Git is
installed using the Windows registry, confirming the existence of
several common Unix tools, and then adding this location to lit's PATH
environment.

Differential Revision: https://reviews.llvm.org/D84380
2020-09-24 08:21:43 -04:00
Rainer Orth f59bec7acb [clang][Driver] Default to /usr/bin/ld on Solaris
`clang` currently requires the native linker on Solaris:

  - It passes `-C` to `ld` which GNU `ld` doesn't understand.

  - To use `gld`, one needs to pass the correct `-m EMU` option to select
    the right emulation.  Solaris `ld` cannot handle that option.

So far I've worked around this by passing `-DCLANG_DEFAULT_LINKER=/usr/bin/ld`
to `cmake`.  However, if someone forgets this, it depends on the user's
`PATH` whether or not `clang` finds the correct linker, which doesn't make
for a good user experience.

While it would be nice to detect the linker flavor at runtime, this is more
involved.  Instead, this patch defaults to `/usr/bin/ld` on Solaris.  This
doesn't work on its own, however: a link fails with

  clang-12: error: unable to execute command: Executable "x86_64-pc-solaris2.11-/usr/bin/ld" doesn't exist!

I avoid this by leaving absolute paths alone in `ToolChain::GetLinkerPath`.

Tested on `amd64-pc-solaris2.11`, `sparcv9-sun-solaris2.11`, and
`x86_64-pc-linux-gnu`.

Differential Revision: https://reviews.llvm.org/D84029
2020-08-13 22:42:58 +02:00
Alex Richardson 3b3cdcc7a5 [lit] Remove ANSI control characters from xunit output
Failing test output sometimes contains control characters like \x1b (e.g.
if there was some -fcolor-diagnostics output) which are not allowed inside
XML files. This causes problems with CI systems: for example, the Jenkins
JUnit XML will throw an exception when ecountering those characters and
similar problems also occur with GitLab CI.

Reviewed By: yln, jdenny

Differential Revision: https://reviews.llvm.org/D84233
2020-08-06 09:16:52 +01:00
Russell Gallop 98827feddb [lit] Add --time-trace-output to lit
This produces a chrome://tracing compatible trace file in the same way
as -ftime-trace.

This can be useful in optimising test time where one long test is causing
long overall test time on a wide machine.

This also helped in finding tests which have side effects on others
(e.g. https://reviews.llvm.org/D84885).

Differential Revision: https://reviews.llvm.org/D84931
2020-08-04 14:25:23 +01:00
Varun Gandhi 417d3d495f [docs] [lit] Add a more helpful description for lit.py's -s flag.
Reviewed By: yln

Differential Revision: https://reviews.llvm.org/D82808
2020-07-28 14:36:03 -07:00
Sergej Jaskiewicz 4dd5c2bee3 [lit] Don't expand escapes until all substitutions have been applied
Otherwise, if a Lit script contains escaped substitutions (like %%p in this test https://github.com/llvm/llvm-project/blob/master/compiler-rt/test/asan/TestCases/Darwin/asan-symbolize-partial-report-with-module-map.cpp#L10), they are unescaped during recursive application of substitutions, and the results are unexpected.

We solve it using the fact that double percent signs are first replaced with #_MARKER_#, and only after all the other substitutions have been applied, #_MARKER_# is replaced with a single percent sign. The only change is that instead of replacing #_MARKER_# at each recursion step, we replace it once after the last recursion step.

Differential Revision: https://reviews.llvm.org/D83894
2020-07-27 18:09:00 +03:00
Alex Richardson 18df607dbe [lit] Don't include tests skipped due to sharding in reports
When running multiple shards, don't include skipped tests in the xunit
output since merging the files will result in duplicates.
In our CHERI Jenkins CI, I configured the libc++ tests to run using sharding
(since we are testing using a single-CPU QEMU). We then merge the generated
XUnit xml files to produce a final result, but if the individual XMLs
report tests excluded due to sharding each test is included N times in the
final result. This also makes it difficult to find the tests that were
skipped due to missing REQUIRES: etc.

Reviewed By: yln

Differential Revision: https://reviews.llvm.org/D84235
2020-07-26 11:39:22 +01:00
David Truby 4ef2e594d5 [flang] Run non-gtest unit tests with lit.
Summary:
As a corrollary, these tests are now run as part of the check-flang
target.

Reviewers: sscalpone

Subscribers: mgorny, delcypher, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D83946
2020-07-24 14:49:39 +01:00
Alex Richardson 722e5d6aae [lit] Include total elapsed time in xunit output
The time= attribute can also be used for the testsuites tag. While this
attribute appears to be ignored by Jenkins (bab34bcc96/src/main/java/hudson/tasks/junit/SuiteResult.java (L202)),
it is still useful if you manually inspect the xml file.

Reviewed By: yln

Differential Revision: https://reviews.llvm.org/D84230
2020-07-23 11:19:18 +01:00
Marcel Hlopko b49d47f6a0 Make lit TestRunner.py work in Python 3
Summary: In Python3 SubstituteCaptures are no longer converted to String implicitly behind the scenes. Converting explicitly makes the TestRunner to work in Python3.

Reviewers: gribozavr2, compnerd

Reviewed By: gribozavr2

Subscribers: tbkka, delcypher, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D81361
2020-07-22 11:44:00 +02:00
Hans Wennborg 7ab7b979d2 Bump the trunk major version to 12
and clear the release notes.
2020-07-15 12:05:05 +02:00
Richard Barton f4476b72fb [lit] Prevent hang when lit sees non-ASCII characters
As per discussion in D69207, have lit ignore UnicodeDecodeErrors
when running with python 2 in an ASCII shell.

Differential Revision: https://reviews.llvm.org/D82754
2020-07-14 18:41:15 +01:00
Julian Lettner f06d2420b7 [lit] Add --show-xxx command line options
Provide `--show-xxx` flags for all non-failure result codes, just as we
already do for `--show-xfail` and `--show-unsupported`.

Reviewed By: jdenny

Differential Revision: https://reviews.llvm.org/D82233
2020-07-08 17:01:05 -07:00
Shuhong Liu 2402f9385e [AIX] Add system-aix to lit config file
Summary: This is a complementary patch to D82100 since the aix builbot is still running the unsupported test shtest-format-argv0. Add system-aix to the sub llvm-lit config.

Reviewers: daltenty, hubert.reinterpretcast

Reviewed By: hubert.reinterpretcast

Subscribers: delcypher, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D82905
2020-07-06 12:54:12 -04:00
Shuhong Liu af8aee7c07 [AIX] Split lit test shtest-format into two separate tests and add AIX as UNSUPPORTED for shtest-format-argv0
Summary: lit test `shtest-format.py` fails on AIX  because one of the subtest of shtest-format requires the tool `[` to be installed under the system PATH. For AIX, `[` is only available as a shell builtin and does not present as an executable file under PATH. Hence, split the original shtest-format into two separate test files and added AIX as UNSUPPORTED for the test using `[` .

Reviewers: daltenty, hubert.reinterpretcast

Reviewed By: hubert.reinterpretcast

Subscribers: delcypher, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D82100
2020-06-18 16:58:07 -04:00
Julian Lettner 5c244115c9 [lit] Avoid re-iterating result codes in different places
Let's have one canonical place to define ResultCode instances and their
labels.

Also make ResultCode's `__init__` function self-registering to better
support custom ResultCodes.
2020-06-17 11:40:19 -07:00
Julian Lettner 7837de1394 [lit] Improve consistency for showing result groups
Before this change we showed all result groups with a code that was not
explicitly hard-coded set.  This set missed the FLAKYPASS result code.

Let's generalize the code to always show failures and the additionally
requested result codes.
2020-06-16 12:40:06 -07:00
Casey Carter d66428cb99 [lit] Allow for tests to have non-parsed requirements
MSVC uses lit for STL testing to run both the libcxx tests and our "native" suite of tests which has feature requirements that are not parsed from the test content. For consistency, the change treats the `unsupported` and `xfails` `Test` properties similarly to `requires`.

Differential Revision: https://reviews.llvm.org/D81782
2020-06-15 16:43:02 -07:00
Julian Lettner 853e61142d Follow up for "[lit] Include unexecuted tests in xUnit report"
The extended test fails on bots that still run lit tests with Python 2.

https://reviews.llvm.org/D81316
2020-06-15 16:19:43 -07:00
Julian Lettner 968f58c684 [lit] Include unexecuted tests in xUnit report
Pass in all discovered tests to report generators.

The XunitReport generator now creates testcase items for unexecuted
tests and documents why they have been skipped.  This makes it easier
to compare test runs with different filters or configurations, or across
platforms.

I don't know who is using the JsonReport generator and what the
expectations there are (it doesn't have tests), so decided to preserve
the old behavior by filtering out the unexecuted tests.

Reviewed By: jdenny

Differential Revision: https://reviews.llvm.org/D81316
2020-06-15 11:36:31 -07:00
Louis Dionne 9bf4c9e416 [Lit] Pass through SSH_AUTH_SOCK from the surrounding environment
This allows running Lit tests that run ssh without having to manually
enter a password (which is inconvenient), by just having ssh-agent
setup properly when running the test suite.
2020-06-12 13:59:29 -04:00
Joel E. Denny 2866f9db9e [lit] Fix handling of various keyword parse errors
In TestRunner.py, D78589 extracts a `_parseKeywords` function from
`parseIntegratedTestScript`, which then expects `_parseKeywords` to
always return a list of keyword/value pairs.  However, the extracted
code sometimes returns an unresolved `lit.Test.Result` on a keyword
parsing error, which then produces a stack dump instead of the
expected diagnostic.

This patch fixes that, makes the style of those diagnostics more
consistent, and extends the lit test suite to cover them.

Reviewed By: ldionne

Differential Revision: https://reviews.llvm.org/D81665
2020-06-12 09:37:40 -04:00
Mehdi Amini d31c9e5a46 Change filecheck default to dump input on failure
Having the input dumped on failure seems like a better
default: I debugged FileCheck tests for a while without knowing
about this option, which really helps to understand failures.

Remove `-dump-input-on-failure` and the environment variable
FILECHECK_DUMP_INPUT_ON_FAILURE which are now obsolete.

Differential Revision: https://reviews.llvm.org/D81422
2020-06-09 18:57:46 +00:00
Julian Lettner 99d6e05e71 [lit] Improve naming of test result categories
Improve consistency when printing test results:
Previously we were using different labels for group names (the header
for the list of, e.g., failing tests) and summary count lines.  For
example, "Failing Tests"/"Unexpected Failures".  This commit changes lit
to label things consistently.

Improve wording of labels:
When talking about individual test results, the first word in
"Unexpected Failures", "Expected Passes", and "Individual Timeouts" is
superfluous.  Some labels contain the word "Tests" and some don't.
Let's simplify the names.

Before:
```
Failing Tests (1):
  ...

Expected Passes    : 3
Unexpected Failures: 1
```

After:
```
Failed Tests (1):
  ...

Passed: 3
Failed: 1
```

Reviewed By: ldionne

Differential Revision: https://reviews.llvm.org/D77708
2020-06-05 08:14:42 -07:00
Louis Dionne dac21fd29c [lit] Add an option to print all features used in tests
Lit test suites can tend to accumulate annotations that are not necessarily
relevant as time goes by, for example XFAILS on old compilers or platforms.
To help spot old annotations that can be cleaned up, it can be useful to
look at all features used inside a test suite.

This commit adds a new Lit option '--show-used-features' that prints all
the features used in XFAIL, REQUIRES and UNSUPPORTED of all tests that
are discovered.

Differential Revision: https://reviews.llvm.org/D78589
2020-05-29 07:00:05 -04:00
Stephen Neuendorffer 0231227e5d Reapply "[lit] GoogleTest framework should report failures if test binary crashes"
This reverts commit 78dea0e8fb.

The offending lldb test (which is a real bug exposed by this patch)
has been disabled on windows (see a67b2faa7c)
and lldb is queued for inclusion into precommit testing, which would
have caught this.

Differential Revision: https://reviews.llvm.org/D80389
2020-05-22 14:01:08 -07:00
Jonas Devlieghere 78dea0e8fb Revert "[lit] GoogleTest framework should report failures if test binary crashes"
This reverts commit ef21031822 because it
breaks the Windows bot:

http://lab.llvm.org:8011/builders/lldb-x64-windows-ninja/builds/16447

Failing Tests (2):
  ...
  lldb-unit :: API/./APITests.exe/failed_to_discover_tests_from_gtest
2020-05-20 23:22:47 -07:00
Stephen Neuendorffer ef21031822 [lit] GoogleTest framework should report failures if test binary crashes
lit runs a gtest executable multiple times. First it runs it to
discover tests, then later it runs the executable again for each test.
However, if the discovery fails (perhaps because of a broken
executable), then no tests were previously run and no failures were
reported.  This patch creates a dummy test if discovery fails, which
will later fail when test are run and be reported as a failure.

Differential Revision: https://reviews.llvm.org/D80096
2020-05-20 13:36:08 -07:00
Pete Steinfeld 72416b136e Fixed a problem removing temp files
Summary:
Before making this change, whenever I ran "check-flang", I'd get an
error message like:

llvm-lit: /mnt/c/GitHub/f18/c751/flang/build/bin/../../../llvm/utils/lit/lit/main.py:252: warning: Failed to delete temp directory '/tmp/lit_tmp_gOKUIh'

With this change, there's no such message in the output, and the temp
directory is successfully removed.

Note that my working environment is on Windows 10 running Windows
Subsystem for Linux using the Ubuntu app.  I'm running Python version
2.7.1.

Earlier versions of Python do not contain `shutil`.  It may be that this
module was available on Windows systems later than other platforms.
Upgrading my version of Python made the problem go away

I don't believe that timing was a problem since inserting a long delay
didn't fix things.

So I added some text to the error message recommending that the user
upgrade their version of Python if they run into this problem.

Reviewers: yln, DavidTruby

Subscribers: delcypher, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D79861
2020-05-13 11:40:19 -07:00
Julian Lettner 47b25c3323 [lit] Create one output file when `--output` is specified more than once
The argparse 'append' action concatenates multiple occurrences of an
argument (even when we specify `nargs=1` or `nargs='?'`).  This means
that we create multiple identical output files if the `--output`
argument is given more than once.  This isn't useful and we instead want
this to behave like a standard optional argument: last occurrence wins.
2020-05-04 21:36:20 -07:00
Julian Lettner 3610fd8c5c [lit] Disable test on Windows
abhinavgaba reported that that the custom-result-category.py test hangs
on a Windows build bot [1].  Disable it for now.

[1] https://reviews.llvm.org/D78164#2018178
2020-05-04 13:06:56 -07:00
Julian Lettner 9ed6f03189 [lit] Don't use Python3 yet
String interpolation is a Python3 feature, we can't use it yet.
2020-05-01 17:36:21 -07:00
Julian Lettner 840bc47f8b [lit] Extract by_suite_and_test_path sort key function 2020-05-01 17:03:55 -07:00
Julian Lettner 3eb1d7ffb5 [lit] Create report generators during argument parsing 2020-05-01 17:03:55 -07:00
Julian Lettner 201e73cbbb [lit] Small refinements for xunit report output 2020-05-01 17:03:55 -07:00
Julian Lettner 7ffb5bc2a2 [lit] Factor out report generators into separate file
Factor out the report generators from main.py into reports.py.

I verified that we generate the exact same output by running `check-all`
and comparing the new and old output for both report flavors.
2020-05-01 17:03:55 -07:00
Julian Lettner 11e02d5c24 [lit] Only update specific fields from remote test object
Don't update whole test object from the remote (pickled) finished test
object.  Doing so also changes the config and suite members, which we
want to avoid.
2020-05-01 17:03:54 -07:00
Julian Lettner 70605ff452 [lit] Don't crash for --time-tests in aborted test run
Properly print the test times histogram even for aborted (user
interrupt, [Ctrl+C]) test runs.
2020-04-30 18:19:15 -07:00
Julian Lettner 8cb8fe909b [lit] Add EXCLUDED test result category
Track and print the number of tests that were discovered but not
executed due to test selection options:
  * --filter (regex filter)
  * --max-tests (limits number of tests)
  * sharding feature

With this change all discovered tests are accounted for: every
discovered test is included in one of the counts printed in the summary.

Reviewed By: jdenny

Differential Revision: https://reviews.llvm.org/D78078
2020-04-30 17:58:11 -07:00
Julian Lettner 3cd6e02920 [lit] Push computation of workers into run_tests
This also avoids assigning to the option object.
2020-04-30 13:49:30 -07:00
Hans Wennborg 88aad9b9f0 lit googletest.py: Don't raise StopIteration in generator
The intention here seems to be to end the generator function, but with
modern Python, raising StopIteration causes a runtime error
(https://www.python.org/dev/peps/pep-0479/).

Differential revision: https://reviews.llvm.org/D79169
2020-04-30 18:16:17 +02:00
Julian Lettner fbdcfcd4c3 [lit] Provide extension API for custom result categories
The lnt test suite defines custom result codes [1].  Support those via
an extension API instead of "by accident", which should offer the
advantage of properly handling them when we print test results.

[1] https://reviews.llvm.org/D77986

Reviewed By: Meinersbur

Differential Revision: https://reviews.llvm.org/D78164
2020-04-29 19:45:55 -07:00
Nico Weber b9fd375d75 Revert "[lit] Keep original cfg file case around."
This reverts commit bc3f54de18.

The patch breaks in the following two scenarios:

1. When manually passing an absolute path to llvm-lit with a lower-case
   drive letter: `python bin\llvm-lit.py -sv c:\llvm-project\clang\test\PCH`

2. When the PWD has a lower-case drive letter, like after running
   `cd c:\` with a lower-case "c:" (cmd's default is upper-case, but
   it takes case-ness from what's passed to `cd` apparently).
2020-04-15 17:19:39 -04:00
Nico Weber bc3f54de18 [lit] Keep original cfg file case around.
There's been some back and forth if the cfg paths in the
config_map should be normcase()d. The argument for is that
it allows using all-lower spelling in cmd on Windows, the
argument against that doing so is lossy.

Before the relative-paths-in-generated-lit.site.cfg.py work,
there was no downside to calling normcase(), but with it
we need a hack to recover the original case.

This time, normcase() the hashtable key, but store the original
cased key in addition to the value. This fixes both cons, at the
cost of a few bytes more memory.

Differential Revision: https://reviews.llvm.org/D78169
2020-04-15 14:18:11 -04:00
Nico Weber f42baaab4f Remove an apparently unneeded normcase() call.
I believe this call is unneeded after https://reviews.llvm.org/D34855.

Reviewed as part of https://reviews.llvm.org/D78169
2020-04-15 14:02:48 -04:00
Nico Weber 6a887d22a1 Remove a function that has been dead since r313889. 2020-04-15 12:02:30 -04:00
Julian Lettner 23f896a096 [lit] Update local test objects "in place" from remote test objects
Update local test object "in place" from remote test object.  We need to
do this to ensure that discovered test object which is used for printing
test results reflect the changes.

> Why are we sending back the whole test object from the worker process
> (lit.worker.execute) instead of just the result?

Unfortunately, the test result is not the only "result" of test
execution.  Other members (e.g., xfails, requires) of the Test class are
set only during execution.  Those members affect the behavior of
`isExpectedToFail` and `setResult`, and are accessed when printing
results.  For example, xunit.xml test results include missing features
for "skip reasons".  The lack of separation between an immutable "test
definition" and "generated outputs" (including the primary result and
other secondary state) is unfortunate historical design decision in lit.

> Why do we update the initial test object instead of just discarding it
> and continuing with the pickled test object?

Both of these approaches would work.  However, note that we need a fully
populated test object for printing results.  Updating the existing one
seems to be the easier path.
2020-04-13 21:02:58 -07:00
Louis Dionne dd3feecd73 [lit] Print substitutions with --show-suites
We already print available features, and it can be useful to print
substitutions as well since those are a pretty fundamental part of
a test suite. We could also consider printing other things like the
test environment, however the need doesn't appear to be as strong.

As a fly-by fix, we also always print available features, even when
there are none.

Before:

  $ lit -sv libcxx/test --show-suites
  -- Test Suites --
    libc++ - 6350 tests
      Source Root: [...]
      Exec Root  : [...]
      Available Features : -faligned-allocation -fsized-deallocation [...]

After:

  $ lit -sv libcxx/test --show-suites
  -- Test Suites --
    libc++ - 6350 tests
      Source Root: [...]
      Exec Root  : [...]
      Available Features: -faligned-allocation -fsized-deallocation [...]
      Available Substitutions: %{build_module} => [...]
                               %{build} => %{cxx} -o [...]

Differential Revision: https://reviews.llvm.org/D77818
2020-04-13 12:01:12 -04:00
Julian Lettner c610807afe [lit] Temporarily disable failing tests on Windows 2020-04-10 20:03:44 -07:00
Julian Lettner 15000650a6 [lit] Fix tests on Windows
max-time.py:
  Windows does not have a native `sleep` command, use `time.sleep()` in
  Python instead.

max-failures.py:
  The max-failure test reused the shtest-shell test inputs instead of
  defining its own "test domain".  However, the output of this
  shtest-shell "test domain" is slightly different on Windows, which now
  bites us since we made the max-failures test stricter.  Let's define
  our own "max failures" test domain.
2020-04-10 17:33:49 -07:00
Julian Lettner 5925c4a0ff [lit] Increase sleep time in timeout test
Fixup for cbe42a9d5f.  Increase values for testing the overall lit
timeout (--max-time) which wasn't enough for the test to complete on
very slow build bots.
2020-04-10 16:22:00 -07:00
Julian Lettner cbe42a9d5f [lit] Add SKIPPED test result category
Track and print the number of skipped tests.  Skipped tests are tests
that should have been executed but weren't due to:
  * user interrupt [Ctrl+C]
  * --max-time (overall lit timeout)
  * --max-failures

This is part of a larger effort to ensure that all discovered tests are
properly accounted for.

Add test for overall lit timeout feature (`--max-time` option) to
observe skipped tests.  Extend test for `--max-failures` option.

Reviewed By: jdenny

Differential Revision: https://reviews.llvm.org/D77819
2020-04-10 15:13:30 -07:00
Julian Lettner 0bc2eab6f5 [lit] Print slowest tests and time histogram before result groups 2020-04-07 22:19:50 -07:00
Julian Lettner 414745026c [lit] Improve test summary output
This change aligns the test summary output along the longest
category label.  We also properly align test counts.

Before:
```
Testing Time: 10.30s
  Unsupported Tests  : 1
  Expected Passes    : 30
```

After:
```
Testing Time: 10.29s
  Unsupported Tests:  1
  Expected Passes  : 30
```
2020-04-07 22:19:50 -07:00
Julian Lettner 09f345080e [lit] Print slowest test first when timing tests
lit supports `--time-tests` which will report the 20 slowest tests and
print a nice histogram for test times.  This change prints this list and
the histogram rows by decreasing test times.  After all, we are most
interested in the slowest tests.
2020-04-07 18:18:33 -07:00
Julian Lettner 2ac96d61c2 [lit] Improve consistency when printing test results 2020-04-07 18:12:18 -07:00
Julian Lettner eb5ca295d7 [lit] Cleanup printing of discovered suites and tests 2020-04-07 10:39:35 -07:00
Julian Lettner 38edab1c40 [lit] Improve handling of parallelism group semaphores 2020-04-06 20:52:06 -07:00
Louis Dionne 8a42bf24ae [lit] Move the recursiveExpansionLimit setting to TestingConfig
The LitConfig is shared across the whole test suite. However, since
enabling recursive expansion can be a breaking change for some test
suites, it's important to confine the setting to test suites that
enable it explicitly.

Note that other issues were raised with the way recursiveExpansionLimit
operates. However, this commit simply moves the setting to the right
place -- the mechanism by which it works can be improved independently.

Differential Revision: https://reviews.llvm.org/D77415
2020-04-06 13:58:00 -04:00
Lang Hames 1b39c6f62c [ORC] Add MachO universal binary support to StaticLibraryDefinitionGenerator.
Add a new overload of StaticLibraryDefinitionGenerator::Load that takes a triple
argument and supports loading archives from MachO universal binaries in addition
to regular archives.

The LLI tool is updated to use this overload.
2020-04-05 20:21:05 -07:00
Julian Lettner 6f8c45067b [lit] Cleanly exit on user keyboard interrupt
Graceful lit shutdown on user keyboard interrupt [Ctrl+C] was a
longstanding goal of mine.  After a few refactorings this revision
finally enables it.  We use the following strategy to deal with
KeyboardInterrupt:
https://noswap.com/blog/python-multiprocessing-keyboardinterrupt

Printing of a helpful summary for interrupted runs (just as the one for
completed runs) will be tackled in future revisions.

Reviewed By: serge-sans-paille, rnk

Differential Revision: https://reviews.llvm.org/D77365
2020-04-03 13:03:44 -07:00
Julian Lettner 022f1e2cc8 [lit] Refine filter error handling
Picking a default filter `.*` that matches everything lets us streamline
some error handling code.
2020-04-02 14:45:54 -07:00
Julian Lettner cab904c6d3 [lit] Remove unnecessary indirection in progress_callback
On shutdown, the result complete handler is not racing with the main
thread anymore because we are now always waiting for process pool
termination via
```
  finally:
    pool.join()
```
2020-04-02 14:45:54 -07:00
Julian Lettner c3ef971d36 [lit] Improve handling of timeouts and max failures
This work prepares us for the overall goal of clean shutdown on user
keyboard interrupt [Ctrl+C].
2020-04-02 01:24:02 -07:00
Julian Lettner a3fab31acd [lit] Refine adoption of argparse --version action
The real work for this was already done by serge-sans-paille [1].
Thanks for this!

[1] 1d4849379f
2020-03-31 16:02:16 -07:00
Julian Lettner 1e8900cc82 [lit] Fix test that relied on "single process" mode
The shtest-inject test relied on being executed in "single process" mode
and started to fail with a `PicklingError` after it was removed:
```
  Can't pickle <class 'lit.TestingConfig.CustomFormat'>: attribute
  lookup lit.TestingConfig.CustomFormat failed
```

This happened because the test config has to be serialized to the worker
process, but apparently the `CustomFormat` class defined inline is not
serializable.

This change allows passing the tested functionality (preamble_commands)
directly to `lit.formats.ShTest` so we can use it directly in the test.
2020-03-30 21:58:48 -07:00
Julian Lettner f3c329986c [lit] Remove single process mode
Remove the "serial run" abstraction which bypasses Python's
`multiprocessing.Pool` and instead directly runs tests without spawning
worker processes.  This abstraction has not offered the benefits I hoped
it would and therefore does not carry its weight.
2020-03-30 21:58:48 -07:00
Julian Lettner 357a17e298 [lit] Send back whole lit.Test object from worker process
In previous commits [1,2] I changed worker.py to only send back the test
result from the worker process instead of the whole test object.  This
was a mistake.  lit.Test contains fields (e.g., xfials, requires,
unsupported) that are only populated when we actually execute the test,
but are queried when we report the results in the parent process.  This
commit essentially reverts the following changes:

[1] a3d2f9b53a
[2] 17bb660fb8
2020-03-30 21:58:48 -07:00
Julian Lettner d16ba9b0bd [lit] Use Python's support for None in array slice indexing 2020-03-30 12:44:03 -07:00
Julian Lettner 8896d12315 [lit] Avoid global imports in module declaration
A previous attempt to cleanup module imports broke installing via
pip/setup.py [1].  This should be fixed now.

[1] cf252240e8

Reviewed By: paquette

Differential Revision: https://reviews.llvm.org/D76940
2020-03-27 12:07:19 -07:00
Louis Dionne 08d5426981 [lit] NFC: Move the flaky test logic to _runShTest
This minor refactoring allows reducing the amount of processing that
is duplicated when we re-run a flaky test. It also has the nice
side effect that libc++'s current test format supports flaky .sh.cpp
tests, because those are built on top of _runShTest, not executeShTest.
2020-03-27 09:32:58 -04:00
Louis Dionne faf415a1de [lit] Recursively expand substitutions
This allows defining substitutions in terms of other substitutions. For
example, a %build substitution could be defined in terms of a %cxx
substitution as '%cxx %s -o %t.exe' and the script would be properly
expanded.

Differential Revision: https://reviews.llvm.org/D76178
2020-03-27 09:25:26 -04:00
Louis Dionne 0bd1276eed [lit] NFC: Remove trailing whitespace
I keep having to remove them from my diffs!
2020-03-26 11:05:18 -04:00
Louis Dionne 8f64b02d33 [lit] Allow passing extra commands to executeShTest
This allows creating custom test formats on top of `executeShTest` that
inject commands at the beginning of the file being parsed, without
requiring these commands to physically appear in the test file itself.

For example, one could define a test format that prints out additional
debug information at the beginning of each test. More realistically,
this has been used to define custom test formats like one that supports
compilation failure tests (e.g. with the extension `compile.fail.cpp`)
by injecting a command that calls the compiler on the file itself and
expects it to fail.

Without this change, the only alternative is to create a temporary file
with the same content as the original test, then prepend the desired
`// RUN:` lines to that file, and call `executeShTest` on that file
instead. This is both slow and cumbersome to do.

Differential Revision: https://reviews.llvm.org/D76290
2020-03-24 15:02:37 -04:00
Louis Dionne f951b0f82d [lit] Add builtin support for flaky tests in lit
This commit adds a new keyword in lit called ALLOW_RETRIES. This keyword
takes a single integer as an argument, and it allows the test to fail that
number of times before it first succeeds.

This work attempts to make the existing test_retry_attempts more flexible
by allowing by-test customization, as well as eliminate libc++'s FLAKY_TEST
custom logic.

Differential Revision: https://reviews.llvm.org/D76288
2020-03-18 18:04:01 -04:00
Louis Dionne 1458bb92df [lit] NFC: Fix typo in log statement 2020-03-17 16:49:56 -04:00
Peter Collingbourne 0414c56940 Revert "Rework go bindings so that validation works fine"
And add llvm-go back to the test dependencies.

No longer necessary now that llvm-go has been brought back.

This reverts commit e8f8873da5.
2020-02-24 09:20:08 -08:00
serge-sans-paille f8b8a1ca8b Fix lit version test
Looks like on some system, version is printed on stderr, on some it's on stdout...
2020-02-13 21:49:48 +01:00
serge-sans-paille 1d4849379f Fix handling of --version in lit
There's no reason why we should require a directory when asking for the version.

Differential Revision: https://reviews.llvm.org/D74553
2020-02-13 21:36:12 +01:00
serge-sans-paille e8f8873da5 Rework go bindings so that validation works fine
Basically change the layout to please `go build` and remove references to
`llvm-go`.

Update llvm/test/Bindings/Go/ to use the system go compiler

Differential Revision: https://reviews.llvm.org/D74540
2020-02-13 14:13:03 +01:00
Hans Wennborg 5852475e2c Bump the trunk major version to 11
and clear the release notes.
2020-01-15 13:38:01 +01:00
Nico Weber f6544934b9 Make check-llvm run 50% faster on macOS, 18% faster on Windows.
While looking at cycle time graphs of some of my bots, I noticed
that 327894859c made check-llvm noticeably slower on macOS and
Windows.

As it turns out, the 5 substitutions added in that change were
enough to cause lit to thrash the build-in cache in re.compile()
(re.sub() is implemented as re.compile().sub()), and apparently
applySubstitutions() is on the cricital path and slow when all
regexes need to compile all the time.

(See `_MAXCACHE = 512` in cpython/Lib/re.py)

Supporting full regexes for lit substitutions seems a bit like
overkill, but for now add a simple unbounded cache to recover
the lost performance.

No intended behavior change.
2020-01-06 12:57:42 -05:00
Jessica Paquette dcaaede1e5 [lit] Add "from .main import main" back into lit/__init__.py
A refactoring commit (cf252240) removed this line. Removing it broke installing
lit with pip and setup.py.

This adds the line back in so that we can install lit again.

For an example of how this appeared, see:

http://green.lab.llvm.org/green/job/LNT_Tests/5853/

File "/Users/buildslave/jenkins/...s/__init__.py", line 2453, in resolve
    raise ImportError(str(exc))
ImportError: 'module' object has no attribute 'main'
2019-12-19 16:09:27 -08:00
Nemanja Ivanovic ba5a00167b Fix buildbot failures after removing REQUIRES-ANY
It would appear that the removal of this lit feature was incomplete
and there is a test case that still tests for this. This patch removes
the remaining tests to bring the bots back to green. I would encourage the
author to do a post-commit review on this in case there is a more desirable fix.
2019-12-17 15:27:45 -06:00
Joel E. Denny 9658e77e27 [lit] Fix internal diff newlines for -w/-b
For example, without this patch:

```
$ python $LIT_BUILTINS/diff.py -b foo.txt bar.txt
*** /tmp/foo.txt
--- /tmp/bar.txt
***************
*** 1,2 ****
  1! 2--- 1,2 ----
  1! 20
```

With this patch:

```
$ python $LIT_BUILTINS/diff.py -b foo.txt bar.txt
*** /tmp/foo.txt
--- /tmp/bar.txt
***************
*** 1,2 ****
  1
! 2
--- 1,2 ----
  1
! 20
```

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D71577
2019-12-17 10:49:34 -05:00
Thomas Preud'homme ddd0bb8dba [lit] Remove lit's REQUIRES-ANY directive
Summary:
Remove REQUIRES-ANY alias lit directive since it is hardly used and can
be easily implemented using an OR expression using REQUIRES. Fixup
remaining testcases still using REQUIRES-ANY.

Reviewers: probinson, jdenny, gparker42

Reviewed By: gparker42

Subscribers: eugenis, asb, rbar, johnrusso, simoncook, sabuasal, niosHD, delcypher, jrtc27, zzheng, edward-jones, rogfer01, MartinMosbeck, brucehoult, the_o, PkmX, jocewei, lenary, s.egerton, pzheng, sameer.abuasal, apazos, luismarques, cfe-commits, #sanitizers, llvm-commits

Tags: #llvm, #clang, #sanitizers

Differential Revision: https://reviews.llvm.org/D71408
2019-12-17 10:36:36 +00:00
Martin Storsjö d39510ec1c [lit] [windows] Make sure to convert all path separators to backslashes in NT style \\?\... paths
E.g. the mingw python distributed in msys2 (the mingw one, which is a
normal win32 application and doesn't use the msys2 runtime itself),
despite being a normal win32 python, still uses forward slashes. This
works fine for other cases (many, but not all), but when constructing a
raw NT path, all path separators must be backslashes.

Differential Revision: https://reviews.llvm.org/D71490
2019-12-17 10:08:53 +02:00
Julian Lettner 00ffa744f9 [lit] max_failures does not need to be stored in LitConfig 2019-12-16 10:08:57 -08:00
Julian Lettner cf252240e8 [lit] Small cleanups. NFCI
Remove unnecessary (argument same as default), cleanup imports, use
"pythonic" names for variables, and general formatting.
2019-12-13 17:00:04 -08:00
Julian Lettner f38b543b97 [lit] Improve formatting of error messages. NFC 2019-12-11 14:39:39 -08:00