We always want to use a deadline when calling `result.await`. Let's
synthesize an artificial deadline (positive infinity) to simplify code
and do less busy waiting.
llvm-svn: 375129
When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:
```
# RUN: program | diff file -
```
Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` doesn't
recognize `-` as a command-line option. This patch adds support for
`-` to mean stdin.
Reviewed By: probinson, rnk
Differential Revision: https://reviews.llvm.org/D67643
llvm-svn: 375116
When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:
```
# RUN: program | diff file -
# RUN: not diff file1 file2 | FileCheck %s
```
Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` cannot
currently be used in pipelines and doesn't recognize `-` as a
command-line option.
To enable pipelines, this patch moves lit's `diff` implementation into
an out-of-process script, similar to lit's `cat` implementation. A
follow-up patch will implement `-` to mean stdin.
Reviewed By: probinson, stella.stamenova
Differential Revision: https://reviews.llvm.org/D66574
llvm-svn: 375114
* Remove outdated precautions for Python versions < 2.7
* Remove dead code related to `maxIndividualTestTime` option
* Move printing of test and result summary out of main into its own
function
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D68847
llvm-svn: 375046
Using GNU diff, `--strip-trailing-cr` removes a `\r` appearing before
a `\n` at the end of a line. Without this patch, lit's internal diff
only removes `\r` if it appears as the last character. That seems
useless. This patch fixes that.
This patch also adds `--strip-trailing-cr` to some tests that fail on
Windows bots when D68664 is applied. Based on what I see in the bot
logs, I think the following is happening. In each test there, lit
diff is comparing a file with `\r\n` line endings to a file with `\n`
line endings. Without D68664, lit diff reads those files in text
mode, which in Windows causes `\r\n` to be replaced with `\n`.
However, with D68664, lit diff reads the files in binary mode instead
and thus reports that every line is different, just as GNU diff does
(at least under Ubuntu). Adding `--strip-trailing-cr` to those tests
restores the previous behavior while permitting the behavior of lit
diff to be more like GNU diff.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D68839
llvm-svn: 375020
As suggested by rnk at D67643#1673043, instead of reading files
multiple times until an appropriate encoding is found, read them once
as binary, and then try to decode what was read.
For Python >= 3.5, don't fail when attempting to decode the
`diff_bytes` output in order to print it.
Avoid failures for Python 2.7 used on some Windows bots by
transforming diff output with `lit.util.to_string` before writing it
to stdout.
Finally, add some tests for encoding handling.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D68664
llvm-svn: 375018
This essentially reverts a commit [1] that removed the adaptor for
Python unittests. The code has been slightly refactored to make it more
additive: all code is contained in LitTestCase.py.
Usage sites will require a small adaption:
```
[old]
import lit.discovery
...
test_suite = lit.discovery.load_test_suite(...)
[new]
import lit.LitTestCase
...
test_suite = lit.LitTestCase.load_test_suite(...)
```
This was put back on request by Daniel Dunbar, since I wrongly assumed
that the functionality is unused. At least llbuild still uses this [2].
[1] 70ca752ccf
[2] https://github.com/apple/swift-llbuild/blob/master/utils/Xcode/LitXCTestAdaptor/LitTests.py#L16
Reviewed By: ddunbar
Differential Revision: https://reviews.llvm.org/D69002
llvm-svn: 374947
The goal is to have 100% fidelity in clang-scan-deps behavior when
--analyze is present in compilation command.
At the same time I don't want to break clang-tidy which expects
__static_analyzer__ macro defined as built-in.
I introduce new cc1 options (-setup-static-analyzer) that controls
the macro definition and is conditionally set in driver.
Differential Revision: https://reviews.llvm.org/D68093
llvm-svn: 374815
When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:
```
# RUN: program | diff -U1 file -
```
Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` doesn't
recognize `-U` as a command-line option. This patch adds `-U`
support.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D68668
llvm-svn: 374814
On that decode, Windows bots fail with:
```
UnicodeEncodeError: 'ascii' codec can't encode characters in position 7-8: ordinal not in range(128)
```
That's the same error as before r374665 except it's now at the decode
before the write to stdout.
llvm-svn: 374666
I seem to have misread the bot logs on my last attempt. When lit's
internal diff runs on Windows under Python 2.7, it's text diffs not
binary diffs that need decoding to avoid this error when writing the
diff to stdout:
```
UnicodeEncodeError: 'ascii' codec can't encode characters in position 7-8: ordinal not in range(128)
```
There is no `decode` attribute in this case under Python 3.6.8 under
Ubuntu, so this patch checks for the `decode` attribute before using
it here. Hopefully nothing else is needed when `decode` isn't
available.
It might take a couple more attempts to figure out what error
handling, if any, is needed for this decoding.
llvm-svn: 374665
Based on the bot logs, when lit's internal diff runs on Windows, it
looks like binary diffs must be decoded also for Python 2.7.
Otherwise, writing the diff to stdout fails with:
```
UnicodeEncodeError: 'ascii' codec can't encode characters in position 7-8: ordinal not in range(128)
```
I did not need to decode using Python 2.7.15 under Ubuntu. When I do
it anyway in that case, `errors="backslashreplace"` fails for me:
```
TypeError: don't know how to handle UnicodeDecodeError in error callback
```
However, `errors="ignore"` works, so this patch uses that, hoping
it'll work on Windows as well.
This patch leaves `errors="backslashreplace"` for Python >= 3.5 as
there's no evidence yet that doesn't work and it produces more
informative binary diffs. This patch also adjusts some lit tests to
succeed for either error handler.
This patch adjusts changes introduced by D68664.
llvm-svn: 374657
Using GNU diff, `--strip-trailing-cr` removes a `\r` appearing before
a `\n` at the end of a line. Without this patch, lit's internal diff
only removes `\r` if it appears as the last character. That seems
useless. This patch fixes that.
This patch also adds `--strip-trailing-cr` to some tests that fail on
Windows bots when D68664 is applied. Based on what I see in the bot
logs, I think the following is happening. In each test there, lit
diff is comparing a file with `\r\n` line endings to a file with `\n`
line endings. Without D68664, lit diff reads those files with
Python's universal newlines support activated, causing `\r` to be
dropped. However, with D68664, lit diff reads the files in binary
mode instead and thus reports that every line is different, just as
GNU diff does (at least under Ubuntu). Adding `--strip-trailing-cr`
to those tests restores the previous behavior while permitting the
behavior of lit diff to be more like GNU diff.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D68839
llvm-svn: 374652
To avoid breaking some tests, D66574, D68664, D67643, and D68668
landed together. However, D68664 introduced an issue now addressed by
D68839, with which these are now all relanding.
Differential Revision: https://reviews.llvm.org/D68668
llvm-svn: 374651
To avoid breaking some tests, D66574, D68664, D67643, and D68668
landed together. However, D68664 introduced an issue now addressed by
D68839, with which these are now all relanding.
Differential Revision: https://reviews.llvm.org/D67643
llvm-svn: 374650
To avoid breaking some tests, D66574, D68664, D67643, and D68668
landed together. However, D68664 introduced an issue now addressed by
D68839, with which these are now all relanding.
Differential Revision: https://reviews.llvm.org/D68664
llvm-svn: 374649
To avoid breaking some tests, D66574, D68664, D67643, and D68668
landed together. However, D68664 introduced an issue now addressed by
D68839, with which these are now all relanding.
Differential Revision: https://reviews.llvm.org/D66574
llvm-svn: 374648
* Extract separate function for running tests from main
* Push single-usage imports to point of usage
* Remove unnecessary sys.exit(0) calls
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D68836
llvm-svn: 374602
This change is purely mechanical. I will do further cleanups of
parameter usages.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D68830
llvm-svn: 374452
Bring back `--threads` option which was lost in the move of the
command line argument parsing code to cl_arguments.py. Update docs
since `--workers` is preferred.
llvm-svn: 374432
When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:
```
# RUN: program | diff -U1 file -
```
Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` doesn't
recognize `-U` as a command-line option. This patch adds `-U`
support.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D68668
llvm-svn: 374392
When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:
```
# RUN: program | diff file -
```
Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` doesn't
recognize `-` as a command-line option. This patch adds support for
`-` to mean stdin.
Reviewed By: probinson, rnk
Differential Revision: https://reviews.llvm.org/D67643
llvm-svn: 374390
As suggested by rnk at D67643#1673043, instead of reading files
multiple times until an appropriate encoding is found, read them once
as binary, and then try to decode what was read.
For python >= 3.5, don't fail when attempting to decode the
`diff_bytes` output in order to print it.
Finally, add some tests for encoding handling.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D68664
llvm-svn: 374389
When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:
```
# RUN: program | diff file -
# RUN: not diff file1 file2 | FileCheck %s
```
Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` cannot
currently be used in pipelines and doesn't recognize `-` as a
command-line option.
To enable pipelines, this patch moves lit's `diff` implementation into
an out-of-process script, similar to lit's `cat` implementation. A
follow-up patch will implement `-` to mean stdin.
Reviewed By: probinson, stella.stamenova
Differential Revision: https://reviews.llvm.org/D66574
llvm-svn: 374388
Move progress display to separate file. Simplify some code paths.
Decouple from other components via progress callback. Remove unused
`_Display` class.
Reviewed By: serge-sans-paille
Differential Revision: https://reviews.llvm.org/D68525
llvm-svn: 374194
Lit has a "quiet" option, -q, which is documented to "suppress no
error output". Previously, LitConfig displayed notes and warnings when
the quiet option was specified. The result was that it was not
possible to get only pertinent file/line information to be used by an
editor to jump to the location where checks were failing without
passing a number of unhelpful locations first. Here, the
implementations of LitConfig.note and LitConfig.warning are modified
to account for the quiet flag and avoid displaying if the flag has
indeed been set.
Patch by Nate Chandler
Reviewed by yln
Differential Revision: https://reviews.llvm.org/D68044
llvm-svn: 374009
In the past, lit used threads to run tests in parallel. Today we use
`multiprocessing.Pool`, which uses processes. Let's stay more abstract
and use "worker" everywhere.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D68475
llvm-svn: 373794
When using lit's internal shell, RUN lines like the following
accidentally execute an external `diff` instead of lit's internal
`diff`:
```
# RUN: program | diff file -
# RUN: not diff file1 file2 | FileCheck %s
```
Such cases exist now, in `clang/test/Analysis` for example. We are
preparing patches to ensure lit's internal `diff` is called in such
cases, which will then fail because lit's internal `diff` cannot
currently be used in pipelines and doesn't recognize `-` as a
command-line option.
To enable pipelines, this patch moves lit's `diff` implementation into
an out-of-process script, similar to lit's `cat` implementation. A
follow-up patch will implement `-` to mean stdin.
Reviewed By: probinson, stella.stamenova
Differential Revision: https://reviews.llvm.org/D66574
llvm-svn: 372035
Without this patch, failing to provide a subcommand to lit's internal
`env` results in either a python `IndexError` or an attempt to execute
the final `env` argument, such as `FOO=1`, as a command. This patch
diagnoses those cases with a more helpful message.
Reviewed By: stella.stamenova
Differential Revision: https://reviews.llvm.org/D66482
llvm-svn: 369620
This patch extends lit's test suite to check that lit's internal shell
doesn't accidentally execute internal commands as external commands.
It does so by putting fake failing versions of those commands in
`PATH` while the entire lit test suite is running. Without the fixes
in D65697 but with its tests, this approach catches accidental
external `env` calls.
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D66293
llvm-svn: 369309
Summary:
'OSI Approved :: Apache-2.0 with LLVM exception' is not a valid
classifier. 'OSI Approved :: Apache Software License' is the closest
fit for the new license, so we've decided to use this one.
The classifiers seem to only be used for searching on the pypi website,
so this does not actually change the license of the code.
We still pass 'Apache-2.0 with LLVM exception' as the license to setup(),
and this appears alongside the classifier on the pypi webpage for lit.
Reviewers: chandlerc, ddunbar, joerg
Reviewed By: joerg
Subscribers: delcypher, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D65762
llvm-svn: 368315
Without this patch, the internal `env` command removes `env` and its
args from the command line while parsing it. This patch modifies a
copy instead so that the original command line is printed.
Reviewed By: stella.stamenova, rnk
Differential Revision: https://reviews.llvm.org/D65624
llvm-svn: 367752
Put the main test script in the right directory, and fix a python bug
in a local script.
Reviewed By: stella.stamenova
Differential Revision: https://reviews.llvm.org/D65623
llvm-svn: 367751
Summary:
This change updates the lit.cfg file to use llvm_config when it is available, but when it is not, it directly modifies the config object. This makes it possible to run the lit tests standalone without having built llvm (as long as the correct binaries are present in the path such as FileCheck and not).
Because the lit tests don't take a hard dependency on llvm_config, some features such as system-windows have to have definitions in lit's cfg file as well. This is a potential issue as the os features sometimes change names (for example, we went from windows to system-windows, etc.). This can cause drift between lit's tests and the rest of the llvm tests.
Reviewers: probinson, mgorny
Reviewed By: mgorny
Subscribers: delcypher, llvm-commits, asmith
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D65674
llvm-svn: 367730
Set environment variables to empty values rather than attempting
to unset them via 'env -u', in order to fix NetBSD test regression
caused by r366980. POSIX does not guarantee that env(1) supports '-u'
option, and indeed NetBSD env(1) does not support it.
Differential Revision: https://reviews.llvm.org/D65335
llvm-svn: 367123
lit's test suite calls lit multiple times for various sample test
suites. `FILECHECK_OPTS` is safe for FileCheck calls in lit's test
suite. It's not safe for FileCheck calls in the sample test suites,
whose output affects the results of lit's test suite.
Without this patch, only one such sample test suite is protected from
`FILECHECK_OPTS`, and I admit I haven't discovered other cases for
which I can produce false failures using `FILECHECK_OPTS`. However,
it's hard to predict the future, especially false passes. Thus, this
patch protects all existing and future sample test suites from
`FILECHECK_OPTS` (and the deprecated
`FILECHECK_DUMP_INPUT_ON_FAILURE`).
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D65156
llvm-svn: 366980
Summary:
On AIX psutil can run into problems with permissions to read the process
tree, which causes problems for python timeout tests which need to kill off
a test and it's children.
This patch adds a workaround by invoking shell via subprocess and using a
platform specific option to ps to list all the descendant processes so we can
kill them. We add some checks so lit can tell whether timeout tests are
supported with out exposing whether we are utilizing the psutil
implementation or the alternative.
Reviewers: hubert.reinterpretcast, andusy, davide, delcypher
Reviewed By: delcypher
Subscribers: davide, delcypher, christof, lldb-commits, libcxx-commits, llvm-commits
Tags: #lldb, #libc, #llvm
Differential Revision: https://reviews.llvm.org/D64251
llvm-svn: 366912
Summary:
This improves readability of LIT output: previously
error messages gets emitted that say that there was no error:
error: command reached timeout: False
Patch by Alexey Sachkov.
Reviewers: ddunbar, mgorny, modocache
Reviewed By: mgorny
Subscribers: delcypher, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D64240
llvm-svn: 365895
Similar to `FILECHECK_OPTS` for FileCheck, `LIT_OPTS` makes it easy to
adjust lit behavior when running the test suite via ninja. For
example:
```
$ LIT_OPTS='--time-tests -vv --filter=threadprivate' \
ninja check-clang-openmp
```
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D64135
llvm-svn: 365313
When running LLDB lit tests on Windows, the system selects a debug version
of Python, which was issuing lots of ResourceWarnings about files that
weren't closed. There are two kinds of them, and each test triggered one
of each.
This patch fixes one kind by ensuring TestRunner explicitly close the
temporary files created for routing stderr. This is important on Windows
but has no net effect on Posix systems.
The remaining ResourceWarnings are more elusive; the bug may lie in
the Python library subprocess.py, and it may be Windows-specific.
Differential Revision: https://reviews.llvm.org/D63102
llvm-svn: 363700
Ensure that the bash script written by lit TestRunner is open with UTF-8
encoding when using Python 3. Otherwise, attempt to write non-ASCII
characters causes UnicodeEncodeError. This happened e.g. with
the following LLD test:
UNRESOLVED: lld :: ELF/format-binary-non-ascii.s (657 of 2119)
******************** TEST 'lld :: ELF/format-binary-non-ascii.s' FAILED ********************
Exception during script execution:
Traceback (most recent call last):
File "/home/mgorny/llvm-project/llvm/utils/lit/lit/worker.py", line 63, in _execute_test
result = test.config.test_format.execute(test, lit_config)
File "/home/mgorny/llvm-project/llvm/utils/lit/lit/formats/shtest.py", line 25, in execute
self.execute_external)
File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1644, in executeShTest
res = _runShTest(test, litConfig, useExternalSh, script, tmpBase)
File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1590, in _runShTest
res = executeScript(test, litConfig, tmpBase, script, execdir)
File "/home/mgorny/llvm-project/llvm/utils/lit/lit/TestRunner.py", line 1157, in executeScript
f.write('{ ' + '; } &&\n{ '.join(commands) + '; }')
UnicodeEncodeError: 'ascii' codec can't encode character '\xa3' in position 274: ordinal not in range(128)
Differential Revision: https://reviews.llvm.org/D63254
llvm-svn: 363388
Summary:
This test fails to link shared libraries because tries to run
a copied version of clang-check to see if the mock version of libcxx
in the same directory can be loaded dynamically. Since the test is
specifically designed not to look in the default just-built lib
directory, it must be disabled when building with
BUILD_SHARED_LIBS=ON.
Currently only disabling it on Darwin and basing it on the
enable_shared flag.
Reviewed By: ilya-biryukov
Tags: #clang, #llvm
Differential Revision: https://reviews.llvm.org/D61697
llvm-svn: 363298
In LLDB, where tests run with the debug version of Python, we get a
series of deprecation warnings because escape sequences like `\(` are
being treated as part of the string literal rather than an escape for
the regexp pattern.
NFC intended.
Differential Revision: https://reviews.llvm.org/D62882
llvm-svn: 362846
Summary: This also normalizes the config feature that represents the windows platform to "system-windows" as opposed to having both "windows" and "system-windows"
Reviewers: asmith, probinson
Subscribers: delcypher, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D61798
llvm-svn: 361998
zlib/nozlib, asan/not_asan, msan/not_msan, ubsan/not_ubsan.
We still have two other ways to express the absence of a feature.
First, we have the '!' operator to invert the sense of a keyword. For
example, given a feature that depends on zlib being unavailable, its
test can say:
REQUIRES: !zlib
Second, if a test doesn't play well with some features, such as
sanitizers, that test can say:
UNSUPPORTED: asan, msan
The different ways of writing these exclusions both have the same
technical effect, but have different implications to the reader.
llvm-svn: 360603
Summary:
Various tests in the `lit` testing suite expect specific return codes
and forms of diagnostic message from utility programs. As per
POSIX.1-2017 XCU Section 1.4, Utility Description Defaults, "[the]
format of diagnostic messages for most utilities is unspecified".
The STDERR subsections of the `cat` and `wc` utilities merely indicate
that "[the] standard error shall be used only for diagnostic messages".
The corresponding EXIT STATUS subsections merely indicate, with regard
to errors, an exit value of >0.
The affected tests are updated to accept the applicable diagnostic
message as produced by the utilities on AIX. The exit value is
normalized using `not` as necessary.
Reviewers: xingxue, sfertile, jasonliu
Reviewed By: xingxue
Subscribers: delcypher, jsji, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D60553
llvm-svn: 359690
Add a 'target-x86' and 'target-x86_64' feature sthat indicates that
the default target is 32-bit or 64-bit x86, appropriately. Combined
with 'native' feature, we're going to use this to control x86-specific
LLDB native process tests.
Differential Revision: https://reviews.llvm.org/D60474
llvm-svn: 358177
Use ctypes to call into SHFileOperationW with the extended NT path to allow us
to remove paths which exceed 261 characters on Windows. This functionality is
exercised by swift's test suite.
llvm-svn: 357778
This enables lit to work with unicode file names via mkdir, rm, and redirection.
Lit still uses utf-8 internally, but converts to utf-16 on Windows, or just utf-8
bytes on everything else.
Committed on behalf of Jason Mittertreiner
Differential Revision: https://reviews.llvm.org/D56754
llvm-svn: 355122
Check that we do not crash if a parallelism group is explicitly set to
None. Permits usage of the following pattern.
[lit.common.cfg]
lit_config.parallelism_groups['my_group'] = None
if <condition>:
lit_config.parallelism_groups['my_group'] = 3
[project/lit.cfg]
config.parallelism_group = 'my_group'
Reviewers: rnk
Differential Revision: https://reviews.llvm.org/D58305
llvm-svn: 354912
From the docs: `class LitTestCase(unittest.TestCase)`
LitTestCase is an adaptor for providing a 'unittest' compatible
interface to 'lit' tests so that we can run lit tests with standard
python test runners.
It does not seem to be used anywhere.
Differential Revision: https://reviews.llvm.org/D58264
llvm-svn: 354188
Move code that is executed on worker process to separate file. This
makes the use of the pickled arguments stored in global variables in the
worker a bit clearer. (Still not pretty though.)
Extract handling of parallelism groups to it's own function.
Use BoundedSemaphore instead of Semaphore. BoundedSemaphore raises for
unmatched release() calls.
Cleanup imports.
Differential Revision: https://reviews.llvm.org/D58196
llvm-svn: 354187
Summary:
Automatically upgrade debugging experience (single process, no thread
pool) when:
1) we only run a single test
2) user specifies `-j1`
Details:
Fix `--max-failures` in single process mode. Option did not have an
effect in single process mode.
Add display feedback for single process mode. Adapted test.
Improve argument checking (require positive integers).
`--single-process` is now essentially an alias for `-j1`. Should we
remove it?
Reviewers: rnk
Differential Revision: https://reviews.llvm.org/D58249
llvm-svn: 354068
LLVMConfig.with_environment() uses os.path.normcase(os.path.normpath(x)) to
normalize temporary env vars. LLVMConfig.use_clang() uses with_environment() to
temporarily set PATH and then look for clang there. This means that on Windows,
clang will be run with a path like c:\foo\bin\clang.EXE (with a lower-case
"C:").
lit.util.which() used to not do this, which means the executables added in
clang/test/lit.cfg.py (e.g. c-index-test) were run with a path like
C:\foo\bin\c-index-test.EXE (because both CMake and GN happen to write
clang_tools_dir with an upper-case C to lit.site.cfg.py).
clang/test/Index/pch-from-libclang.c requires that both c-index-test and clang
use _exactly_ the same resource dir path (same case and everything), because a
hash of the resource directory is used as module cache path.
This patch is necessary but not sufficient to make pch-from-libclang.c pass on
Windows.
Differential Revision: https://reviews.llvm.org/D57343
llvm-svn: 352704
This fixes most references to the paths:
llvm.org/svn/
llvm.org/git/
llvm.org/viewvc/
github.com/llvm-mirror/
github.com/llvm-project/
reviews.llvm.org/diffusion/
to instead point to https://github.com/llvm/llvm-project.
This is *not* a trivial substitution, because additionally, all the
checkout instructions had to be migrated to instruct users on how to
use the monorepo layout, setting LLVM_ENABLE_PROJECTS instead of
checking out various projects into various subdirectories.
I've attempted to not change any scripts here, only documentation. The
scripts will have to be addressed separately.
Additionally, I've deleted one document which appeared to be outdated
and unneeded:
lldb/docs/building-with-debug-llvm.txt
Differential Revision: https://reviews.llvm.org/D57330
llvm-svn: 352514
Since these are intended to be short and succinct, I've used the SPDX
full name. It's human readable, but formally agreed upon and will be
part of the SPDX spec for licenses.
llvm-svn: 351649
If a user has PYTHONPATH set in the environment, append new entries to
it rather than blindly setting PYTHONPATH to a fixed string. This
allows tests to, for example, find psutil if it is in
PYTHONPATH. Without this change, lit will detect psutil but then
various tests will fail because PYTHONPATH has been overwritten and
psutil cannot be found.
llvm-svn: 350536
Make sure all print statements are compatible with Python 2 and Python3 using
the `from __future__ import print_function` statement.
Differential Revision: https://reviews.llvm.org/D56249
llvm-svn: 350307
Adds a build file for clang-tblgen and an action for running it, and uses that
to process all the .td files in include/clang/Basic.
Also adds an action to write include/clang/Config/config.h and
include/clang/Basic/Version.inc.
Differential Revision: https://reviews.llvm.org/D55847
llvm-svn: 349677
Split timestamp preservation tests into atime and mtime test, and skip
the former on NetBSD. When the filesystem is mounted noatime, NetBSD
not only inhibits implicit atime updates but also prevents setting atime
via utime(), causing the test to fail.
Differential Revision: https://reviews.llvm.org/D55271
llvm-svn: 348354
This adds a script called build.py as well as a lit substitution
called %build that we can use to invoke it. The idea is that
this allows a lit test to build test inferiors without having
to worry about architecture / platform specific differences,
command line syntax, finding / configurationg a proper toolchain,
and other issues. They can simply write something like:
%build --arch=32 -o %t.exe %p/Inputs/foo.cpp
and it will just work. This paves the way for being able to
run lit tests with multiple configurations, platforms, and
compilers with a single test.
Differential Revision: https://reviews.llvm.org/D54914
llvm-svn: 348058
This arose when I was trying to have a substitution which invoked a
python script P, and that python script tried to invoke clang-cl (or
even cl). Since we invoke P with a custom environment, it doesn't
inherit the environment of the parent, and then when we go to invoke
clang-cl, it's unable to find the MSVC installation directory. There
were many more I could have passed through which are set by vcvarsall,
but I tried to keep it simple and only pass through the important ones.
Differential Revision: https://reviews.llvm.org/D54963
llvm-svn: 347691
Otherwise, the clang analyzer tests fail on Windows when attempting to
unpickle AnalyzerTest objects in the worker processes. The pattern of,
add to path, import, remove from path, serialize, deserialize, doesn't
work. Once something gets added to the path, if we want to move it
across the wire for multiprocessing, we need to keep the module on
sys.path.
llvm-svn: 347254
Recently I tried to port LLDB's lit configuration files over to use a
on the surface, but broke some cases that weren't broken before and also
exposed some additional problems with the old approach that we were just
getting lucky with.
When we set up a lit environment, the goal is to make it as hermetic as
possible. We should not be relying on PATH and enabling the use of
arbitrary shell commands. Instead, only whitelisted commands should be
allowed. These are, generally speaking, the lit builtins such as echo,
cd, etc, as well as anything for which substitutions have been
explicitly set up for. These substitutions should map to the build
output directory, but in some cases it's useful to be able to override
this (for example to point to an installed tools directory).
This is, of course, how it's supposed to work. What was actually
happening is that we were bringing in PATH and LD_LIBRARY_PATH and then
just running the given run line as a shell command. This led to problems
such as finding the wrong version of clang-cl on PATH since it wasn't
even a substitution, and flakiness / non-determinism since the
environment the tests were running in would change per-machine. On the
other hand, it also made other things possible. For example, we had some
tests that were explicitly running cl.exe and link.exe instead of
clang-cl and lld-link and the only reason it worked at all is because it
was finding them on PATH. Unfortunately we can't entirely get rid of
these tests, because they support a few things in debug info that
clang-cl and lld-link don't (notably, the LF_UDT_MOD_SRC_LINE record
which makes some of the tests fail.
The high level changes introduced in this patch are:
1. Removal of functionality - The lit test suite no longer respects
LLDB_TEST_C_COMPILER and LLDB_TEST_CXX_COMPILER. This means there is no
more support for gcc, but nobody was using this anyway (note: The
functionality is still there for the dotest suite, just not the lit test
suite). There is no longer a single substitution %cxx and %cc which maps
to <arbitrary-compiler>, you now explicitly specify the compiler with a
substitution like %clang or %clangxx or %clang_cl. We can revisit this
in the future when someone needs gcc.
2. Introduction of the LLDB_LIT_TOOLS_DIR directory. This does in spirit
what LLDB_TEST_C_COMPILER and LLDB_TEST_CXX_COMPILER used to do, but now
more friendly. If this is not specified, all tools are expected to be
the just-built tools. If it is specified, the tools which are not
themselves being tested but are being used to construct and run checks
(e.g. clang, FileCheck, llvm-mc, etc) will be searched for in this
directory first, then the build output directory.
3. Changes to core llvm lit files. The use_lld() and use_clang()
functions were introduced long ago in anticipation of using them in
lldb, but since they were never actually used anywhere but their
respective problems, there were some issues to be resolved regarding
generality and ability to use them outside their project.
4. Changes to .test files - These are all just replacing things like
clang-cl with %clang_cl and %cxx with %clangxx, etc.
5. Changes to lit.cfg.py - Previously we would load up some system
environment variables and then add some new things to them. Then do a
bunch of work building out our own substitutions. First, we delete the
system environment variable code, making the environment hermetic. Then,
we refactor the substitution logic into two separate helper functions,
one which sets up substitutions for the tools we want to test (which
must come from the build output directory), and another which sets up
substitutions for support tools (like compilers, etc).
6. New substitutions for MSVC -- Previously we relied on location of
MSVC by bringing in the entire parent's PATH and letting
subprocess.Popen just run the command line. Now we set up real
substitutions that should have the same effect. We use PATH to find
them, and then look for INCLUDE and LIB to construct a substitution
command line with appropriate /I and /LIBPATH: arguments. The nice thing
about this is that it opens the door to having separate %msvc-cl32 and
%msvc-cl64 substitutions, rather than only requiring the user to run
vcvars first. Because we can deduce the path to 32-bit libraries from
64-bit library directories, and vice versa. Without these substitutions
this would have been impossible.
Differential Revision: https://reviews.llvm.org/D54567
llvm-svn: 347216
This feature makes it easy to tune FileCheck diagnostic output when
running the test suite via ninja, a bot, or an IDE. For example:
```
$ FILECHECK_OPTS='-color -v -dump-input-on-failure' \
LIT_FILTER='OpenMP/for_codegen.cpp' ninja check-clang \
| less -R
```
Reviewed By: probinson
Differential Revision: https://reviews.llvm.org/D53517
llvm-svn: 346272
A year or so ago, I re-wrote most of the lit infrastructure in LLVM so
that it wasn't so boilerplate-y. I added lots of common helper type
stuff, simplifed usage patterns, and made the code more elegant and
maintainable.
We migrated to this in LLVM, clang, and lld's lit files, but not in
LLDBs. This started to bite me recently, as the 4 most recent times I
tried to run the lit test suite in LLDB on a fresh checkout the first
thing that would happen is that python would just start crashing with
unhelpful backtraces and I would have to spend time investigating.
You can reproduce this today by doing a fresh cmake generation, doing
ninja lldb and then python bin/llvm-lit.py -sv ~/lldb/lit/SymbolFile at
which point you'll get a segfault that tells you nothing about what your
problem is.
I started trying to fix the issues with bandaids, but it became clear
that the proper solution was to just bring in the work I did in the rest
of the projects. The side benefit of this is that the lit configuration
files become much cleaner and more understandable as a result.
Differential Revision: https://reviews.llvm.org/D54009
llvm-svn: 346008
Some versions of bash.exe, for example WSL's version expect paths in the form
/mnt/c/path/to/dir rather than c:\\path\\to\\dir so will cause failures
for any tests that require an external shell if used by lit. If we're on
Windows and looking for an external shell, check that the found version
of bash is able to parse a native path before returning that version.
This patch also partially reverts the behaviour of r228221 by
restoring the warning if bash cannot be found. This shouldn't pollute
the lit stderr anymore as we're now using internal shell by default on
Windows. If someone is explicitly specifying to use an external shell, it's
probably worth alerting them to the fact that bash could not be found.
Differential Revision: https://reviews.llvm.org/D52831
llvm-svn: 345019
Summary: This directory was missing from the lit package on pypi.org.
Reviewers: ddunbar
Subscribers: delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D51670
llvm-svn: 343115
Summary:
The multiprocess module uses pickling to transfer
information between processes and does not know how to pickle
the class created in the lit.cfg file and thus the example
fails.
Implement ManyTests in a separate file and import for the
example test passes
Patch by Nathan Lanza <nathan@lanza.io>
Differential Revision: https://reviews.llvm.org/D51328
llvm-svn: 342269
Summary: This is the only test that is still failing on Windows - or rather, it is expected to fail on the bots, but passes on the new bot that we're preparing causing a failure, so I'm going to disable it. Since the test has rarely, if ever, passed on the bots, this should have the same effect and it will unblock the creation of the new bot.
Reviewers: asmith, delcypher, zturner
Subscribers: stella.stamenova, llvm-commits
Differential Revision: https://reviews.llvm.org/D51871
llvm-svn: 341856
Summary:
The python executable may not exist on all systems so use sys.executable
instead.
Reviewers: ddunbar, stella.stamenova
Subscribers: delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D51511
llvm-svn: 341244
Summary:
Right now this test is failing on the builtbots on Windows but we have a very similar setup where the test passes. The test is meant to test that specifying a timeout works correctly by running an infnite loop and having it timeout - on the buildbot, the infinite loop doesn't actually execute. This change runs all of the tests in the set using an internal shell rather than an external shell. I expect this will make the test pass which means that either the way the external shell is invoked or the external shell setup on the buildbots is not correct. Regardless of whether the test passes with this change, we'll need to undo this change and have a real fix.
@gkistanova was able to get logs from the buildbot to rule out a number of theories as to why this test is failing, but they didn't have enough information to confirm exactly what the issue is. The purpose of this change is to narrow it down, but if someone has a local repro and can aid in debugging, that would make it much speedier (and less prone to making the bots fail).
Reviewers: gkistanova, asmith, zturner, modocache, rnk, delcypher
Reviewed By: rnk
Subscribers: delcypher, llvm-commits, gkistanova
Differential Revision: https://reviews.llvm.org/D51326
llvm-svn: 340840
Changes the default Windows target triple returned by
GetHostTriple.cmake from the old environment names (which we wanted to
move away from) to newer, normalized ones. This also requires updating
all tests to use the new systems names in constraints.
Differential Revision: https://reviews.llvm.org/D47381
llvm-svn: 339307
This test passes on Windows when using Python 3 but fails when using Python 2, so it needs more investigation before it can be enabled as the bots use Python 2.
llvm-svn: 339184
Summary:
In Python2 'unicode' is a distinct type from 'str', but in Python3 'unicode' does not exist and instead all 'str' objects are Unicode string. This change updates the logic in the test logging for lit to correctly process each of the types, and more importantly, to not just fail in Python3.
This change also reverses the use of quotes in several of the cfg files. By using '""' we are guaranteeing that the resulting path will work correctly on Windows while "''" only works correctly sometimes. This also fixes one of the failing tests.
Reviewers: asmith, zturner
Subscribers: stella.stamenova, delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D50397
llvm-svn: 339179
Summary:
The problem here is that on windows double quotes are used for paths (usually) while single quotes are not. This is not generally a problem for the tests because the lit infrastructure tends to treat both the same. One (and possibly only) exception is when some tests are run in an external shell such as some of the shtest-format tests. In this case on windows the path to python was not created correctly because it had single quotes and the test failed.
This same test is already failing with python 3 which is why our testing missed the new failure. This patch will take care of the immediate failure with python 2 and I'll send a follow up for the python 3 failure.
Reviewers: asmith, zturner
Subscribers: delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D50373
llvm-svn: 339091
Summary:
The issue with the python path is that the path to python on Windows can contain spaces. To make the tests always work, the path to python needs to be surrounded by quotes.
This change updates several configuration files which specify the path to python as a substitution and also remove quotes from existing tests.
Reviewers: asmith, zturner, alexshap, jakehehrlich
Reviewed By: zturner, alexshap, jakehehrlich
Subscribers: mehdi_amini, nemanjai, eraman, kbarton, jakehehrlich, steven_wu, dexonsmith, stella.stamenova, delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D50206
llvm-svn: 339073
These two tests are operating on the same test suite, which causes
them to be racy about writing temporary files and can cause spurious
failures. Merge them into one test to avoid the issue.
llvm-svn: 337718
The test was failing on Windows machines which had bash.exe on PATH (but
not in the so called lit tools dir, containing cmp.exe, grep.exe etc.).
The problem was that the outer lit invocation would load LLVMConfig
from utils/lit/lit/llvm/config.py, which looks up the tools path with
getToolsPath(). That has a surprising side effect of also setting
bashPath, in our case setting it to empty.
The outer lit invocation would thus configure the pdbg0 and pdbg1
substitutions based on not running with bash.
But the inner lit invocation would not load LLVMConfig, so bash
would be found on PATH, that would be used as external shell,
and so the output wouldn't match pdbg0 and pdbg1.
It seems weird to me that getBashPath() will return different results
depending on whether getToolsPath() has been called before, but I
also don't know how to fix it properly.
This commit just relaxes the test case, because there doesn't seem
to be much point in testing for the exact syntax of the run file
as long as it works.
(See https://crbug.com/850023)
llvm-svn: 334100
Summary:
The '%analyze' extra_args config argument seems to have been erroneously
deleted in r315627 disabling Z3 tests for the clang analyzer. Add the
flag back.
Reviewers: george.karpenkov, NoQ, ddcc
Reviewed By: george.karpenkov
Subscribers: xazax.hun, szepet, delcypher, a.sidorin, llvm-commits
Differential Revision: https://reviews.llvm.org/D47722
llvm-svn: 334066
This fixes projects/compiler-rt/test/fuzzer/sigusr.test, which was
broken by r333614. The trouble was that "&&" changes the command for
which "$!" gives the pid.
llvm-svn: 333620
(Relands r333584, reverted in 333592.)
When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.
When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.
To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.
To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files. (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)
Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk
Differential Revision: https://reviews.llvm.org/D44598
llvm-svn: 333614
(Relands r330755 (reverted in r330848) with fix for PR37239.)
When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.
When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.
To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.
To support reporting RUN line numbers in the case of windows cmd.exe
as the external shell, this patch extends -vv to set "echo on" instead
of "echo off" in bat files. (Support for windows cmd.exe as a lit
external shell will likely be dropped later, but I found out too
late.)
Reviewed By: delcypher, asmith, stella.stamenova, jmorse, lebedev.ri, rnk
Differential Revision: https://reviews.llvm.org/D44598
llvm-svn: 333584
We were making malformed XML on tests with ' in the name. Switch to
using saxutils to set all of our attributes, so it can handle quotes
etc correctly.
llvm-svn: 333249
larger timeout value. This really isn't very good because it will
still be susceptible to machine performance.
While we are here also fix a bug in validation of
`maxIndividualTestTime` where previously it wasn't checked if the
type was an int.
rdar://problem/40221572
llvm-svn: 332987
The program used to be used in `quick_then_slow.py` but that was
removed in r328702. The tests always run `slow.py` on its own but
this doesn't really test additional code so we'll just drop running
`slow.py` so the tests run faster.
rdar://problem/40221572
llvm-svn: 332986
If the system is under heavy load 1 second might not be long enough
for it to produce output which could lead to spurious test failures.
What matters is that the right test cases reach a timeout.
rdar://problem/40221572
llvm-svn: 332985
Summary:
This sequence ends the CDATA block so any characters after that are no
longer escaped. This can be fixed by replacing "]]>" with "]]]]><
Julian Lettner