We were making malformed XML on tests with ' in the name. Switch to
using saxutils to set all of our attributes, so it can handle quotes
etc correctly.
llvm-svn: 333249
larger timeout value. This really isn't very good because it will
still be susceptible to machine performance.
While we are here also fix a bug in validation of
`maxIndividualTestTime` where previously it wasn't checked if the
type was an int.
rdar://problem/40221572
llvm-svn: 332987
The program used to be used in `quick_then_slow.py` but that was
removed in r328702. The tests always run `slow.py` on its own but
this doesn't really test additional code so we'll just drop running
`slow.py` so the tests run faster.
rdar://problem/40221572
llvm-svn: 332986
If the system is under heavy load 1 second might not be long enough
for it to produce output which could lead to spurious test failures.
What matters is that the right test cases reach a timeout.
rdar://problem/40221572
llvm-svn: 332985
Summary:
This sequence ends the CDATA block so any characters after that are no
longer escaped. This can be fixed by replacing "]]>" with "]]]]><![CDATA[>".
Reviewers: cmatthews
Reviewed By: cmatthews
Differential Revision: https://reviews.llvm.org/D46886
llvm-svn: 332440
We were reporting "Unsupported" tests in xunit as passes, however since
they are not run, it make more sense to mark them as skipped. The Junit
xml standard has support for that, so lets use it.
llvm-svn: 332065
String concatenation in python is slow. Refactor to not concatenate the
possibly large strings of test output and instead write them directly
to the output file.
llvm-svn: 332064
Lit creates malformed xml when the test case has an & in the name.
Escape those correctly.
This also adds a test case which I will add other nasty encoding issues to in some followup commits.
llvm-svn: 331942
Its only two uses were removed in r311730.
Effectively reverts r304851 (but that code has removed around a bit since then).
https://reviews.llvm.org/D46619
clang side done in r331871.
llvm-svn: 331872
Add overloads for `__len__` and `__getitem__` to allow use of this class
on Linux as well as Windows. With these overloads, lit can be used on
both hosts for the swift testsuite.
llvm-svn: 331431
When debugging test failures with -vv (or -v in the case of the
internal shell), this makes it easier to locate the RUN line that
failed. For example, clang's test/Driver/linux-ld.c has 892 total RUN
lines, and clang's test/Driver/arm-cortex-cpus.c has 424 RUN lines
after concatenation for line continuations.
When reading the generated shell script, this also makes it easier to
locate the RUN line that produced each command.
To support reporting RUN line numbers in the case of the internal
shell, this patch extends the internal shell to support the null
command, ":", except pipelines are not supported.
Reviewed By: asmith, delcypher
Differential Revision: https://reviews.llvm.org/D44598
llvm-svn: 330755
`shtest-xunit-output.py` test.
Although there is no `-` file Jeremy Morse has reported to me that it
causes problems in their setup because lit tries to find it and ends up
loading an out of tree lit configuration file.
llvm-svn: 330728
The test is apparently needed e.g. for check-cfi on Windows where we get
'C:/b/slave/sanitizer-windows/build/./bin/clang.exe': command not found
without it. Try to fix the problem that was fixed by r330672 by also checking
for isabs() instead.
llvm-svn: 330673
lit's util.which() would check if the passed-in path existed directly,
and if so return it as-is. This is never the case when running llvm's, clang's,
or lld's tests normally. But when running `./llvm-lit path/to/clang/test`
with a cwd of llvm-build/bin, this if would detect that clang exists at path
'clang' and return 'clang' as the discovered clang binary -- and then lit would
use the " clang " -> "*** Do not use 'clang' in tests, use '%clang'. ***"
substitution to replace that with a broken test. By removing this early
return, lit ends up with the usual absolute path and everything works even
in this uncommon case.
llvm-svn: 330672
XML printer.
A test has been added that tries to comprehensively test emitting
XUnit XML output for shell tests.
Differential Revision: https://reviews.llvm.org/D45567
llvm-svn: 330409
There are two versions of to_string used by TestRunner.py. The one defined
in TestRunner.py and the one defined in utils/lit/lit/util.py. The util.py
version is superior to the TestRunner.py version.
This change removes the duplicate to_string in TestRunner.py in favor of
always using the version from util.py. Beside removing duplicate code, this
makes it easier to debug TestRunner.py since only one version of to_string
is used.
Patch by Stella Stamenova!
llvm-svn: 329972
- In Python 2.x, basestring is the base string type, but in
Python 3.x basestring is not defined and instead str includes
unicode strings.
- When Python is in a path that includes spaces, it needs to
be specified with quotes in the test files for it to run.
- The cache.ll test relies on files of a specific size being
created by Python, but on some versions of Windows the
files that are created by the current code are one byte
larger than expected. To fix the test, update file creation
to always make files of the expected size.
Patch by Stella Stamenova!
llvm-svn: 329466
LLVM Bug Id : 36449
Revision 328563 caused tests to fail under python 3.
This patch modified cat.py file to support both python 2 and 3.
This patch also fixes CRLF issues on Windows.
Patch by Chamal de Silva
Differential Revision: https://reviews.llvm.org/D45077
llvm-svn: 329123
Only rely on Python 3 (io.open) when necessary. This puts TestRunnyer.py closer to how it behaved
before the changes introduced in D43165 and silences a few Windows build bot failures.
Thanks to Stella Stamenova for the patch!
llvm-svn: 329037
Reapply D43165 which was reverted because of different versions of python failing.
The one line fix for the different python versions was commited at the same time
that D43165 was reverted. If this change is giving you issues then get in touch
with your python version and we will fix it.
llvm-svn: 329022
Summary:
This issue was found when running the clang unit test on Windows. Python 3.x cannot open some of the files that the tests are using with a simple open because of their encoding. Python 2.7+ and Python 3.x both support io.open which allows for an encoding to be specified.
This change will determine whether two files being compared should be opened (and then compared) as text or binary and whether to use utf-8 or the default encoding before proceeding with a line-by-line comparison.
Patch by Stella Stamenova!
Reviewers: zturner, llvm-commits, rnk, MaggieYi
Reviewed By: zturner
Subscribers: asmith, MatzeB, stella.stamenova, delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D43165
llvm-svn: 329012
This reverts commit 771829b640a5494ab65c810dd6b4330522bf3a33 (rr328598)
Hopefully the test will now pass on the bots.
rdar://problem/38774530
llvm-svn: 328703
The `shtest-timeout.py` test was failing intermittently. It looks like
the issue is that on a resource constrained system lit is unable to run
`quick_then_slow.py` twice and print out the messages the tests expects
within the one second timeout.
The underlying issue is that the test is dependent on the performance of
the host machine is a rather fragile way. This is due to hardcoding
timeout values and having assumptions that the host machine is able to
perform a certain amount of work within the hardcoded timeout values.
We could increase the timeout values but that doesn't really fix the
underlying issue. Instead this patch removes one of fragile assumptions
in the hope that this will be enough to fix the bots.
There are other fragile assumptions in this test (e.g. `quick.py` can be
executed in less than 1 second). If the bots continue to fail we'll have
to revisit this.
rdar://problem/38774530
llvm-svn: 328702
Summary:
This reverts commit r328596.
Checking if the arguments are strings before testing if they contain "/dev/null".
Reviewers: rnk
Reviewed By: rnk
Subscribers: delcypher, llvm-commits
Differential Revision: https://reviews.llvm.org/D44914
llvm-svn: 328603
Summary:
These changes are to allow to a Result object to have nested Result objects in
order to support microbenchmarks. Currently lit is restricted to reporting one
result object for one test, this change provides support tests that want to
report individual timings for individual kernels.
This revision is the result of the discussions in
https://reviews.llvm.org/D32272#794759,
https://reviews.llvm.org/D37421#f8003b27 and https://reviews.llvm.org/D38496.
It is a separation of the changes purposed in https://reviews.llvm.org/D40077.
This change will enable adding LCALS (Livermore Compiler Analysis Loop Suite)
collection of loop kernels to the llvm test suite using the google benchmark
library (https://reviews.llvm.org/D43319) with tracking of individual kernel
timings.
Previously microbenchmarks had been handled by using macros to section groups
of microbenchmarks together and build many executables while still getting a
grouped timing (MultiSource/TSVC). Recently the google benchmark library was
added to the test suite and utilized with a litsupport plugin. However the
limitation of 1 test 1 result limited its use to passing a runtime option to
run only 1 microbenchmark with several hand written tests
(MicroBenchmarks/XRay). This runs the same executable many times with different
hand-written tests. I will update the litsupport plugin to utilize the new
functionality (https://reviews.llvm.org/D43316).
These changes allow lit to report micro test results if desired in order to get
many precise timing results from 1 run of 1 test executable.
Reviewers: MatzeB, hfinkel, rengolin, delcypher
Differential Revision: https://reviews.llvm.org/D43314
llvm-svn: 327422
On Windows, if the substitution contains a back reference, it would
removed due to the replacement of the escape character in lit. Create a
helper class to avoid this which will simply ignore the replacement and
mark the substitution as having capture groups being referenced.
llvm-svn: 327082
We are running lld tests with "--full-shutdown" option because we don't
want to call _exit() in lld if it is running tests. Regular shutdown
is needed for leak sanitizer.
This patch changes the way how we tell lld that it is running tests.
Now "--full-shutdown" is removed, and LLD_IN_TEST environment variable
is used instead.
This patch enables full shutdown on all ports, e.g. ELF, COFF and wasm.
Previously, we enabled it only for ELF.
Differential Revision: https://reviews.llvm.org/D43410
llvm-svn: 325413
Allow CLANG environment variable be copied into the testing configuration
and proper support testing with a custom path to the clang executable.
Signed-off-by: Vladimir Vereschaka <vvereschaka@accesssoftek.com>
llvm-svn: 324706
Summary:
That would allow to recursively compare directories in tests using
"diff -r" on Windows in a similar way as it can be done on Linux or Mac.
Reviewers: zturner, morehouse, vsk
Reviewed By: zturner
Subscribers: kcc, llvm-commits
Differential Revision: https://reviews.llvm.org/D41776
llvm-svn: 322102
The variable named `minor` was actually pointing to the patch part of
the version. While I was changing this I also made the check for Apple
clang more robust by checking both patch and minor rather than just
minor.
llvm-svn: 319656
The latest clang that ships with Xcode (clang 900 or 9.0.0) does not
support LSan. This fixes the lit configuration to reflect that.
Differential revision: https://reviews.llvm.org/D40672
llvm-svn: 319530
This is still breaking greendragon.
At this point I give up until someone can fix the greendragon
bots, and I will probably abandon this effort in favor of using
a private github repository.
llvm-svn: 318722
This was reverted due to the tests being run twice on some
build bots. Each run had a slightly different configuration
due to the way in which it was being invoked. This fixes
the problem (albeit in a somewhat hacky way). Hopefully in
the future we can get rid of the workflow of running
debuginfo-tests as part of clang, and then this hack can
go away.
llvm-svn: 318697
Summary:
Currently, LIT configures the LLVM binary path before the Clang binary path. However this breaks testing out-of-tree Clang builds (where the LLVM binary path includes a copy of Clang).
This patch reverses the order of the paths when looking for Clang, putting the Clang binary directory first.
Reviewers: zturner, beanz, chapuni, modocache, EricWF
Reviewed By: EricWF
Subscribers: mgorny, cfe-commits, llvm-commits
Differential Revision: https://reviews.llvm.org/D40217
llvm-svn: 318607
This is still broken because it causes certain tests to be
run twice with slightly different configurations, which is
wrong in some cases.
You can observe this by running:
ninja -nv check-all | grep debuginfo-tests
And seeing that it passes clang/test and clang/test/debuginfo-tests
to lit, which causes it to run debuginfo-tests twice. The fix is
going to involve either:
a) figuring out that we're running in this "deprecated" configuration,
and then deleting the clang/test/debuginfo-tests path, which should
cause it to behave identically to before, or:
b) make lit smart enough that it doesn't descend into a sub-suite if
that sub-suite already has a lit.cfg file.
llvm-svn: 318486
This was reverted due to some failures on specific darwin buildbots,
the issue being that the new lit configuration was not setting the
SDKROOT environment variable. We've tested a fix locally and confirmed
that it works, so this patch resubmits everything with the fix
applied.
llvm-svn: 318435
This reverts the aforementioned patch and 2 subsequent follow-ups,
as some buildbots are still failing 2 tests because of it.
Investigation is ongoing into the cause of the failures.
llvm-svn: 318112
Previously, debuginfo-tests was expected to be checked out into
clang/test and then the tests would automatically run as part of
check-clang. This is not a standard workflow for handling
external projects, and it brings with it some serious drawbacks
such as the inability to depend on things other than clang, which
we will need going forward.
The goal of this patch is to migrate towards a more standard
workflow. To ease the transition for build bot maintainers,
this patch tries not to break the existing workflow, but instead
simply deprecate it to give maintainers a chance to update
the build infrastructure.
Differential Revision: https://reviews.llvm.org/D39605
llvm-svn: 317925
I need a test that only runs in a reasonable amount of time on systems
that have sparse files. The broadest class of systems that support
sparse files are linux systems. So restricting my test to linux systems
should suffice. This change adds the system-linux feature to llvm-lit so
that it can be required.
Differential Revision: https://reviews.llvm.org/D39482
llvm-svn: 317055
The new scheme should match the normalization of embedded paths in
linkrepro tar files.
Differential Revision: https://reviews.llvm.org/D39023
llvm-svn: 316044
"No such file or directory: C:\\...\\tests\\Output\\shared-output.py.tmp/Output/Shared/SHARED.tmp"
And yet other forward-slashes don't seem to be causing the same
problem. I'll see if I can get ahold of a Windows machine to poke at
this directly later.
llvm-svn: 315792
This refers to a temporary path that can be shared across all tests,
identified by a particular label. This can be used for things like
caches.
At the moment, the character set for the LABEL is limited to C
identifier characters, plus '-', '+', '=', and '.'. This is the same
set of characters currently allowed in REQUIRES clause identifiers.
llvm-svn: 315697
This paves the way for other projects which might /use/ clang or
lld but not necessarily need to the full set of functionality
available to clang and lld tests to be able to have a basic set
of substitutions that allow a project to run the clang or lld
executables.
llvm-svn: 315627
This addresses two sources of inconsistency in test configuration
files.
1. Substitution boundaries. Previously you would specify a
substitution, such as 'lli', and then additionally a set
of characters that should fail to match before and after
the tool. This was used, for example, so that matches that
are parts of full paths would not be replaced. But not all
tools did this, and those that did would often re-invent
the set of characters themselves, leading to inconsistency.
Now, every tool substitution defaults to using a sane set
of reasonable defaults and you have to explicitly opt out
of it. This actually fixed a few latent bugs that were
never being surfaced, but only on accident.
2. There was no standard way for the system to decide how to
locate a tool. Sometimes you have an explicit path, sometimes
we would search for it and build up a path ourselves, and
sometimes we would build up a full command line. Furthermore,
there was no standardized way to handle missing tools. Do we
warn, fail, ignore, etc? All of this is now encapsulated in
the ToolSubst class. You either specify an exact command to
run, or an instance of FindTool('<tool-name>') and everything
else just works. Furthermore, you can specify an action to
take if the tool cannot be resolved.
Differential Revision: https://reviews.llvm.org/D38565
llvm-svn: 315085
Summary:
normpath() was being called on an empty string and appended to
the environment variable in the case where the environment variable
was unset. This led to ":." being appended to the path, since
normpath() of an empty string is '.', presumably to represent cwd.
Reviewers: zturner, sqlbyme, modocache
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D38542
llvm-svn: 314915
Fix llvm_tools_dir attribute access not to fail when the variable is not
present. This directory is not really necessary to run lit tests,
and the code already accounts for it being None.
The reference was added in r313407, and it breaks the stand-alone lit
package in Gentoo.
Differential Revision: https://reviews.llvm.org/D38442
llvm-svn: 314620
Summary:
Also disables leak checking on lto tests, due to many leaks reported
in the system's ld64.
Reviewers: kcc, pcc, bogner, kubamracek
Subscribers: mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D37781
llvm-svn: 314535
There were two issues, one Python 3 specific related to Unicode,
and another which is that the tool substitution for lld no longer
rejected matches where a / preceded the tool name.
llvm-svn: 313928
debuginfo-tests has need to reuse a lot of common configuration
from clang and lld, and in general it seems like all of the
projects which are tightly coupled (e.g. lld, clang, llvm, lldb,
etc) can benefit from knowing about one other. For example,
lldb needs to know various things about how to run clang in its
test suite. Since there's a lot of common substitutions and
operations that need to be shared among projects, sinking this
up into LLVM makes sense.
In addition, this patch introduces a function add_tool_substitution
which handles all the dirty intricacies of matching tool names
which was previously copied around the various config files. This
is now a simple straightforward interface which is hard to mess
up.
Differential Revision: https://reviews.llvm.org/D37944
llvm-svn: 313919
This has gone back and forth, but it seems this is necessary
after all. realpath is not sufficient because if you have a
file named 'C:\foo.txt', then both realpath('c:\foo.txt') and
realpath(C:\foo.txt') return the string that was passed to them
exactly as is, meaning the case of the drive-letter won't match.
The problem before was not that we were normalizing the case of
items going into the config map, but rather that we were
normalizing the case of something we needed to print. The value
that is used to key on the config map should never be printed.
llvm-svn: 313918
This makes all paths lowercase on Windows, which seemed like a
good idea at the time, but it means that tests can't properly
use FileCheck to match expected path names.
llvm-svn: 313889
Config map is not exposed through the command line, so testing this
is somewhat tricky. But basically we need a test that if a custom
driver builds a config map and passes it to main, it gets respected.
A config map allows config files in the source tree to be mapped
to alternate config files in the build tree. This particular test
works by having two config files in separate directories, and
setting up a config map to have that redirects A/lit.site.cfg
to B/altconfig. Then, we print a message in A/lit.site.cfg
and B/altconfig and check that we do see the output from B
but don't see the output from A. Additionally we test that
the test suite specified by A's config map is properly discovered.
Differential Revision: https://reviews.llvm.org/D38105
llvm-svn: 313887
Summary:
This appears to break some bots, when getToolsPath fails to find some or
all of the tools (for example, an incomplete GnuWin32 installation).
Reviewers: zturner, modocache
Subscribers: sanjoy, llvm-commits
Differential Revision: https://reviews.llvm.org/D38115
llvm-svn: 313854
Many editors and Python-related diagnostics tools such as
debuggers break or fail in mysterious ways when python files
don't end in .py. This is especially true on Windows, but
still exists on other platforms. I don't want to be too heavy
handed in changing everything across the board, but I do want
to at least *allow* lit configs to have .py extensions. This
patch makes the discovery process first look for a config file
with a .py extension, and if one is not found, then looks for
a config file using the old method. So for existing users, there
should be no functional change.
Differential Revision: https://reviews.llvm.org/D37838
llvm-svn: 313849
Bug pointed out by EricWF. This would construct a path where
items would be added in the wrong order, potentially leading
to using the wrong tools for testing.
llvm-svn: 313765
Many svn-based buildbots seem to be getting stuck continually
in tree conflicts due to the output of pyc files. I'm disabling
these as a temporary measure in an attempt to get everything
stable again.
I'll try to remove this code once I understand the problem
better.
llvm-svn: 313698
Since the path a user specifies to the llvm-lit script might be
different than the source tree they built from (since they could
be behind different symlinks), we need to use realpath to make
sure that path comparisons work as expected.
Even better would be to use a custom dictionary comparison with
actual file equivalence comparison semantics, but this is the
least friction to unbreak things for now.
llvm-svn: 313594
It doesn't make sense to me why these bots are failing as the
traceback does not agree with the source code. It's possible
something is stale or there is some other mysterious error,
but in any case hopefully this fixes it.
llvm-svn: 313469
A few tests were manually constructing a LitConfig object, since
I added a new argument to it this was triggering some failures
I didn't detect. `ninja check-lit` passes now.
llvm-svn: 313461
This is helpful for debugging test failures since it removes
the multiprocessing pool from the picture. This will obviously
slow down the test suite by a few orders of magnitude, so it
should only be used for debugging specific failures.
llvm-svn: 313460
It looks like this is going to be non-trivial to get working
in both Py2 and Py3, so for now I'm reverting until I have time
to fully test it under Python 3.
llvm-svn: 313429
This is a resubmission of r313270. It broke standalone builds of
compiler-rt because we were not correctly generating the llvm-lit
script in the standalone build directory.
The fixes incorporated here attempt to find llvm/utils/llvm-lit
from the source tree returned by llvm-config. If present, it
will generate llvm-lit into the output directory. Regardless,
the user can specify -DLLVM_EXTERNAL_LIT to point to a specific
lit.py on their file system. This supports the use case of
someone installing lit via a package manager. If it cannot find
a source tree, and -DLLVM_EXTERNAL_LIT is either unspecified or
invalid, then we print a warning that tests will not be able
to run.
Differential Revision: https://reviews.llvm.org/D37756
llvm-svn: 313407
This patch is still breaking several multi-stage compiler-rt bots.
I already know what the fix is, but I want to get the bots green
for now and then try re-applying in the morning.
llvm-svn: 313335
To further reduce duplicate code, this patch introduces a module
that configs can simply import and get access to a lot of useful
functionality such as setting up paths, adding features that are
useful across all projects, and other utility-type functions.
For now this only updates llvm's suite to use this new library,
but subsequent patches will update other projects.
Differential Revision: https://reviews.llvm.org/D37778
llvm-svn: 313325
This patch simplifies LLVM's lit infrastructure by enforcing an ordering
that a site config is always run before a source-tree config.
A significant amount of the complexity from lit config files arises from
the fact that inside of a source-tree config file, we don't yet know if
the site config has been run. However it is *always* required to run
a site config first, because it passes various variables down through
CMake that the main config depends on. As a result, every config
file has to do a bunch of magic to try to reverse-engineer the location
of the site config file if they detect (heuristically) that the site
config file has not yet been run.
This patch solves the problem by emitting a mapping from source tree
config file to binary tree site config file in llvm-lit.py. Then, during
discovery when we find a config file, we check to see if we have a
target mapping for it, and if so we use that instead.
This mechanism is generic enough that it does not affect external users
of lit. They will just not have a config mapping defined, and everything
will work as normal.
On the other hand, for us it allows us to make many simplifications:
* We are guaranteed that a site config will be executed first
* Inside of a main config, we no longer have to assume that attributes
might not be present and use getattr everywhere.
* We no longer have to pass parameters such as --param llvm_site_config=<path>
on the command line.
* It is future-proof, meaning you don't have to edit llvm-lit.in to add
support for new projects.
* All of the duplicated logic of trying various fallback mechanisms of
finding a site config from the main config are now gone.
One potentially noteworthy thing that was required to implement this
change is that whereas the ninja check targets previously used the first
method to spawn lit, they now use the second. In particular, you can no
longer run lit.py against the source tree while specifying the various
`foo_site_config=<path>` parameters. Instead, you need to run
llvm-lit.py.
Differential Revision: https://reviews.llvm.org/D37756
llvm-svn: 313270
Use os.path.normpath instead of realpath to collapse '..' and '.' path
components. Use realpath when caching search results about a path for
good measure.
I considered rigging up a test involving symlinks for this, but I doubt
I can check a symlink into SVN. The test would have to conditionally
create a symlink at runtime if the host OS supports it. This sounds too
fragile and complicated to me to be worth it.
llvm-svn: 312254
This preserves symlinks in paths, so that someone can symlink more tests
into a larger test suite. For example, debuginfo-tests is currently
designed to be checked out into clang/test. With this change, it can be
symlinked into place instead, which works better with the monorepo.
llvm-svn: 312250
It was marked as unsupported on Windows in r311230 because on some Win10
machines it failed or caused hang. The problem was that on these machines
system bash (C:\Windows\System32\bash.exe) was used which requires paths to be
passed like '/mnt/c/path/to/my/script' instead of 'C:\path\to\my\script'.
TODO: we should make lit detect if system bash is used instead of msys and set
appropriate path format.
llvm-svn: 311558
This is an updated version of https://reviews.llvm.org/D22144 by @jlpeyton.
The patch was accepted but not landed.
This is useful functionality and I would like to use this to enable lit tests for environment variable behaviour.
Differential Revision: https://reviews.llvm.org/D36403
llvm-svn: 311180
Multi-configuration CMake generators such as those for Visual Studio or Xcode do not
specify a build config at configure time, but let the user choose at build
time. In these cases binaries go into build/${Configuration}/bin rather than
build/bin. Prior to this commit, check-lit would fail when using multi-configuration
generators as it did not know how to resolve ${Configuration} in order
to find tools such as FileCheck. This commit teaches it to resolve
llvm_tools_dir within lit using the value specified with --param
build_mode.
Differential Revision: https://reviews.llvm.org/D36263
llvm-svn: 309967
Summary:
This is an alternative solution to running the lit test suite on bots
without polluting the source directory. Each input test suite gets an
auto-generated site config in the build directory that points back to
the test input source directory.
This adds some cmake comlexity, but now we don't need to remove and
re-copy the test input directory before every test.
Reviewers: delcypher, modocache
Subscribers: mgorny, llvm-commits
Differential Revision: https://reviews.llvm.org/D36026
llvm-svn: 309602
Traceback (most recent call last):
File "llvm/utils/lit/tests/Inputs/shtest-format/external_shell/write-bad-encoding.py", line 5, in <module>
sys.stdout.write(b"a line with bad encoding: \xc2.")
sys.stdout.write doesn't accept bytes but sys.stdout.buffer.write accepts.
llvm-svn: 309473
This should fix googletest-format test failures on the clang modules
buildbots, which have a stale copy of the OneTest script in the build
directory.
llvm-svn: 309432
When using win32 cmd.exe, turn off command echoing at the beginning of
the script (@echo off).
Replace a bash shell script with a python script for the
fail_with_bad_encoding test.
llvm-svn: 309399
Summary:
The technique of directly calling subprocess.Popen on a python script
doesn't work on Windows. The executable path of the command must refer
to a valid win32 executable.
Instead, rename all the python scripts masquerading as gtest executables
to have .py extensions, so we can easily detect then and call the python
executable for them. Do this on Linux as well as Windows for
consistency.
The test suite directory names also come out in lower-case on Windows.
We can consider removing that in a later patch. This change just updates
the FileCheck lines to match on Windows.
Fixes PR33933
Reviewers: modocache, mgorny
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D35909
llvm-svn: 309347
Summary:
Normally Python converts all newline characters, Windows or Unix,
to Unix newlines when opening a file. However, lit opens files in
binary mode, which does not perform this conversion. As a result,
trailing Windows newlines are not stripped from test input, which
caused a failure in the TestRunner unit test:
```
FAIL: test_custom (__main__.TestIntegratedTestKeywordParser)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Users\bgesiak\src\llvm\llvm\utils\lit\tests\unit\TestRunner.py", line 109, in test_custom
self.assertItemsEqual(value, ['a', 'b', 'c'])
AssertionError: Element counts were not equal:
First has 1, Second has 0: 'c\r'
First has 0, Second has 1: 'c'
```
Fix the discrepancy in behavior across the two platforms by
manually stripping Windows newlines before yielding each line in
the test file.
Reviewers: echristo, beanz, ddunbar, delcypher, rnk
Reviewed By: rnk
Subscribers: mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D27746
llvm-svn: 309312
Summary:
Depends on https://reviews.llvm.org/D35879.
This reverts rL257268, which in turn was a revert of rL257221.
https://reviews.llvm.org/D35879 marks the tests in the lit test suite
that fail on Windows as XFAIL, which should allow these tests to pass
on Windows-based buildbots.
Reviewers: delcypher, beanz, mgorny, jroelofs, rnk
Reviewed By: mgorny
Subscribers: rnk, ddunbar, george.karpenkov, llvm-commits
Differential Revision: https://reviews.llvm.org/D35880
llvm-svn: 309310
Summary:
An expectation in `utils/lit/tests/Inputs/shtest-shell/redirects.txt`
expects that first a string printed to stdout is seen, and then a
string printed to stderr. Add `flush()` calls to ensure that stdout is
printed before stderr, as expected.
Reviewers: rnk, mgorny, jroelofs
Reviewed By: rnk
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D35947
llvm-svn: 309292
Rewrite the write-to-stderr.sh and write-to-stdout-and-stderr.sh shell
scripts as python scripts and call python on them.
Fixes PR33940
llvm-svn: 309200
This passes locally for me, which fails the overall lit test suite. I
can't debug a passing test, but I will try to help debug the test when
we get some failing logs.
llvm-svn: 309190
Summary:
rL257221 attempted to run lit's own test suite continuously, but that
commit was reverted because lit's test suite does not pass on Windows.
Because lit's tests do not run continuously, they often regress.
In order to un-revert rL257221, mark lit tests that fail as XFAIL for
Windows platforms.
Test Plan:
On a Windows development environment, follow the instructions in
utils/lit/README.txt to run lit's test suite:
```
utils/lit/lit.py \
--path /path/to/your/llvm/build/bin \
utils/lit/tests
```
Verify that the test suite is run and a successful exit code is
returned.
Reviewers: mgorny, rnk, delcypher, beanz
Reviewed By: rnk
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D35879
llvm-svn: 309123
Summary:
Whereas rL299560 and rL309071 call `parallelism_groups.items()`, under the
assumption that `parallelism_groups` is a `dict` type, the default
parameter for that attribute is a `list`. Change the default to a
`dict` for type correctness.
This regression in the unit tests would have been caught if the
unit tests were being run continously. It also would have been caught
if the lit project used a Python type checker such as `mypy`.
Test Plan:
As per the instructions in `utils/lit/README.txt`, run the lit unit
test suite:
```
utils/lit/lit.py \
--path /path/to/your/llvm/build/bin \
utils/lit/tests
```
Verify that the test `lit :: unit/TestRunner.py` fails before applying this
patch, but passes once this patch is applied.
Reviewers: mgorny, rnk, rafael
Reviewed By: mgorny
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D35878
llvm-svn: 309122
Summary:
This reverts rL306623, which removed `FileBasedTest`, an abstract base class,
but did not also remove the usages of that class in the lit unit tests.
The revert fixes four test failures in the lit unit test suite.
Test plan:
As per the instructions in `utils/lit/README.txt`, run the lit unit
test suite:
```
utils/lit/lit.py \
--path /path/to/your/llvm/build/bin \
utils/lit/tests
```
Verify that the following tests fail before applying this patch, and
pass once the patch is applied:
```
lit :: test-data.py
lit :: test-output.py
lit :: xunit-output.py
```
In addition, run `check-llvm` to make sure the existing LLVM test suite
executes normally.
Reviewers: george.karpenkov, mgorny, dlj
Reviewed By: mgorny
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D35877
llvm-svn: 309120
Replace the incorrect variable reference when invalid redirect is used.
This fixes the following issue:
File "/usr/src/llvm/utils/lit/lit/TestRunner.py", line 316, in processRedirects
raise InternalShellError(cmd, "Unsupported redirect: %r" % (r,))
UnboundLocalError: local variable 'r' referenced before assignment
which in turn broke shtest-shell.py and max-failures.py lit tests.
The breakage was introduced during refactoring in rL307310.
Differential Revision: https://reviews.llvm.org/D35857
llvm-svn: 309044
Debugging LIT scripts can be rather painful, as LIT directly does not
specify which line has failed.
Rather, FileCheck is expected to report the failing location, but it can
be often ambiguous if multiple commands are tested against the same
prefix. This change adds a -vv option, which echoes all output.
Then detecting the error becomes straightforward: last printed line is
the failing one.
Of course, it could be desired to try to get failing line number
directly from bash, but it involves excessive hacks on older bash
versions (cf.
https://stackoverflow.com/questions/24398691/how-to-get-the-real-line-number-of-a-failing-bash-command)
Differential Revision: https://reviews.llvm.org/D35330
llvm-svn: 307938
Summary:
This speeds up the LLD test suite on Windows by 3x. Most of the time is
spent on lld/test/ELF/linkerscript/diagnostics.s, which repeatedly
constructs linker scripts with appending echo commands.
Reviewers: dlj, zturner, modocache
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D35093
llvm-svn: 307668
This is especially useful when lit is invoked indirectly by the build
system, and additional arguments can not be easily specified.
Differential Revision: https://reviews.llvm.org/D35091
llvm-svn: 307339
Fix by Andrew Ng!
The Visual Studio build can contain output for multiple configuration types (
e.g. Debug, Release & RelWithDebInfo) within the same build output
directory. Therefore when discovering unit tests, the "build mode" sub directory
containing the appropriate configuration is included in the search. This sub
directory may not always be present, so a test for its existence is required.
Reviewers: zturner, modocache, dlj
Reviewed By: zturner, dlj
Subscribers: grimar, bd1976llvm, gbreynoo, edd, jhenderson, llvm-commits
Differential Revision: https://reviews.llvm.org/D34976
llvm-svn: 307235
Summary:
The lit test formats use largely the same logic for discovering tests. There are
some superficial differences in the logic, which seem reasonable enough to
handle in a single routine.
At a high level, the common goal is "look for files that end with one of these
suffixes, and skip anything starting with a dot." The balance of the logic
specific to ShTest and GoogleTest collapses quite a bit, so that
getTestsInDirectory is only a couple of lines around a call to the new function.
Reviewers: zturner, MatzeB, modocache
Subscribers: sanjoy, llvm-commits
Differential Revision: https://reviews.llvm.org/D34855
llvm-svn: 306895
This reverts commit da6318a92fba793e4f2447ec478b001392d57d43.
This is causing failures on some build bots due to what appears
to be some kind of lit ordering dependency.
llvm-svn: 306833
Presently lit leaks files in the tests' output directories.
Specifically, if a test creates output files, lit makes no
effort to remove them prior to the next test run. This is
problematic because it leads to false positives whenever a
test passes because stale files were present. In general
it is a source of flakiness that should be removed.
This patch addresses this by building the list of all test
directories that are part of the current run set, and then
deleting those directories and recreating them anew. This
gives each test a clean baseline to start from.
Differential Revision: https://reviews.llvm.org/D34732
llvm-svn: 306832
(Take 2: this patch re-applies r306625, which was reverted in r306629. This
patch includes only trivial fixes.)
In Python2 and Python3, the various (non-)?Unicode string types are sort of
spaghetti. Python2 has unicode support tacked on via the 'unicode' type, which
is distinct from 'str' (which are bytes). Python3 takes the "unicode-everywhere"
approach, with 'str' representing a Unicode string.
Both have a 'bytes' type. In Python3, it is the only way to represent raw bytes.
However, in Python2, 'bytes' is an alias for 'str'. This leads to interesting
problems when an interface requires a precise type, but has to run under both
Python2 and Python3.
The previous logic appeared to be correct in all cases, but went through more
layers of indirection than necessary. This change does the necessary conversions
in one shot, with documentation about which paths might be taken in Python2 or
Python3.
Changes from r306625: some tests just print binary outputs, so in those cases,
fall back to str() in Python3. For googletests, add one missing call to
to_string().
(Tested by verifying the visible breakage with Python3. Verified that everything
works in py2 and py3.)
llvm-svn: 306643
Summary:
In Python2 and Python3, the various (non-)?Unicode string types are sort of
spaghetti. Python2 has unicode support tacked on via the 'unicode' type, which
is distinct from 'str' (which are bytes). Python3 takes the "unicode-everywhere"
approach, with 'str' representing a Unicode string.
Both have a 'bytes' type. In Python3, it is the only way to represent raw bytes.
However, in Python2, 'bytes' is an alias for 'str'. This leads to interesting
problems when an interface requires a precise type, but has to run under both
Python2 and Python3.
The previous logic appeared to be correct in all cases, but went through more
layers of indirection than necessary. This change does the necessary conversions
in one shot, with documentation about which paths might be taken in Python2 or
Python3.
Reviewers: zturner, modocache
Subscribers: llvm-commits, sanjoy
Differential Revision: https://reviews.llvm.org/D34793
llvm-svn: 306625
Summary:
This change removes the intermediate 'FileBasedTest' format from lit. This
format is only ever used by the ShTest format, so the logic can be moved into
ShTest directly.
In order to better clarify what the TestFormat subclasses do, I fleshed out the
TestFormat base class with Python's notion of abstract methods, using
@abc.abstractmethod. This gives a convenient way to document the expected
interface, without the risk of instantiating an abstract class (that's what
ABCMeta does -- it raises an exception if you try to instantiate a class which
has abstract methods, but not if you instantiate a subclass that implements
them).
Reviewers: zturner, modocache
Subscribers: sanjoy, llvm-commits
Differential Revision: https://reviews.llvm.org/D34792
llvm-svn: 306623
Summary:
The dead code seems to be unreferenced, according to textual search across the
LLVM SVN repo.
The clarification part of this change alters the name of a module-level function
so that it is different from the name of the class-methods that call it.
Currently, there are no erroneous references, but stylistically (c.f. PEP-8),
internal "helper" functions should generally be named accordingly by prepending
an underscore. (I also chose to add '_impl', which isn't necessary, but helps me
at least to mentally disambiguate the interface and implementation functions.)
Reviewers: zturner, modocache
Subscribers: sanjoy, llvm-commits
Differential Revision: https://reviews.llvm.org/D34775
llvm-svn: 306600
on macOS
This function will be used to tie Clang's Integeration tests to a particular
SDK version. See https://reviews.llvm.org/D32178 for more context.
llvm-svn: 304541
Summary:
For various clang analyzer tests, which were unsupported, I got lit
exceptions, similar to the following:
Exception during script execution:
Traceback (most recent call last):
File "utils/lit/lit/run.py", line 190, in execute_test
result = test.config.test_format.execute(test, lit_config)
File "tools/clang/test/Analysis/analyzer_test.py", line 11, in execute
if result.code == lit.Test.FAIL:
AttributeError: 'tuple' object has no attribute 'code'
This is because executeShTest() in utils/lit/lit/TestRunner.py is
supposed to return a lit.Test.Result object, but in case of unsupported
tests, it returns a plain tuple.
Fix this by returning a properly initialized lit.Test.Result object
instead.
Reviewers: rnk, rafael, modocache
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D33579
llvm-svn: 303943
This seems to have been present since the beginning of time,
which is quite surprising. The symptom was this: Suppose you
have a test with a run line that looks like this:
RUN: foo | FileCheck %s
foo prints some output and then due to a bug in the program it
asserts. On Windows this results in the program returning a
negative exit code. But if enough output had been printed
already by the tool so that the FileCheck match would succeed
then FileCheck would return 0, and because of bad logic in
lit this 0 return value would overwrite the failed return
value from previous items in the pipeline. This only happened
with negative exit codes.
The most sensible behavior is to just take whatever the first
exit code is. There is no logical ordering defined on exit
codes, so comparing with < and > does not make a lot of sense.
Instead, as soon as we find the first non-successful return
value, that should be the result of the entire expression.
This fixes the issue, as now tests which fail on non-Windows
platforms also fail for me on Windows as well.
llvm-svn: 303440
If all jobs complete successfully, use pool.close() instead of
pool.terminate() before waiting for the workers. Zach Turner reported
that he was getting "access denied" exceptions from pool.terminate().
Make the workers abort immediately without printing to stderr when they
are interrupted.
Finally, catch exceptions when attempting to remove our temporary
testing directory. On abnormal exit, there can often be open handles
that haven't been cleaned up yet.
llvm-svn: 301941
Both pickling errors encountered on clang bots and Darwin compiler-rt
should now be fixed.
This has no impact on testing time on Linux, and on Windows goes from
88s to 63s for 'check'. The tests pass on Mac, but I haven't compared
execution time.
llvm-svn: 299775
This is necessary to pass the lit test suite at llvm/utils/lit/tests.
There are some pre-existing failures here, but now switching to pools
doesn't regress any tests.
I had to change test-data/lit.cfg to import DummyConfig from a module to
fix pickling problems, but I think it'll be OK if we require test
formats to be written in real .py modules outside lit.cfg files.
I also discovered that in some circumstances AsyncResult.wait() will not
raise KeyboardInterrupt in a timely manner, but you can pass a non-zero
timeout to work around this. This makes threading.Condition.wait use a
polling loop that runs through the interpreter, so it's capable of
asynchronously raising KeyboardInterrupt.
llvm-svn: 299605
Summary:
This drastically reduces lit test execution startup time on Windows. Our
previous strategy was to manually create one Process per job and manage
the worker pool ourselves. Instead, let's use the worker pool provided
by multiprocessing. multiprocessing.Pool(jobs) returns almost
immediately, and initializes the appropriate number of workers, so they
can all start executing tests immediately. This avoids the ramp-up
period that the old implementation suffers from. This appears to speed
up small test runs.
Here are some timings of the llvm-readobj tests on Windows using the
various execution strategies:
# multiprocessing.Pool:
$ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-process-pool |& grep real: ; done
real: 0m1.156s
real: 0m1.078s
real: 0m1.094s
# multiprocessing.Process:
$ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-processes |& grep real: ; done
real: 0m6.062s
real: 0m5.860s
real: 0m5.984s
# threading.Thread:
$ for i in `seq 1 3`; do tim python ./bin/llvm-lit.py -sv ../llvm/test/tools/llvm-readobj/ --use-threads |& grep real: ; done
real: 0m9.438s
real: 0m10.765s
real: 0m11.079s
I kept the old code to launch processes in case this change doesn't work
on all platforms that LLVM supports, but at some point I would like to
remove both the threading and old multiprocessing execution strategies.
Reviewers: modocache, rafael
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D31677
llvm-svn: 299560
This is needed by TestCases/Posix/coverage-direct.cc
The problem is that the test does:
mkdir <dir>
cd <dir>
cd ..
rm -rf <dir>
<more commands>
the current directory currently looks like "/.../<dir>/../" which
doesn't exist when dir is deleted.
at some point we should probably switch to using the os current
directory (specially if we want to add subshell), but this is a small
incremental improvement.
llvm-svn: 299113
This adds support for commands like
FileCheck < foobar*
which is used by some asan tests because the file they want to read
has a pid in the name.
llvm-svn: 299111
Summary:
`assert.assertItemEqual` went away in Python 3. Seeing how lists
are ordered, comparing a list against each other should work just
as well.
Patch by @jbergstroem (Johan Bergström).
Reviewers: modocache, gparker42
Reviewed By: modocache
Differential Revision: https://reviews.llvm.org/D31229
llvm-svn: 298479
This will enable removing hacks throughout the codebase
in clang and compiler-rt that feed multiple inputs to a
testing utility by globbing, all of which are either disabled
on Windows currently or using xargs / find hacks.
Differential Revision: https://reviews.llvm.org/D30380
llvm-svn: 296904
and UNSUPPORTED"
After r292904 llvm-lit fails to emit the test results in the XML format for
Apple's internal buildbots.
rdar://30164800
llvm-svn: 292942
A `lit` condition line is now a comma-separated list of boolean expressions.
Comma-separated expressions act as if each expression were on its own
condition line:
For REQUIRES, if every expression is true then the test will run.
For UNSUPPORTED, if every expression is false then the test will run.
For XFAIL, if every expression is false then the test is expected to succeed.
As a special case "XFAIL: *" expects the test to fail.
Examples:
# Test is expected fail on 64-bit Apple simulators and pass everywhere else
XFAIL: x86_64 && apple && !macosx
# Test is unsupported on Windows and on non-Ubuntu Linux
# and supported everywhere else
UNSUPPORTED: linux && !ubuntu, system-windows
Syntax:
* '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false.
* Each test feature is a true identifier.
* Substrings of the target triple are true identifiers for UNSUPPORTED
and XFAIL, but not for REQUIRES. (This matches the current behavior.)
* All other identifiers are false.
* Identifiers are [-+=._a-zA-Z0-9]+
Differential Revision: https://reviews.llvm.org/D18185
llvm-svn: 292904
A `lit` condition line is now a comma-separated list of boolean expressions.
Comma-separated expressions act as if each expression were on its own
condition line:
For REQUIRES, if every expression is true then the test will run.
For UNSUPPORTED, if every expression is false then the test will run.
For XFAIL, if every expression is false then the test is expected to succeed.
As a special case "XFAIL: *" expects the test to fail.
Examples:
# Test is expected fail on 64-bit Apple simulators and pass everywhere else
XFAIL: x86_64 && apple && !macosx
# Test is unsupported on Windows and on non-Ubuntu Linux
# and supported everywhere else
UNSUPPORTED: linux && !ubuntu, system-windows
Syntax:
* '&&', '||', '!', '(', ')'. 'true' is true. 'false' is false.
* Each test feature is a true identifier.
* Substrings of the target triple are true identifiers for UNSUPPORTED
and XFAIL, but not for REQUIRES. (This matches the current behavior.)
* All other identifiers are false.
* Identifiers are [-+=._a-zA-Z0-9]+
Differential Revision: https://reviews.llvm.org/D18185
llvm-svn: 292896
Running lit tests and unit tests of ASan and TSan on macOS has very bad performance when running with a high number of threads. This is caused by xnu (the macOS kernel), which currently doesn't handle mapping and unmapping of sanitizer shadow regions (reserved VM which are several terabytes large) very well. The situation is so bad that increasing the number of threads actually makes the total testing time larger. The macOS buildbots are affected by this. Note that we can't easily limit the number of sanitizer testing threads without affecting the rest of the tests.
This patch adds a special "group" into lit, and limits the number of concurrently running tests in this group. This helps solve the contention problem, while still allowing other tests to run in full, that means running lit with -j8 will still with 8 threads, and parallelism is only limited in sanitizer tests.
Differential Revision: https://reviews.llvm.org/D28420
llvm-svn: 292548
Summary:
This change equips lit.py with two new options, --num-shards=M and
--run-shard=N (set by default from env vars LIT_NUM_SHARDS and LIT_RUN_SHARD).
The options must be used together, and N must be in 1..M.
Together these options effect only test selection: they partition the testsuite
into M equal-sized "shards", then select only the Nth shard. They can be used
in a cluster of test machines to achieve a very crude (static) form of
parallelism, with minimal configuration work.
Reviewers: modocache, ddunbar
Reviewed By: ddunbar
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D28789
llvm-svn: 292417
Summary: The parameter `input` to `subprocess.Popen.communicate(...)` must be an object of type `bytes` . This is strictly enforced in python3. This patch (1) allows `to_bytes` to be safely called redundantly. (2) Explicitly convert `input` within `executeCommand`. This allows for usages like `executeCommand(['clang++', '-'], input='int main() {}\n')`.
Reviewers: ddunbar, BinaryKhaos, modocache, dim, EricWF
Reviewed By: EricWF
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D28736
llvm-svn: 292308
Running lit tests and unit tests of ASan and TSan on macOS has very bad performance when running with a high number of threads. This is caused by xnu (the macOS kernel), which currently doesn't handle mapping and unmapping of sanitizer shadow regions (reserved VM which are several terabytes large) very well. The situation is so bad that increasing the number of threads actually makes the total testing time larger. The macOS buildbots are affected by this. Note that we can't easily limit the number of sanitizer testing threads without affecting the rest of the tests.
This patch adds a special "group" into lit, and limits the number of concurrently running tests in this group. This helps solve the contention problem, while still allowing other tests to run in full, that means running lit with -j8 will still with 8 threads, and parallelism is only limited in sanitizer tests.
Differential Revision: https://reviews.llvm.org/D28420
llvm-svn: 292231
This required re-working the streaming support and lit's support for
'--gtest_list_tests' but otherwise seems to be a clean upgrade.
Differential Revision: https://reviews.llvm.org/D28154
llvm-svn: 291029
Summary:
Libc++ frequently has the need to parse more than just the builtin *test keywords* (`RUN`, `REQUIRES`, `XFAIL`, ect). For example libc++ currently needs a new keyword `MODULES-DEFINES: macro list...`. Instead of re-implementing the script parsing in libc++ this patch allows `parseIntegratedTestScript` to take custom parsers.
This patch introduces a new class `IntegratedTestKeywordParser` which implements the logic to parse/process a test keyword. Parsing of various keyword "kinds" are supported out of the box, including 'TAG', 'COMMAND', and 'LIST', which parse keywords such as `END.`, `RUN:` and `XFAIL:` respectively.
As an example after this change libc++ can implement the `MODULES-DEFINES` simply using:
```
mparser = IntegratedTestKeywordParser('MODULES-DEFINES:', ParserKind.LIST)
parseIntegratedTestScript(test, additional_parsers=[mparser])
macro_list = mparser.getValue()
```
Reviewers: ddunbar, modocache, rnk, danalbert, jroelofs
Subscribers: mgrang, llvm-commits, cfe-commits
Differential Revision: https://reviews.llvm.org/D27005
llvm-svn: 288694
This shouls now be safe and not break any more bots. It's strictly better to use '--sdk macosx', otherwise xcrun can return weird things for example when you have Command Line Tools or the SDK installed into '/'.
llvm-svn: 288385
This reverts commit r287403. It breaks an internal asan bot. According
to Kuba, a fix is up for review here: https://reviews.llvm.org/D26929
llvm-svn: 287804
Summary:
This will allow us to revert LLD r284768, which added spaces to get MSys
echo to print what we want.
Reviewers: ruiu, inglorion, rafael
Subscribers: modocache, llvm-commits
Differential Revision: https://reviews.llvm.org/D26009
llvm-svn: 285237
Summary:
r283710 introduced two regressions, one to llvm-lit, and the other to
lit executables that were installed via setuptools. Add instructions on
how to test for these regressions in the future.
Reviewers: ddunbar, delcypher, beanz, chapuni, cmatthews, echristo
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25459
llvm-svn: 284919
Update the CHECK lines in the shtest-timeout.py lit test to account for
the current output. The output has been changed in r271610 without
adjusting the tests.
Differential Revision: https://reviews.llvm.org/D25236
llvm-svn: 284057
Summary:
The Python file `utils/lit/lit/ShUtil.py` contains:
1. Logic used by lit itself
2. A set of unit tests for that logic, which can be run by invoking
`python utils/lit/lit/ShUtil.py`
Move these unit tests to a `tests/unit` subdirectory of lit, and run
the tests as part of lit's test suite. This ensures that, should the
lit test suite be included in LLVM's own regression test suite, these
unit tests will also be run.
(Instructions on how to run lit's test suite can be found in
`utils/lit/README.txt`.)
Reviewers: ddunbar, echristo, delcypher, beanz
Subscribers: mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D25411
llvm-svn: 283968
Summary:
Using Python linter flake8 on the utils/lit reveals several linter
warnings designated "F401: Unused import". Fix or silence these
warnings.
Some of these unused imports are legitimate, while some are part of lit's API.
For example, users of lit expect to be able to access `lit.formats.ShTest` in
their `lit.cfg`, despite the module hierarchy for that symbol actually being
`lit.formats.shtest.ShTest`. To silence linter errors for these lines,
include a "noqa" directive.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: mehdi_amini, llvm-commits
Differential Revision: https://reviews.llvm.org/D25407
llvm-svn: 283710
Summary:
`TestingProgressDisplay` initializes its `current` attribute to `None`, but
never reads or writes the value again. Remove it.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25415
llvm-svn: 283709
Summary:
`ArgumentError` is not defined by the Python standard library.
Executing this line of code would throw a exception, but not the
intended one. It would throw a `NameError` exception, since `ArgumentError`
is undefined.
Use `ValueError` instead, which is defined by the Python standard
library.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25410
llvm-svn: 283708
Summary:
Semicolons aren't necessary as statement terminators in Python, and
each of these uses are superfluous as they appear at the end of a line.
The convention is to not use semicolons where not needed, so remove them.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25409
llvm-svn: 283707
Summary: `prefix` is written to but never read.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25408
llvm-svn: 283706
Summary:
The minimum version of Python required to run LLVM's test suite is 2.7.
Remove a workaround for older Python versions.
Reviewers: echristo, delcypher, beanz, ddunbar
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25400
llvm-svn: 283705
Summary:
The minimum version of Python necessary to run the LLVM test suite is
2.7. Code to work around Python 2.5 and lower isn't necessary.
Reviewers: ddunbar, echristo, delcypher, beanz
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25209
llvm-svn: 283169
Summary:
optparse is deprecated in Python 2.7, which is the minimum version of
Python required to run the LLVM test suite. Replace its usage in lit
with argparse, optparse's 2.7 replacement module.
argparse has several benefits over optparse, but this commit does not
make use of those benefits yet. Instead, it simply uses the new API,
and attempts to keep the number of changes to a minimum.
Confirmed that lit's test suite, as well as LLVM's regression test suite,
still pass with these changes.
Patch By Brian Gesiak!
Reviewers: ddunbar, echristo, beanz, delcypher
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25173
llvm-svn: 283152
Summary:
lit's `OneCommandFileTest` class implements an abstract method that
raises if called. However, it raises by referencing an undefined
symbol. Instead, raise explicitly by throwing a `NotImplementedError`.
This is clearer, and appeases Python linters.
Patch By Brian Gesiak!
Reviewers: ddunbar, echristo, beanz
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25170
llvm-svn: 283090
Summary:
In Python, `None` is a singleton, so checking whether a variable is
`None` may be done with `is` or `is not`. This has a slight advantage
over equiality comparisons `== None` and `!= None`, since `__eq__` may
be overridden in Python to produce sometimes unexpected results.
Using `is None` and `is not None` is also recommended practice in
https://www.python.org/dev/peps/pep-0008:
> Comparisons to singletons like `None` should always be done with `is` or
> `is not`, never the equality operators.
Patch by Brian Gesiak!
Reviewers: ddunbar, echristo, beanz
Subscribers: llvm-commits, mehdi_amini
Differential Revision: https://reviews.llvm.org/D25168
llvm-svn: 283088
- This is primarily useful as a "fail fast" mode for lit, where it will stop
running tests after the first failure.
- Patch by Max Moiseev.
llvm-svn: 282452
Lots of unittests started failing under asan after r280455. It seems
they've been failing for a long time, but lit silently ignored them.
Downgrade the error so we can figure out what is going on.
Filed http://llvm.org/PR30285.
llvm-svn: 280674
Summary:
This is a follow up to r280455, where a check for the process exit code
was introduced. Some ASAN bots throw this error now, but it's impossible
to understand what's wrong with them, and the issue is not reproducible.
Reviewers: vitalybuka
Differential Revision: https://reviews.llvm.org/D24210
llvm-svn: 280550
Do this by creating a temp directory in the normal system temp
directory, and cleaning it up on exit.
It is still possible for this temp directory to leak if Python exits
abnormally, but this is probably good enough for now.
Fixes PR18335
llvm-svn: 280501
googletest formatted tests are discovered by running the test executable.
Previously testing would silently succeed if the test executable crashed
during the discovery process. Now testing fails with "error: unable to
discover google-tests ..." if the test executable exits with a non-zero status.
llvm-svn: 280455
Apparently nobody evaluated multiprocessing on Windows since Daniel
enabled multiprocessing on Unix in r193279. It works so far as I can
tell.
Today this is worth about an 8x speedup (631.29s to 73.25s) on my 24
core Windows machine. Hopefully this will improve Windows buildbot cycle
time, where currently it takes more time to run check-all than it does
to self-host with assertions enabled:
http://lab.llvm.org:8011/builders/clang-x86-windows-msvc2015/builds/20
build stage 2 ninja all ( 28 mins, 22 secs )
ninja check 2 stage 2 ( 37 mins, 38 secs )
llvm-svn: 280382
This variable is used by ASan (and other sanitizers in the future)
on s390x-linux to override a check for CVE-2016-2143 in the running
kernel (see revision 267747 on compiler-rt). Since the check simply
checks if the kernel version is in a whitelist of known-good versions,
it may miss distribution kernels, or manually-patched kernels - hence
the need for this variable. To enable running the ASan testsuite on
such kernels, this variable should be passed from the environment
down to the testcases.
Differential Revision: http://reviews.llvm.org/D19888
llvm-svn: 273825
- The intended use of this was just in diagnostics, so we shouldn't pay the
cost of reading these all the time.
- This will avoid including the full output of each command in tests which
fail, but the most important use case for this was to gather the output of
the specific command which failed.
llvm-svn: 272365
Use os.devnull instead of tempfiles when substituting '/dev/null' on
Windows machines. This should make the bots just a bit speedier.
Thanks to Yunzhong Gao for testing this patch on Windows!
Differential Revision: http://reviews.llvm.org/D20549
llvm-svn: 272290
- This will cause lit to automatically include the first 1K of data in
redirected output files when a command fails (previously if the command
failed, but the main point of the test was, say, a `FileCheck` later on, then
the log wasn't helpful in showing why the command failed).
llvm-svn: 272021
- This only applies to scripts executed by the _internal_ shell script
interpreter.
- This patch reworks the log to look more like a shell transcript, and be less
verbose (but in the interest of calling attention to the important parts).
Here is an example of the new format, for commands with/without failures and
with/without output:
```
$ true
$ echo hi
hi
$ false
note: command had no output on stdout or stderr
error: command failed with exit status 1
```
llvm-svn: 271610
Summary:
This patch adds a "REQUIRES-ANY" feature test that is disjunctive. This marks a test as `UNSUPPORTED` if none of the specified features are available.
Libc++ has the need to write feature test such as `// REQUIRES-ANY: c++98, c++03` when testing of behavior that is specific to older dialects but has since changed.
Reviewers: rnk, ddunbar
Subscribers: ddunbar, probinson, llvm-commits, cfe-commits
Differential Revision: http://reviews.llvm.org/D20757
llvm-svn: 271468
Summary:
We need these variables to concatenate two absolute paths to construct
a valid path. Currently, %t\%t is, for example, expanded to C:\foo\C:\foo,
which is not a valid path because ":" is not a valid path character
on Windows. With this patch, %t will be expanded to C\foo.
Differential Revision: http://reviews.llvm.org/D19757
llvm-svn: 268168
Summary:
Upstream googletest prints "Running main() from gtest_main.cc" to stdout prior
to running tests. LLVM removed that print statement in r61540. If a user were
to use lit to run tests that use upstream googletest, however, lit
reports "Running main()" as an invalid test name.
To avoid such a failure, add an extra conditional to `formats/googletest.py`.
Also add tests to demonstrate the modified behavior.
Reviewers: abdulras, ddunbar
Subscribers: ddunbar, llvm-commits, kastiglione
Differential Revision: http://reviews.llvm.org/D18606
llvm-svn: 265034
Summary:
The multiprocessing.Queue.put() call can hang if we try queueing all the
tests before starting to take them out of the queue.
The current implementation hangs if tests exceed 2^^15, on Mac OS X.
This might happen with a ninja check-all if one has a bunch of llvm
projects.
Reviewers: delcypher, bkramer
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D17609
llvm-svn: 263731
This lets us for example start running the unit test suite early. For
'check-llvm' on my machine, this drops the tim e from 44s to 32s!!!!!
It's pretty ugly. I barely know how to write Python, so feel free to
just tell me how I should write it instead. =D Thanks to Filipe and
others for help.
Differential Revision: http://reviews.llvm.org/D18089
llvm-svn: 263329
Fix the lit bug that enabled this "feature" (empty triple is substring
of all possible target triples) and change the two outliers to use the
documented * syntax.
llvm-svn: 259799
If a lit test has a RUN line that includes a redirection to "/dev/tty", the
redirection goes to the special device file corresponding to the console. It
is /dev/tty on UNIX-like systems and "CON" on Windows.
This patch is needed to implement a test like PR25717 (caused by the size limit
of the Windows system call WriteConsole() prior to Windows 8) where the test
only breaks when outputing to the console and won't fail if using a pipe.
llvm-svn: 258898
Chris Matthews