It is common to have a default soft process limit, at least on some families of
Linux distributions, of 1024. This is normally more than enough, but if you
have many cores, and you're running tests that create many threads, this can
become a problem. My POWER7 development machine has 48 cores, and when running
the lld regression tests, which often want to create up to 48 threads, I run
into problems. lit, by default, will want to run 48 tests in parallel, and
48*48 < 1024, and so many tests fail like this:
terminate called after throwing an instance of 'std::system_error'
what(): Resource temporarily unavailable
or lit fails like this when launching a test:
OSError: [Errno 11] Resource temporarily unavailable
lit can easily detect this situation and attempt to repair it before launching
tests (by raising the soft process limit to something that will allow ncpus^2
threads to be created), and should do so to prevent spurious test failures.
This is the follow-up to this thread:
http://lists.llvm.org/pipermail/llvm-dev/2015-October/090942.html
llvm-svn: 249161
The variable is actually called ANDROID_SERIAL.
This was not exercised on the bots until today.
Should fix the sanitizer-x86_64-linux failures.
llvm-svn: 246898
The plan is to use this for the sanitizer test suite on Windows. See
PR24554 for more details on why we need this.
Tested manually by injecting rand() into a sanitizer test and watching
what it does.
llvm-svn: 246704
My theory is that somehow Python's refcounting and GC strategy isn't
closing the subprocess handle in a timely fashion. This accesses the
private '_handle' field of the Popen object, but I see no other way to
do this. If this doesn't address the problem on the sanitizer-windows
buildbot, we can revert this change. If it does, then let's keep the
hack.
llvm-svn: 245946
This was affecting test/asan/TestCases/Windows/coverage-basic.cc in
compiler-rt. It does something like:
cd %T/mydir
%clang %s -o t.exe
./t.exe
Previously, we'd end up looking for t.exe relative to the cwd of the lit
process, not the cwd of the test.
llvm-svn: 242941
The MSys 2 version of 'env' cannot be used to set 'TZ' in the
environment due to some portability hacks in the process spawning
compatibility layer[1]. This affects test/Object/archive-toc.test, which
tries to set TZ in the environment.
Other than that, this saves a subprocess invocation of a small unix
utility, which is makes the tests faster.
The internal shell does not support shell variable expansion, so this
idiom in the ASan tests isn't supported yet:
RUN: env ASAN_OPTIONS=$ASAN_OPTIONS:opt=1 ...
[1] https://github.com/Alexpux/MSYS2-packages/issues/294
Differential Revision: http://reviews.llvm.org/D11350
llvm-svn: 242696
Summary: This patch allows executeCommand to pass a string to the processes stdin.
Reviewers: ddunbar, jroelofs
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D11332
llvm-svn: 242631
Pass ADB and ADB_SERIAL environment variables to lit tests.
This would allow running Android tests in compiler-rt when
there is more than one device attached to the host.
llvm-svn: 240459
Summary:
I spend some time trying to get the LIT test suite passing. Here are the changes that I needed to make on my machine.
I made the following changes for the following reasons.
1. google-test.py: The Google test format now checks for "[ PASSED ] 1 test." to check if a test passes.
2. discovery.py: The output appears in a different order on my machine than it did in the test.
3. unittest-adaptor.py: The output appears in a different order on my machine than it did in the test.
4. The classname is now formed differently in `getJUnitXML(...)`.
I'm not sure what is causing the output order to differ in discovery.py and unittest-adaptor.py. Does anybody have any thoughts?
Reviewers: ddunbar, danalbert, jroelofs
Reviewed By: jroelofs
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D9864
llvm-svn: 239663
By setting limit_to_features to a non empty list of features a configuration can
restrict the set of tests to run to only include tests that require a feature in
this list.
rdar://21082253
llvm-svn: 238766
These changes allow usages where you want to pass an additional
commandline option to all invocations of a specific llvm tool. Example:
> llvm-lit -Dllc=llc -enable-misched -verify-machineinstrs
Differential Revision: http://reviews.llvm.org/D9487
llvm-svn: 236461
The sanitizer test suite uses this idiom to disable a test. Now that we
actually check if a test ran after invoking it, we see that zero tests
ran, and complain.
Instead, ignore tests starting with DISABLED_ completely. Fixes the
sanitizer test suite failures on Windows.
llvm-svn: 234247
The '/' character in the test name of a type-parameterized test is not a
path separator, and should not be '\' on Windows. We were passing a test
name to --gtest_filter which found no tests, so the exit code was zero,
indicating a passed test.
This bug has been here since r84387 in 2009, when Jeff Yasskin added the
original lit support for type-paratermized tests. Somewhere along the
line some of the ValueMapTests started failing, but we can fix those
separately.
llvm-svn: 234242
The internal shell was already threading around a 'cwd' parameter. We
just have to make it mutable so that we can update it as the test script
executes.
If the shell ever grows support for environment variable substitution,
we could also implement support for export.
llvm-svn: 231017
Summary: I think this is probably a bug, but I'm putting this up for review just to be sure. I think that `lit.util.capture` should decode the resulting string in the same way `lit.util.executeCommand` does.
Reviewers: ddunbar, EricWF
Reviewed By: EricWF
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D6769
llvm-svn: 225681
Summary:
The following types can be encoded and decoded by the json library:
`dict`, `list`, `tuple`, `str`, `unicode`, `int`, `long`, `float`, `bool`, `NoneType`.
`JSONMetricValue` can be constructed with any of these types, and used as part of Test.Result.
This patch also adds a toMetricValue function that converts a value into a MetricValue.
Reviewers: ddunbar, EricWF
Reviewed By: EricWF
Subscribers: cfe-commits, llvm-commits
Differential Revision: http://reviews.llvm.org/D6576
llvm-svn: 224628
We were already requiring 2.5, which meant that people on old linux distros
had to upgrade anyway.
Requiring python 2.6 will make supporting 3.X easier as we can use the 3.X
exception syntax.
According to the discussion on llvmdev, there is not much value is requiring
just 2.6, we may as well just require 2.7.
llvm-svn: 224129
Summary:
This patch gives me just enough to leverage the existing functionality in `TestRunner` for use in `libc++` and `libc++abi` .
It does the following:
* Adds the `UNSUPPORTED` tag to `TestRunner.parseIntegratedTestScript`.
* Allows `parseIntegratedTestScript` to return an empty script if a script is not required by the caller.
Reviewers: ddunbar, EricWF
Reviewed By: EricWF
Subscribers: cfe-commits, llvm-commits
Differential Revision: http://reviews.llvm.org/D6589
llvm-svn: 223915
--xunit-xml-output saves test results to disk in JUnit's xml format. This will allow Jenkins to report the details of a lit run.
Based on a patch by David Chisnall.
llvm-svn: 223163
execution of a shell command. This can happen for example if the
``RUN:`` line calls a python script which can work correctly under
Linux/OSX but will not work under Windows. A more useful error message
is now shown rather than an unhelpful backtrace.
llvm-svn: 220227
This code is based on the existing LLVM Go bindings project hosted at:
https://github.com/go-llvm/llvm
Note that all contributors to the gollvm project have agreed to relicense
their changes under the LLVM license and submit them to the LLVM project.
Differential Revision: http://reviews.llvm.org/D5684
llvm-svn: 219976
As far as I can tell UTF-8 has been supported since the beginning of Python's
codec support, and it's the de facto standard for text these days, at least
for primarily-English text. This allows us to put Unicode into lit RUN lines.
rdar://problem/18311663
llvm-svn: 217688
Summary:
This patch changes the way xfail and unsupported tests are displayed.
This output is only displayed when the --show-unsupported/--show-xfail flags are passed to lit.
Currently xfail/unsupported tests are printed during the run of the test-suite. I think its better to display this information during the summary instead.
This patch removes the printing of these tests from when they are run to the summary.
Reviewers: ddunbar, EricWF
Reviewed By: EricWF
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D4842
llvm-svn: 215809
Summary:
This patch add a --show-xfail flag. If this flag is specified then each xfail test will be printed to output.
When it is not given xfail tests are ignored. Ignoring xfail tests is the current behavior.
This flag is meant to mirror the --show-unsupported flag that was recently added.
Reviewers: ddunbar, EricWF
Reviewed By: EricWF
Subscribers: llvm-commits
Differential Revision: http://reviews.llvm.org/D4750
llvm-svn: 214609
It was pointed out that this breaks the "virtual test discovery"
mechanism, which allows for narming tests in the test exec root.
Reverting until I can figure out how to fix this.
llvm-svn: 211048
It would previously say things like
warning: input 'test/Frontend/foo.c' contained no tests
and have the user pull their hair trying to figure out what's wrong with that
file. This patch changes the message to the much clearer:
warning: no such file or directory: 'test/Frontend/foo.c'
Differential Revision: http://reviews.llvm.org/D4097
llvm-svn: 210597
Clang's lit cfg already detects the currently selected SDK via
"xcrun --show-sdk-path". The same thing should be done for compiler-rt tests,
to make them work on recent OS X versions. Instead of duplicating the detection
code, this patch extracts the detection function into a lit.util method.
Patch by Kuba Brecka (kuba.brecka@gmail.com),
reviewed at http://reviews.llvm.org/D4072
llvm-svn: 210534
This commit prospectively brings the benefits of r198766 to older supported
Python versions (2.5+).
Tested with Python 2.6, 2.7, 3.1 and 3.3 (!)
llvm-svn: 199009
On the other hand, exec(compile()) doesn't work in older Python versions in the
2.x series.
This commit introduces exec(compile()) with a fallback to plain exec(). That'll
hopefully hit the sweet spot in terms of version support.
Followup to r198766 which added enhanced source locations for lit cfg parsing.
llvm-svn: 199006
Python doesn't do a good job at diagnosing string exec() so use execfile()
where available.
This should be a timesaver when trying to get to the bottom of build bot
failures.
Before:
File "llvm/utils/lit/lit/TestingConfig.py", line 93, in load_from_path
exec("exec data in cfg_globals")
File "<string>", line 1, in <module>
File "<string>", line 194, in <module>
NameError: name 'typo' is not defined
After:
File "llvm/utils/lit/lit/TestingConfig.py", line 95, in load_from_path
execfile(path, cfg_globals)
File "clang/test/lit.cfg", line 194, in <module>
typo
^~~~
NameError: name 'typo' is not defined
llvm-svn: 198766
Add option -i to prioritize test runs by source file modification time and
previous failure state.
This optimal scheduling reduces typical test-and-fix iteration times to a
matter of seconds by rapidly answering the questions:
1) Did my recent change fix tests that were previously failing?
2) Do the tests I just wrote / modified still work?
The current implementation requires write permissions to the source tree
because it uses mtimes to track failures.
llvm-svn: 198150
The error raised by Python varies by platform(!), so let's just catch any
exception and fall back.
Thanks to Sylvestre Ledru for noticing this on a Debian / Python 2.7 system
running code coverage.
llvm-svn: 193516
so try PATH next. Assume it is sane enough to cover the usual system
bash locations too, but the old list is not good enough for NetBSD.
llvm-svn: 193471
If multiprocessing was requested, detected as available and subsequently failed
to initialize it's worth letting the user know about it before falling back to
threads.
This condition can arise in certain OpenBSD / FreeBSD Python versions.
llvm-svn: 193465
This should be a better fix for lit multiprocessing failures, replacing the
OpenBSD and FreeBSD workarounds in r193413 and r193457.
Reference: http://bugs.python.org/issue3770
llvm-svn: 193463
Speculative quick fix based on clang-X86_64-freebsd output:
File "/usr/local/lib/python2.6/multiprocessing/synchronize.py", line 33, in <module>
" function, see issue 3770.")
ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
llvm-svn: 193457
- This is a work-in-progress and all details are subject to change, but I am
trying to build up support for allowing lit to be used as a driver for
performance tests (or other tests which might want to record information
beyond simple PASS/FAIL).
llvm-svn: 190535
- This aligns with how existing test suites end up wanting to use the local
config files, conceptually it makes sense to consider them to be inherited.
llvm-svn: 189885
- At least on OS X, it is important for correct behavior of /bin/[ that argv[0]
is passed as written, and not as the full executable path.
llvm-svn: 189559
- For whatever reason, we have a lot of test files with bogus unicode
characters. This patch allows those scripts to still be parsed on Python3 by
changing the parsing logic to work on binary files, and only require the
actual script commands to be convertible to ascii.
- This patch has been tweaked to now ensure that the command strings are not of
unicode type on Python 2.6-7.
llvm-svn: 188398
- For whatever reason, we have a lot of test files with bogus unicode
characters. This patch allows those scripts to still be parsed on Python3 by
changing the parsing logic to work on binary files, and only require the
actual script commands to be convertible to ascii.
llvm-svn: 188376
- Injecting it as 'lit' is gross, since that name should be used to refer to
the actual package. For now both are available so it is possibly to cleanup
test config files incrementally.
llvm-svn: 188039
Matthias Braun