Tests that require support for Clang-verify are already marked as such
explicitly by their extension, which is .verify.cpp. Requiring the use
of an explicit Lit feature is, after thought, not really helpful.
This is a change in design: we have been bitten in the past by tests not
being enabled when we thought they were. However, the issue was mostly
with file extensions being ignored. The fix for that is not to blindly
require explicit features all the time, but instead to report all files
that are in the suite but that don't match any known test format. This
can be implemented in a follow-up patch.
This reverts commit 51a60ed14c, since the test still doesn't pass on
Windows. Marking the test as UNSUPORTED on Windows again until I've
figured out the problem.
Since 88af3ddb1e, libc++ will prefer Python 3 when available. It is
available on Apple platforms, so subprocess.check_output will return
bytes instead of str. This lead to comparisons against str to be false,
and the MacOS platform not being detected properly.
This allows defining Lit features that can be enabled or disabled based
on compiler support, and parameters that are passed on the command line.
The main benefits are:
- Feature detection is entirely based on the substitutions provided in
the TestingConfig object, which is simpler and decouples it from the
complicated compiler emulation infrastructure.
- The syntax is declarative, which makes it easy to see what features
and parameters are accepted by the test suite. This is significantly
less entangled than the current config.py logic.
- Since feature detection is based on substitutions, it works really
well on top of the new format, and custom Lit configurations can be
created easily without being based on `config.py`.
This commit is a reapplication of 6d58030c8c, which was reverted in
8f24c4b72f because it broke Python 3 support. This re-application
supports Python 3.
Differential Revision: https://reviews.llvm.org/D78381
Instead of using the libc++ headers provided alongside the toolchain,
use those in the sibling libcxx directory that we know is checked out.
Before the days of the monorepo, we couldn't assume that the libc++
repository was present when building libcxxabi. Since we can now make
that assumption, it's always better to use the version of libc++ that
is in lockstep with libc++abi, to avoid subtle bugs.
This allows defining Lit features that can be enabled or disabled based
on compiler support, and parameters that are passed on the command line.
The main benefits are:
- Feature detection is entirely based on the substitutions provided in
the TestingConfig object, which is simpler and decouples it from the
complicated compiler emulation infrastructure.
- The syntax is declarative, which makes it easy to see what features
and parameters are accepted by the test suite. This is significantly
less entangled than the current config.py logic.
- Since feature detection is based on substitutions, it works really
well on top of the new format, and custom Lit configurations can be
created easily without being based on `config.py`.
Differential Revision: https://reviews.llvm.org/D78381
Instead of having different names for the same Lit feature accross code
bases, use the same name everywhere. This NFC commit is in preparation
for a refactor where all three projects will be using the same Lit
feature detection logic, and hence it won't be convenient to use
different names for the feature.
Differential Revision: https://reviews.llvm.org/D78370
When the new libc++ test format was enabled, warnings were accidentally
dropped cause they were not part of the %{compile_flags} substitution.
This commit adds them back, however `-Werror` is only used for non-verify
tests (cause it doesn't make sense for verify tests).
This commit is a re-application of 20fd624380, which was reverted in
5ec6fdb058 because it broke the C++03 bot. This failure should have
been fixed in b4fb705e77.
When the new libc++ test format was enabled, warnings were accidentally
dropped cause they were not part of the %{compile_flags} substitution.
This commit adds them back, however `-Werror` is only used for non-verify
tests (cause it doesn't make sense for verify tests).
Summary:
This patch add the dataflow option to LLVM_USE_SANITIZER and documents
it.
Tested via check-cxx (wip to fix the errors).
Reviewers: morehouse, #libc!
Subscribers: mgorny, cfe-commits, libcxx-commits
Tags: #clang, #libc
Differential Revision: https://reviews.llvm.org/D78390
This was originally committed as f8452ddfcc and reverted in 7cb1aa9d93.
The issue was that shell builtins were being escaped too, and apparently
Bash won't execute a builtin when it is quoted e.g. '!'. Instead, it
thinks it's a command and it can't find it.
Re-committing the change with that issue fixed.
Cherrypick the upstream fix commit a77d5f7 onto llvm/utils/benchmark
and libcxx/utils/google-benchmark.
This fixes LLVM's 32-bit RISC-V compilation, and the issues
mentioned in https://github.com/google/benchmark/pull/955
An additional cherrypick of ecc1685 fixes some minor formatting
issues introduced by the preceding commit.
Differential Revision: https://reviews.llvm.org/D78084
Instead of creating Lit features for all __config_site macros automatically,
only do so for macros that generate features actually used in the test
suite. This makes it easier to know which ones are supported by the test
suite at a glance.
Note that the `libcpp-abi-version-vN` is dropped altogether, but it
wasn't used anywhere.
These substitutions are strongly tied to the operation of the test
format, so it makes sense to have them defined by the test format
instead of the Lit configuration. They should be defined regardless
of which configuration is in use.
The new format requires using an external shell, and as we transition
and we can simplify config.py as we transition to the new format. Also,
frankly, I'd be quite surprised if that setting was still working anyway
because we have several .sh.cpp tests that likely don't work in Lit's
internal shell.
By renaming .fail.cpp tests that don't need clang-verify to .compile.fail.cpp,
the new test format will not try to compile these tests with clang-verify,
and the old test format will work just the same. However, this allows
removing a workaround that requires parsing each test looking for
clang-verify markup.
After this change, a .fail.cpp test should always have clang-verify markup.
When clang-verify is not supported by the compiler, we will just check that
these tests fail to compile. When clang-verify is supported, these tests
will be compiled with clang-verify whether they have markup or not (so
they should have markup, or they will fail).
This simplifies the test suite and also ensures that all of our .fail.cpp
tests provide clang-verify markup. If it's impossible for a test to have
clang-verify markup, it can be moved to a .compile.fail.cpp test, which
are unconditionally just checked for compilation failure.
With this patch, .verify.cpp tests explicitly require clang-verify, but
no other test types require clang-verify out of the box. This will allow
making several .fail.cpp tests that don't have any clang-verify markup
to be just .compile.fail.cpp tests, which in turn should allow removing
a long standing workaround that requires parsing tests to detect whether
they have any clang-verify markup in them.
There are no such tests in the libc++ test suite, and I want to move
away from `.fail.cpp` tests (in favour of something else) too, which
require a workaround.
Instead of spamming a bunch of available features that are not actually
used anywhere, only set those that are actually used in the test suite.
In the future, this should probably be based on the target triple only,
with the ability to have wildcards in the triple.
Remove mentions of the ValgrindExecutor, which doesn't exist. That
executor is literally nowhere in the code base, so this is dead code
as far as we're concerned.
Also, inline a one-liner function that was called exactly once.
The libc++ test suite currently defines several features that are not
used anywhere in the tests, or that are redundant with other features.
For the purpose of simplifying config.py and to ease the bring up of a
new configuration, this commit removes some of these features:
- rename dylib-has-no-filesystem to c++filesystem-disabled, which exists
- rename apple-darwin to just darwin, which is already set
- remove useless setting of libstdc++, which is already set correctly
- remove libcpp-abi-unstable, which is not used anywhere
- remove the glibc-XXX features, which are not used anywhere
It has never been used, and it actually doesn't really work because it
assumes that the target supports Python. Instead, it's better to just
use `!` since we're running ShTests in system shells anyway.
Instead of creating a temporary directory inside /tmp and running the
tests there, use a directory name based on LIT's %t substitution. This
has the benefit of not hitting /tmp so much (which is slow on some
filesystems). It also has the benefit that `ninja -C build clean` will
automatically remove the artifacts even if a test somehow failed to
remove its temporary directory (I've seen this happen when CTRL-C is
received).
This allows both the old and the new testing formats to handle these
tests with modules enabled.
We also include the modules flags in the %{flags} substitution, which
means that .sh.cpp tests in the old format and all tests in the new
format will use modules flags when enabled.
The LitConfig is shared across the whole test suite. However, since
enabling recursive expansion can be a breaking change for some test
suites, it's important to confine the setting to test suites that
enable it explicitly.
Note that other issues were raised with the way recursiveExpansionLimit
operates. However, this commit simply moves the setting to the right
place -- the mechanism by which it works can be improved independently.
Differential Revision: https://reviews.llvm.org/D77415
We had a workaround because GCC 5 does not evaluate static assertions
that are dependent on template parameters. This commit removes the
workaround and marks the corresponding tests as unsupported with GCC 5.
This has the benefit of bringing the new and the old test formats closer
without having to carry a workaround for an old compiler in the new
test format.
Otherwise, we're missing some flags like the flags that are used by
sanitizer builds and the 32-bit builds. In the long term, I think it
would be better to have only %{compile_flags} and %{link_flags}, but
for the benefit of adopting the new format by default, I think it's OK
to add %{flags} to it.
On Windows, we must make sure to close the temporary tar file before we
try to scp it.
This is an alternative approach to https://reviews.llvm.org/D77500.
This new test format is simpler and more flexible. It creates Lit ShTests
on the fly that reuse existing substitutions (like %{cxx}) instead of
having complex logic in Python to run the tests. This has the benefit
that virtually no coding is required to customize how the test suite is
run -- one can achieve pretty much anything by defining the appropriate
substitutions in a simple lit.cfg file.
For example, in order to run the tests on an embedded device after
building with a specific SDK, one can set the %{cxx} and %{compile_flags}
substitutions to use that SDK, and the %{exec} substitution to the ssh.py
script currently used for .sh.cpp tests with a remote executor. Dealing with
the SSHExecutor becomes unnecessary, since all tests are treated like ShTests.
As a side effect of this design, configuration files for the test
suite can be as simple as:
config.substitutions.append(('%{cxx}', '<path-to-compiler>'))
config.substitutions.append(('%{compile_flags}', '<flags>'))
config.substitutions.append(('%{link_flags}', '<flags>'))
config.substitutions.append(('%{exec}', '<script-to-execute>'))
This should allow storing lit.cfg files for various configurations
directly in the repository instead of relying on complicated logic
in config.py to set up the right flags. I've found numerous problems
in that logic in the past years, and it seems like having simple and
explicit configuration files for the configurations we support is
going to solve most of these problems. Specifically, I am hoping to
store configuration files for testing other Standard Libraries in
the repository.
Improving the interaction with the test suite configuration is still a
work in progress, so for now this test format reuses the substitutions and
available features that are set up by the current config.py.
This new test format should support pretty much everything that the current
test format supports, however it will not be enabled by default at first to
make sure we're satisfied with it. For a short period of time, the new format
will require `--param=use_new_format=True` to be enabled, however it is a very
short term goal to replace the current testing format entirely and to simplify
the configuration accordingly.
Differential Revision: https://reviews.llvm.org/D77338
Instead of executing tests from within the libc++ test suite, we execute
them from the Lit execution directory. However, since some tests have
file dependencies, we must copy those dependencies to the execution
directory where they are executed.
This has the major benefit that if a test modifies a file (whether it
is wanted or not), other tests will not see those modifications. This
is good because current tests assume that input data is never modified,
however this could be an incorrect assumption if some test does not
behave properly.
The benefit of doing this is that we can now handle directories that
contain symlinks and other arbitrary things, such as the static_test_env
required by filesystem tests.
As a fly-by fix, we also accumulate several commands to perform over SSH
and execute them at once instead of SSHing several times. This should be
faster on average.
If a ShTest has for example another command in front of the test
executable it wants to execute, ssh.py needs to properly translate
the path of that test executable to the executable on the remote host.
For example, running '%{exec} ! %t.exe', we can't assume that the
test-executable is the first argument after '%{exec}'.
This makes it closer to how one would run the tests by hand, and it is
also closer to how the SSHExecutor runs the tests remotely. It also
allows using shell builtins in .sh.cpp tests when using %{exec}.
That way, local lit configuration files don't have to worry about
deep-copying the compiler instance of the test format, which is
arguably an implementation detail.
We pass the config to this method even though it is not used by the
current test format because this allows replacing the current test
format by other test formats that would require the config to add
new compile flags.
This reduces the complexity of our already complex global lit configuration,
and also avoids cluttering the compilation commands for all tests with
things that are only relevant to the filesystem tests.
Differential Revision: https://reviews.llvm.org/D76785
Previously, filesystem tests would require LIBCXX_FILESYSTEM_DYNAMIC_TEST_ROOT
to be present in the environment and to match the value provided when
compiling, as a macro. This has the problem that it only allows for the
filesystem tests to be run on the same machine they are created.
Instead, we create a temporary directory for each test. Technically,
this is tricky to do because we're relying on some of the code that
we're testing to do this. However, there's no other portable way of
creating temporary direcories in C++, so this is difficult to avoid.
Differential Revision: https://reviews.llvm.org/D76731
This re-commits cd7f9751c3, which was reverted in 12f6b024f9 because
it broke the LLVM `check-all` target. This commit addresses the underlying
issue by not setting the lit_config.recursiveExpansionLimit parameter of
the libc++ test suite, which is otherwise picked up by other test suites
in LLVM.
Once we've settled on a fix for the underlying issue with
lit_config.recursiveExpansionLimit, we can start using it
again in libc++, but for now we can just work around it.
This allows adding compilation flags for a single test, which can help
eliminate some .sh.cpp tests and some custom handling in the libc++
test format.
It also works around the issue that .sh.cpp substitutions are _not_
equivalent to the actual compiler command lines used to compile tests,
since the compiler flags can be modified in local lit configurations,
and substitutions are frozen at that point. For example using %{compile}
in a .sh.cpp test in the coroutines subdirectory will not include the
-fcoroutines-ts flag, which is added in the local lit config, because
the %{compile} substitution is created long before we add -fcoroutines-ts
to the compiler flags (in the lit.local.cfg for coroutines).
This reverts commit cd7f9751c3 which has
unintended breakage to non-libcxx projects when using the documented way
of building LLVM. (See the Getting Started guide. I.e. one big CMake setup.)
Since lit supports expanding substitutions recursively, we can define
substitutions in terms of other substitutions. This allows us to simplify
how libc++ substitutions are defined.
This doesn't change the substitutions at all, it only makes them simpler
to define.
lit is not very clever when it performs substitution on RUN lines. It
simply looks for a match anywhere in the line (without tokenization)
and replaces it by the expansion. This means that a RUN line containing
e.g. `-verify-ignore-unexpected=note` wouod be expanded to
`-verify-ignore-unexpected=<substitution for not>e`, which is
surprising and nonsensical.
It also means that something like `%compile_module` could be expanded
to `<substitution-for-%compile>_module` or to the correct substitution,
depending on the order in which substitutions are evaluated by lit.
To avoid such problems, it is a good habit to delimit custom substitutions
with some token. This commit does that for all substitutions used in the
libc++ and libc++abi test suites.
Summary:
The gdb pretty printer misprints variables declared via
using declarations of the form:
namespace foo {
using string_view = std::string_view;
string_view bar;
}
This change fixes that, by deferring the decision to ignore
types not inside std until after desugaring.
Reviewers: #libc!
Subscribers: broadwaylamb, libcxx-commits
Tags: #libc
Differential Revision: https://reviews.llvm.org/D76816
This reverts commit a32b94c6c3.
The buildbot startup scripts need to run as root. The buildbot
worker should have already been running as a different account.
More investigation needed.
Forcing -Werror and other warnings means that the test suite isn't
actually testing what most people are seeing in their code -- it seems
better and less arbitrary to compile these tests as close as possible
to the compiler default instead.
Removing -Werror also means that we get to differentiate between
diagnostics that are errors and those that are warnings, which makes
the test suite more precise.
Differential Revision: https://reviews.llvm.org/D76311
Some tests do not fail at all when -verify is not supported, unless some
arbitrary warning flag is added to make them fail. We currently used
-Werror=unused-result to make them fail, but doing so makes the test
suite a lot more inscrutable. It seems better to just disable those
tests when -verify is not supported.
Differential Revision: https://reviews.llvm.org/D76256
It's hard to imagine someone using a recent version of libc++ with a
roughly 3 years old Clang. Since we're not testing libc++ with Clang 3.5
anyway, claiming support for it is somewhat of a lie.
Note that we don't test Clang 4 either, however I have no reason to bump
the requirement beyond Clang 4 at the moment, whereas removing Clang 3.5
allows simplifying the test suite.
Differential Revision: https://reviews.llvm.org/D76618
This commit rewrites/removes the docker files used to create
the libc++ buildbots.
The major changes in this patch are:
1. Delete Dockerfiles used to build compilers. These have moved to
github.com/efcs/compiler-images
2. Minimize the llvm-buildbot docker image. Instead of running the
buildbots from a committed docker image, the builders now build the
image on startup. This means changes to the docker file automatically
propogate to the builders (within ~24 hours without restart).
3. Version the compilers used by the builders. This means the bots
won't start failing because the apt.llvm.org clang package updated.
Before this patch, the %run substitution did not contain the same
environment variables as normal `pass.cpp` tests. It also didn't
have the right working directory and the script wasn't aware of
potential file dependencies.
With this change, the combination of %build and %run in a .sh.cpp script
should match how pass.cpp tests are actually executed much more closely.
Summary: The return type modification has already been implemented in rL364840 and rL365290.
Reviewers: ldionne, mclow.lists, EricWF, #libc!
Reviewed By: ldionne
Subscribers: christof, dexonsmith, libcxx-commits
Tags: #libc
Differential Revision: https://reviews.llvm.org/D70275
We always want to build the table of contents. Additionally, we also
set the flag to make the output deterministic which is already the
default for llvm-ar.
Differential Revision: https://reviews.llvm.org/D74108
Louis Dionne