Summary:
When files often get touched during builds, the mtime based validation
leads to different problems in implicit modules builds, even when the
content doesn't actually change:
- Modules only: module invalidation due to out of date files. Usually causing rebuild traffic.
- Modules + PCH: build failures because clang cannot rebuild a module if it comes from building a PCH.
- PCH: build failures because clang cannot rebuild a PCH in case one of the input headers has different mtime.
This patch proposes hashing the content of input files (headers and
module maps), which is performed during serialization time. When looking
at input files for validation, clang only computes the hash in case
there's a mtime mismatch.
I've tested a couple of different hash algorithms availble in LLVM in
face of building modules+pch for `#import <Cocoa/Cocoa.h>`:
- `hash_code`: performace diff within the noise, total module cache increased by 0.07%.
- `SHA1`: 5% slowdown. Haven't done real size measurements, but it'd be BLOCK_ID+20 bytes per input file, instead of BLOCK_ID+8 bytes from `hash_code`.
- `MD5`: 3% slowdown. Like above, but BLOCK_ID+16 bytes per input file.
Given the numbers above, the patch uses `hash_code`. The patch also
improves invalidation error msgs to point out which type of problem the
user is facing: "mtime", "size" or "content".
rdar://problem/29320105
Reviewers: dexonsmith, arphaman, rsmith, aprantl
Subscribers: jkorous, cfe-commits, ributzka
Tags: #clang
Differential Revision: https://reviews.llvm.org/D67249
> llvm-svn: 374841
llvm-svn: 374895
Summary:
When files often get touched during builds, the mtime based validation
leads to different problems in implicit modules builds, even when the
content doesn't actually change:
- Modules only: module invalidation due to out of date files. Usually causing rebuild traffic.
- Modules + PCH: build failures because clang cannot rebuild a module if it comes from building a PCH.
- PCH: build failures because clang cannot rebuild a PCH in case one of the input headers has different mtime.
This patch proposes hashing the content of input files (headers and
module maps), which is performed during serialization time. When looking
at input files for validation, clang only computes the hash in case
there's a mtime mismatch.
I've tested a couple of different hash algorithms availble in LLVM in
face of building modules+pch for `#import <Cocoa/Cocoa.h>`:
- `hash_code`: performace diff within the noise, total module cache increased by 0.07%.
- `SHA1`: 5% slowdown. Haven't done real size measurements, but it'd be BLOCK_ID+20 bytes per input file, instead of BLOCK_ID+8 bytes from `hash_code`.
- `MD5`: 3% slowdown. Like above, but BLOCK_ID+16 bytes per input file.
Given the numbers above, the patch uses `hash_code`. The patch also
improves invalidation error msgs to point out which type of problem the
user is facing: "mtime", "size" or "content".
rdar://problem/29320105
Reviewers: dexonsmith, arphaman, rsmith, aprantl
Subscribers: jkorous, cfe-commits, ributzka
Tags: #clang
Differential Revision: https://reviews.llvm.org/D67249
llvm-svn: 374841
That way, lit's builtin 'env' command can be used for the 'env' bit.
Also it's clearer that way that the 'not' shouldn't cover 'env'
failures.
llvm-svn: 374749
This patch provides support for DW_LANG_C_plus_plus_11,
DW_LANG_C_plus_plus_14 tags in the Clang C++ frontend.
Patch by Sourabh Singh Tomar!
Differential Revision: https://reviews.llvm.org/D67613
Reapplies r372663 after adapting a failing test in the LLDB testsuite.
llvm-svn: 372681
This patch provides support for DW_LANG_C_plus_plus_11,
DW_LANG_C_plus_plus_14 tags in the Clang C++ frontend.
Patch by Sourabh Singh Tomar!
Differential Revision: https://reviews.llvm.org/D67613
llvm-svn: 372663
Those conditions may use __has_include, which needs to be rewritten.
The existing code has already tried to rewrite just __has_include,
but it didn't work with macro expansion, so e.g. Qt's
"#define QT_HAS_INCLUDE(x) __has_include(x)" didn't get handled
properly. Since the preprocessor run knows what each condition evaluates
to, just rewrite the entire condition. This of course requires that
the -frewrite-include pass has the same setup as the following
compilation, but that has always been the requirement.
Differential Revision: https://reviews.llvm.org/D63508
llvm-svn: 372248
Summary:
Otherwise the definition (first found) for ObjCInterfaceDecl's might
precede the module one, which will eventually lead to crash, since
diagnoseMissingImport needs one coming from a module.
This behavior changed after Richard's r342018, which started to look
into the definition of ObjCInterfaceDecls.
rdar://problem/49237144
Reviewers: rsmith, arphaman
Subscribers: jkorous, dexonsmith, ributzka, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66982
llvm-svn: 372039
Part of PR43272, the changes are:
1. Use @ as the sed pattern delimiter instead of : so that the drive
letter in lit substitutions isn't an issue.
2. Use the %/t and %/S substitutions to get paths with forward slashes
to work around string quoting issues in the yaml file.
3. Replace REQUIRES:shell with XFAIL:windows. These tests should pass on
Windows, but do not for reasons that are not yet understood. We would
like to know if they pass unexpectedly.
I was able to remove the XFAILs from two tests, since they already pass
with my sed fix:
clang/test/VFS/module_missing_vfs.m
clang/test/VFS/test_nonmodular.c
Reviewers: amccarth
Differential Revision: https://reviews.llvm.org/D67454
llvm-svn: 371663
This reverts r371497 (git commit 3d7e9ab7b9)
Reorder `not` with `env` in these two tests so they pass:
Driver/rewrite-map-in-diagnostics.c
Index/crash-recovery-modules.m.
This will not be necessary after D66531 lands.
llvm-svn: 371552
I see in the history for some of these tests REQUIRES:shell was used as
a way to disable tests on Windows because they are flaky there. I tried
not to re-enable such tests, but it's possible that I missed some and
this will re-enable flaky tests on Windows. If so, we should disable
them with UNSUPPORTED:system-windows and add a comment that they are
flaky there. So far as I can tell, the lit internal shell is capable of
running all of these tests, and we shouldn't use REQUIRES:shell as a
proxy for Windows.
llvm-svn: 371478
Marking a module for a rebuild when its signature differs from the
expected one causes redundant module rebuilds for incremental builds.
When a module is updated, its signature changes. But its consumers still
have the old signature and loading them will result in signature
mismatches. It will correctly cause the rebuilds for the consumers but
we don't need to rebuild the common module for each of them as it is
already up to date.
In practice this bug causes longer build times. We are doing more work
than required and only a single process can build a module, so parallel
builds degrade to a single-process mode where extra processes are just
waiting on a file lock.
Fix by not marking a module dependency for a rebuild on signature
mismatch. We'll check if it is up to date when we load it.
rdar://problem/50212358
Reviewers: dexonsmith, bruno, rsmith
Reviewed By: dexonsmith, bruno
Subscribers: jkorous, ributzka, cfe-commits, aprantl
Differential Revision: https://reviews.llvm.org/D66907
llvm-svn: 370274
D63793 removed float-divide-by-zero from the "undefined" set but it
failed to add it to getSupportedSanitizers(), thus the sanitizer is
rejected by the driver:
clang-9: error: unsupported option '-fsanitize=float-divide-by-zero' for target 'x86_64-unknown-linux-gnu'
Also, add SanitizerMask::FloatDivideByZero to a few other masks to make -fsanitize-trap, -fsanitize-recover, -fsanitize-minimal-runtime and -fsanitize-coverage work.
Reviewed By: rsmith, vitalybuka
Differential Revision: https://reviews.llvm.org/D64317
llvm-svn: 365587
`Selector::getIdentifierInfoForSlot` returns NULL if a slot has no
corresponding identifier. Add a boolean to the hash and a NULL check.
rdar://problem/51615164
Reviewers: rtrieu
Reviewed By: rtrieu
Subscribers: dexonsmith, cfe-commits, jkorous
Differential Revision: https://reviews.llvm.org/D63789
llvm-svn: 364664
In some cases, a typedef only strips aways a keyword for a type, keeping the
same name as the root record type. This causes some confusion when the type
is defined in one modules but only forward declared in another. Skipping the
typedef and going straight to the record will avoid this issue.
typedef struct S {} S;
S* s; // S is TypedefType here
struct S;
S* s; // S is RecordType here
llvm-svn: 364119
Summary:
This is the first in a series of changes trying to align clang -cc1
flags for Split DWARF with those of llc. The unfortunate side effect of
having -split-dwarf-output for single file Split DWARF will disappear
again in a subsequent change.
The change is the result of a discussion in D59673.
Reviewers: dblaikie, echristo
Reviewed By: dblaikie
Differential Revision: https://reviews.llvm.org/D63130
llvm-svn: 363494
Found in a bootstrap of LLVM with implicit modules, resulting in a
deadlock of some Orc unit tests with libstdc++ 8.1. An enum was used as
part of the implementation of std::recursive_mutex and this bug resulted
in the constant initialization of zero instead of the desired non-zero
value. => Badness.
Richard Smith tells me neither of these fields are necessarily canonical
& so using declaresSamEntity is the right solution here (rather than
changing both of these Fields to be canonical by construction/from their
source)
llvm-svn: 361428
This patch implements a limited form of autolinking primarily designed to allow
either the --dependent-library compiler option, or "comment lib" pragmas (
https://docs.microsoft.com/en-us/cpp/preprocessor/comment-c-cpp?view=vs-2017) in
C/C++ e.g. #pragma comment(lib, "foo"), to cause an ELF linker to automatically
add the specified library to the link when processing the input file generated
by the compiler.
Currently this extension is unique to LLVM and LLD. However, care has been taken
to design this feature so that it could be supported by other ELF linkers.
The design goals were to provide:
- A simple linking model for developers to reason about.
- The ability to to override autolinking from the linker command line.
- Source code compatibility, where possible, with "comment lib" pragmas in other
environments (MSVC in particular).
Dependent library support is implemented differently for ELF platforms than on
the other platforms. Primarily this difference is that on ELF we pass the
dependent library specifiers directly to the linker without manipulating them.
This is in contrast to other platforms where they are mapped to a specific
linker option by the compiler. This difference is a result of the greater
variety of ELF linkers and the fact that ELF linkers tend to handle libraries in
a more complicated fashion than on other platforms. This forces us to defer
handling the specifiers to the linker.
In order to achieve a level of source code compatibility with other platforms
we have restricted this feature to work with libraries that meet the following
"reasonable" requirements:
1. There are no competing defined symbols in a given set of libraries, or
if they exist, the program owner doesn't care which is linked to their
program.
2. There may be circular dependencies between libraries.
The binary representation is a mergeable string section (SHF_MERGE,
SHF_STRINGS), called .deplibs, with custom type SHT_LLVM_DEPENDENT_LIBRARIES
(0x6fff4c04). The compiler forms this section by concatenating the arguments of
the "comment lib" pragmas and --dependent-library options in the order they are
encountered. Partial (-r, -Ur) links are handled by concatenating .deplibs
sections with the normal mergeable string section rules. As an example, #pragma
comment(lib, "foo") would result in:
.section ".deplibs","MS",@llvm_dependent_libraries,1
.asciz "foo"
For LTO, equivalent information to the contents of a the .deplibs section can be
retrieved by the LLD for bitcode input files.
LLD processes the dependent library specifiers in the following way:
1. Dependent libraries which are found from the specifiers in .deplibs sections
of relocatable object files are added when the linker decides to include that
file (which could itself be in a library) in the link. Dependent libraries
behave as if they were appended to the command line after all other options. As
a consequence the set of dependent libraries are searched last to resolve
symbols.
2. It is an error if a file cannot be found for a given specifier.
3. Any command line options in effect at the end of the command line parsing apply
to the dependent libraries, e.g. --whole-archive.
4. The linker tries to add a library or relocatable object file from each of the
strings in a .deplibs section by; first, handling the string as if it was
specified on the command line; second, by looking for the string in each of the
library search paths in turn; third, by looking for a lib<string>.a or
lib<string>.so (depending on the current mode of the linker) in each of the
library search paths.
5. A new command line option --no-dependent-libraries tells LLD to ignore the
dependent libraries.
Rationale for the above points:
1. Adding the dependent libraries last makes the process simple to understand
from a developers perspective. All linkers are able to implement this scheme.
2. Error-ing for libraries that are not found seems like better behavior than
failing the link during symbol resolution.
3. It seems useful for the user to be able to apply command line options which
will affect all of the dependent libraries. There is a potential problem of
surprise for developers, who might not realize that these options would apply
to these "invisible" input files; however, despite the potential for surprise,
this is easy for developers to reason about and gives developers the control
that they may require.
4. This algorithm takes into account all of the different ways that ELF linkers
find input files. The different search methods are tried by the linker in most
obvious to least obvious order.
5. I considered adding finer grained control over which dependent libraries were
ignored (e.g. MSVC has /nodefaultlib:<library>); however, I concluded that this
is not necessary: if finer control is required developers can fall back to using
the command line directly.
RFC thread: http://lists.llvm.org/pipermail/llvm-dev/2019-March/131004.html.
Differential Revision: https://reviews.llvm.org/D60274
llvm-svn: 360984
Currently `immintrin.h` includes `pconfigintrin.h` and `sgxintrin.h`
which contain inline assembly. It causes failures when building with the
flag `-fno-gnu-inline-asm`.
Fix by excluding functions with inline assembly when this extension is
disabled. So far there was no need to support `_pconfig_u32`,
`_enclu_u32`, `_encls_u32`, `_enclv_u32` on platforms that require
`-fno-gnu-inline-asm`. But if developers start using these functions,
they'll have compile-time undeclared identifier errors which is
preferrable to runtime errors.
rdar://problem/49540880
Reviewers: craig.topper, GBuella, rnk, echristo
Reviewed By: rnk
Subscribers: jkorous, dexonsmith, cfe-commits
Differential Revision: https://reviews.llvm.org/D61621
llvm-svn: 360630
template name is not visible to unqualified lookup.
In order to support this without a severe degradation in our ability to
diagnose typos in template names, this change significantly restructures
the way we handle template-id-shaped syntax for which lookup of the
template name finds nothing.
Instead of eagerly diagnosing an undeclared template name, we now form a
placeholder template-name representing a name that is known to not find
any templates. When the parser sees such a name, it attempts to
disambiguate whether we have a less-than comparison or a template-id.
Any diagnostics or typo-correction for the name are delayed until its
point of use.
The upshot should be a small improvement of our diagostic quality
overall: we now take more syntactic context into account when trying to
resolve an undeclared identifier on the left hand side of a '<'. In
fact, this works well enough that the backwards-compatible portion (for
an undeclared identifier rather than a lookup that finds functions but
no function templates) is enabled in all language modes.
llvm-svn: 360308
This trips over a few other limitations, but in the interests of incremental development I'm starting here & I'll look at the issues with -verify and filesystem checks (the fact that the behavior depends on the existence of a 'foo' directory even though it shouldn't need it), etc.
Reviewers: rsmith
Differential Revision: https://reviews.llvm.org/D61656
llvm-svn: 360195
When a FunctionProtoType is in the original type in a DecayedType, the decayed
type is a PointerType which points back the original FunctionProtoType. The
visitor for ODRHashing will attempt to process both Type's, doing double work.
By chaining together multiple DecayedType's and FunctionProtoType's, this would
result in 2^N Type's visited only N DecayedType's and N FunctionProtoType's
exsit. Another bug where VisitDecayedType and VisitAdjustedType did
redundant work doubled the work at each level, giving 4^N Type's visited. This
patch removed the double work and detects when a FunctionProtoType decays to
itself to only check the Type once. This lowers the exponential runtime to
linear runtime. Fixes https://bugs.llvm.org/show_bug.cgi?id=41625
llvm-svn: 359960
(this would regress size without a corresponding LLVM change that avoids
putting other user defined types inside type units when they aren't in
their own type units - instead emitting declarations inside the type
unit and a definition in the primary CU)
Reviewers: aprantl
Differential Revision: https://reviews.llvm.org/D61079
llvm-svn: 359235
Exposed by a related bug about visibility of default arguments of nested
templates - without the correct decl context, default template
parameters of variable templates nested in classes would have incorrect
visibility computed.
llvm-svn: 358796
The code is/was already correct for the case where a parameter is a
parameter of its enclosing lexical DeclContext (functions and classes).
But for other templates (alias and variable templates) they don't create
their own scope to be members of - in those cases, they parameter should
be considered visible if any definition of the lexical decl context is
visible.
[this should cleanup the failure on the libstdc++ modules buildbot]
[this doesn't actually fix the variable template case for a
secondary/compounding reason (its lexical decl context is incorrectly
considered to be the translation unit)]
Test covers all 4 kinds of templates with default args, including a
regression test for the still broken variable template case.
Reviewers: rsmith
Differential Revision: https://reviews.llvm.org/D60892
llvm-svn: 358795
Leverage the InMemoryModuleCache to invalidate a module the first time
it fails to import (and to lock a module as soon as it's built or
imported successfully). For implicit module builds, this optimizes
importing deep graphs where the leaf module is out-of-date; see example
near the end of the commit message.
Previously the cache finalized ("locked in") all modules imported so far
when starting a new module build. This was sufficient to prevent
loading two versions of the same module, but was somewhat arbitrary and
hard to reason about.
Now the cache explicitly tracks module state, where each module must be
one of:
- Unknown: module not in the cache (yet).
- Tentative: module in the cache, but not yet fully imported.
- ToBuild: module found on disk could not be imported; need to build.
- Final: module in the cache has been successfully built or imported.
Preventing repeated failed imports avoids variation in builds based on
shifting filesystem state. Now it's guaranteed that a module is loaded
from disk exactly once. It now seems safe to remove
FileManager::invalidateCache, but I'm leaving that for a later commit.
The new, precise logic uncovered a pre-existing problem in the cache:
the map key is the module filename, and different contexts use different
filenames for the same PCM file. (In particular, the test
Modules/relative-import-path.c does not build without this commit.
r223577 started using a relative path to describe a module's base
directory when importing it within another module. As a result, the
module cache sees an absolute path when (a) building the module or
importing it at the top-level, and a relative path when (b) importing
the module underneath another one.)
The "obvious" fix is to resolve paths using FileManager::getVirtualFile
and change the map key for the cache to a FileEntry, but some contexts
(particularly related to ASTUnit) have a shorter lifetime for their
FileManager than the InMemoryModuleCache. This is worth pursuing
further in a later commit; perhaps by tying together the FileManager and
InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a
VFS layer.
For now, use the PCM's base directory as-written for constructing the
filename to check the ModuleCache.
Example
=======
To understand the build optimization, first consider the build of a
module graph TU -> A -> B -> C -> D with an empty cache:
TU builds A'
A' builds B'
B' builds C'
C' builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
If we build TU again, where A, B, C, and D are in the cache and D is
out-of-date, we would previously get this build:
TU imports A
imports B
imports C
imports D (out-of-date)
TU builds A'
A' imports B
imports C
imports D (out-of-date)
builds B'
B' imports C
imports D (out-of-date)
builds C'
C' imports D (out-of-date)
builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
After this commit, we'll immediateley invalidate A, B, C, and D when we
first observe that D is out-of-date, giving this build:
TU imports A
imports B
imports C
imports D (out-of-date)
TU builds A' // The same graph as an empty cache.
A' builds B'
B' builds C'
C' builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
The new build matches what we'd naively expect, pretty closely matching
the original build with the empty cache.
rdar://problem/48545366
llvm-svn: 355778
Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module build
more explicitly, but this is split out because it's mostly mechanical.
Because of the move to Serialization we can no longer abuse the
Preprocessor to forward it to the ASTReader. Besides the rename and
file move, that means Preprocessor::Preprocessor has one fewer parameter
and ASTReader::ASTReader has one more.
llvm-svn: 355777
Add a remark for importing modules. Depending on whether this is a
direct import (into the TU being built by this compiler instance) or
transitive import (into an already-imported module), the diagnostic has
two forms:
importing module 'Foo' from 'path/to/Foo.pcm'
importing module 'Foo' into 'Bar' from 'path/to/Foo.pcm'
Also drop a redundant FileCheck invocation in Rmodule-build.m that was
using -Reverything, since the notes from -Rmodule-import were confusing
it.
https://reviews.llvm.org/D58891
llvm-svn: 355477
When a module name is specified as -fmodule-name, that module gets a
clang::Module object, but it won't actually be built or imported; it
will be textual. CGDebugInfo wouldn't detect this and them emit a
DICompileUnit that had a hash but no name and that confused both
dsymutil, LLDB, and myself.
rdar://problem/47926508
Differential Revision: https://reviews.llvm.org/D57976
llvm-svn: 353578
Some of these functions take some extraneous arguments, e.g. EltSize,
Offset, which are computable from the Type and DataLayout.
Add some asserts to ensure that the computed values are consistent
with the passed-in values, in preparation for eliminating the
extraneous arguments. This also asserts that the Type is an Array for
the calls named "Array" and a Struct for the calls named "Struct".
Then, correct a couple of errors:
1. Using CreateStructGEP on an array type. (this causes the majority
of the test differences, as struct GEPs are created with i32
indices, while array GEPs are created with i64 indices)
2. Passing the wrong Offset to CreateStructGEP in TargetInfo.cpp on
x86-64 NACL (which uses 32-bit pointers).
Differential Revision: https://reviews.llvm.org/D57766
llvm-svn: 353529
For global variables with unordered initialization that are instantiated
within a module, we previously did not emit the global (or its
initializer) at all unless it was used in the importing translation unit
(and sometimes not even then!), leading to misbehavior and link errors.
We now emit the initializer for an instantiated global variable with
unordered initialization with side-effects in a module into every
translation unit that imports the module. This is unfortunate, but
mostly matches the behavior of a non-modular compilation and seems to be
the best that we can reasonably do.
llvm-svn: 353240
When Clang/LLVM is built with the CLANG_DEFAULT_STD_CXX CMake macro that sets
the default standard to something other than C++14, there are a number of lit
tests that fail as they rely on the C++14 default.
This patch just adds the language standard option explicitly to such test cases.
Differential revision: https://reviews.llvm.org/D57581
llvm-svn: 353163
It is intended to disable _all_ warnings, even those upgraded to
errors via `-Werror=warningname` or `#pragma clang diagnostic error'
Fixes: https://llvm.org/PR38231
Differential Revision: https://reviews.llvm.org/D53199
llvm-svn: 352535
The autolinking extension for ELF uses a slightly different format for
encoding the autolink information compared to COFF and MachO. Account
for this in the CGM to ensure that we do not assert when emitting
assembly or an object file.
llvm-svn: 350476
Bruno Cardoso Lopes