This patch adds support for a `header` declaration in a module map to specify
certain `stat` information (currently, size and mtime) about that header file.
This has two purposes:
- It removes the need to eagerly `stat` every file referenced by a module map.
Instead, we track a list of unresolved header files with each size / mtime
(actually, for simplicity, we track submodules with such headers), and when
attempting to look up a header file based on a `FileEntry`, we check if there
are any unresolved header directives with that `FileEntry`'s size / mtime and
perform deferred `stat`s if so.
- It permits a preprocessed module to be compiled without the original files
being present on disk. The only reason we used to need those files was to get
the `stat` information in order to do header -> module lookups when using the
module. If we're provided with the `stat` information in the preprocessed
module, we can avoid requiring the files to exist.
Unlike most `header` directives, if a `header` directive with `stat`
information has no corresponding on-disk file the enclosing module is *not*
marked unavailable (so that behavior is consistent regardless of whether we've
resolved a header directive, and so that preprocessed modules don't get marked
unavailable). We could actually do this for all `header` directives: the only
reason we mark the module unavailable if headers are missing is to give a
diagnostic slightly earlier (rather than waiting until we actually try to build
the module / load and validate its .pcm file).
Differential Revision: https://reviews.llvm.org/D33703
llvm-svn: 304515
to the original module map.
Also use the path and name of the original module map when emitting that
information into the .pcm file. The upshot of this is that the produced .pcm
file will track information for headers in their original locations (where the
module was preprocessed), not relative to whatever directory the preprocessed
module map was in when it was built.
llvm-svn: 304346
Previously, a preamble only included #if blocks (and friends like
ifdef) if there was a corresponding #endif before any declaration or
definition. The problem is that any header file that uses include guards
will not have a preamble generated, which can make code-completion very
slow.
To prevent errors about unbalanced preprocessor conditionals in the
preamble, and unbalanced preprocessor conditionals after a preamble
containing unfinished conditionals, the conditional stack is stored
in the pch file.
This fixes PR26045.
Differential Revision: http://reviews.llvm.org/D15994
llvm-svn: 304207
inferring based on the current module at the point of creation.
This should result in no functional change except when building a preprocessed
module (or more generally when using #pragma clang module begin/end to switch
module in the middle of a file), in which case it allows us to correctly track
the owning module for declarations. We can't map from FileID to module in the
preprocessed module case, since all modules would have the same FileID.
There are still a couple of remaining places that try to infer a module from a
source location; I'll clean those up in follow-up changes.
llvm-svn: 303322
I think this is a false positive in GCC's warning, but nonetheless, we
should try to be warning-free. Smaller reproducer (reproduces with GCC
6.3):
https://godbolt.org/g/cJuO2z
llvm-svn: 302003
The intent for an explicit module build is that the diagnostics produced within
the module are those that were configured when the module was built, not those
that are enabled within a user of the module. This includes diagnostics that
don't actually show up until the module is used (for instance, diagnostics
produced during template instantiation and weird cases like -Wpadded).
We serialized and restored the diagnostic state for individual warning groups,
but previously did not track the state for flags like -Werror and -Weverything,
which are implemented as separate bits rather than as part of the diagnostics
mapping information.
llvm-svn: 301992
The modules side of r299226, which serializes #pragma pack state,
doesn't work well.
The main purpose was to make -include and -include-pch match semantics
(the PCH side). We also started serializing #pragma pack in PCMs, in
the hopes of making modules and non-modules builds more consistent. But
consider:
$ cat a.h
$ cat b.h
#pragma pack(push, 2)
$ cat module.modulemap
module M {
module a { header "a.h" }
module b { header "b.h" }
}
$ cat t.cpp
#include "a.h"
#pragma pack(show)
As of r299226, the #pragma pack(show) gives "2", even though we've only
included "a.h".
- With -fmodules-local-submodule-visibility, this is clearly wrong. We
should get the default state (8 on x86_64).
- Without -fmodules-local-submodule-visibility, this kind of matches how
other things work (as if include-the-whole-module), but it's still
really terrible, and it doesn't actually make modules and non-modules
builds more consistent.
This commit disables the serialization for modules, essentially a
partial revert of r299226.
Going forward:
1. Having this #pragma pack stuff escape is terrible design (or, more
often, a horrible bug). We should prioritize adding warnings (maybe
-Werror by default?).
2. If we eventually reintroduce this for modules, it should only apply
to -fmodules-local-submodule-visibility, and it should be tracked on
a per-submodule basis.
llvm-svn: 300380
This allows using and testing these two features separately. (noteably,
debug info is, so far as I know, always a win (basically). But function
modular codegen is currently a loss for highly optimized code - where
most of the linkonce_odr definitions are optimized away, so providing
weak_odr definitions is only overhead)
llvm-svn: 300104
r293123 started serializing diagnostic pragma state for modules. This
makes the serialization work properly for implicit modules.
An implicit module build (using Clang's internal build system) uses the
same PCM file location for different `-Werror` levels.
E.g., if a TU has `-Werror=format` and tries to load a PCM built without
`-Werror=format`, a new PCM will be built in its place (and the new PCM
should have the same signature, since r297655). In the other direction,
if a TU does not have `-Werror=format` and tries to load a PCM built
with `-Werror=format`, it should "just work".
The idea is to evolve the PCM toward the strictest -Werror flags that
anyone tries.
r293123 started serializing the diagnostic pragma state for each PCM.
Since this encodes the -Werror settings at module-build time, it breaks
the implicit build model.
This commit filters the diagnostic state in order to simulate the
current compilation's diagnostic settings. Firstly, it ignores the
module's serialized first diagnostic state, replacing it with the state
from this compilation's command-line. Secondly, if a pragma warning was
upgraded to error/fatal when generating the PCM (e.g., due to `-Werror`
on the command-line), it checks whether it should still be upgraded in
its current context.
llvm-svn: 300025
Emit the final diagnostic state last to match source order. This also
prepares for a follow-up commit for implicit modules.
There's no real functionaliy change, just a slightly different AST file
format.
llvm-svn: 300024
Calculating the hash in Sema::ActOnTagFinishDefinition could happen before
all sub-Decls were parsed or processed, which would produce the wrong hash
value. Change to calculating the hash on the first use and storing the value
instead. Also, avoid using the macros that were only for Boolean fields and
use an explicit checker during the DefintionData merge. No functional change,
but was this blocking other ODRHash patches.
llvm-svn: 299989
Matching the function-homing support for modular codegen. Any type
implicitly (implicit template specializations) or explicitly defined in
a module is attached to that module's object file and omitted elsewhere
(only a declaration used if necessary for references).
llvm-svn: 299987
Some decls are created not where they are written, but in other module
files/users (implicit special members and function template implicit
specializations). To correctly identify them, use a bit next to the definition
to track the modular codegen property.
Discussed whether the module file bit could be omitted in favor of
reconstituting from the modular codegen decls list - best guess today is that
the efficiency improvement of not having to deserialize the whole list whenever
any function is queried by a module user is worth it for the small size
increase of this redundant (list + bit-on-def) representation.
Reviewers: rsmith
Differential Revision: https://reviews.llvm.org/D29901
llvm-svn: 299982
This patch serializes the state of #pragma pack. It preserves the state of the
pragma from a PCH/from modules in a file that uses that PCH/those modules.
rdar://21359084
Differential Revision: https://reviews.llvm.org/D31241
llvm-svn: 299226
Sema holds the current FPOptions which is adjusted by 'pragma STDC
FP_CONTRACT'. This then gets propagated into expression nodes as they are
built.
This encapsulates FPOptions so that this propagation happens opaquely rather
than directly with the fp_contractable on/off bit. This allows controlled
transitioning of fp_contractable to a ternary value (off, on, fast). It will
also allow adding more fast-math flags later.
This is toward moving fp-contraction=fast from an LLVM TargetOption to a
FastMathFlag in order to fix PR25721.
Differential Revision: https://reviews.llvm.org/D31166
llvm-svn: 298877
This reverts commit r298185, effectively reapplying r298165, after fixing the
new unit tests (PR32338). The memory buffer generator doesn't null-terminate
the MemoryBuffer it creates; this version of the commit informs getMemBuffer
about that to avoid the assert.
Original commit message follows:
----
Clang's internal build system for implicit modules uses lock files to
ensure that after a process writes a PCM it will read the same one back
in (without contention from other -cc1 commands). Since PCMs are read
from disk repeatedly while invalidating, building, and importing, the
lock is not released quickly. Furthermore, the LockFileManager is not
robust in every environment. Other -cc1 commands can stall until
timeout (after about eight minutes).
This commit changes the lock file from being necessary for correctness
to a (possibly dubious) performance hack. The remaining benefit is to
reduce duplicate work in competing -cc1 commands which depend on the
same module. Follow-up commits will change the internal build system to
continue after a timeout, and reduce the timeout. Perhaps we should
reconsider blocking at all.
This also fixes a use-after-free, when one part of a compilation
validates a PCM and starts using it, and another tries to swap out the
PCM for something new.
The PCMCache is a new type called MemoryBufferCache, which saves memory
buffers based on their filename. Its ownership is shared by the
CompilerInstance and ModuleManager.
- The ModuleManager stores PCMs there that it loads from disk, never
touching the disk if the cache is hot.
- When modules fail to validate, they're removed from the cache.
- When a CompilerInstance is spawned to build a new module, each
already-loaded PCM is assumed to be valid, and is frozen to avoid
the use-after-free.
- Any newly-built module is written directly to the cache to avoid the
round-trip to the filesystem, making lock files unnecessary for
correctness.
Original patch by Manman Ren; most testcases by Adrian Prantl!
llvm-svn: 298278
Clang's internal build system for implicit modules uses lock files to
ensure that after a process writes a PCM it will read the same one back
in (without contention from other -cc1 commands). Since PCMs are read
from disk repeatedly while invalidating, building, and importing, the
lock is not released quickly. Furthermore, the LockFileManager is not
robust in every environment. Other -cc1 commands can stall until
timeout (after about eight minutes).
This commit changes the lock file from being necessary for correctness
to a (possibly dubious) performance hack. The remaining benefit is to
reduce duplicate work in competing -cc1 commands which depend on the
same module. Follow-up commits will change the internal build system to
continue after a timeout, and reduce the timeout. Perhaps we should
reconsider blocking at all.
This also fixes a use-after-free, when one part of a compilation
validates a PCM and starts using it, and another tries to swap out the
PCM for something new.
The PCMCache is a new type called MemoryBufferCache, which saves memory
buffers based on their filename. Its ownership is shared by the
CompilerInstance and ModuleManager.
- The ModuleManager stores PCMs there that it loads from disk, never
touching the disk if the cache is hot.
- When modules fail to validate, they're removed from the cache.
- When a CompilerInstance is spawned to build a new module, each
already-loaded PCM is assumed to be valid, and is frozen to avoid
the use-after-free.
- Any newly-built module is written directly to the cache to avoid the
round-trip to the filesystem, making lock files unnecessary for
correctness.
Original patch by Manman Ren; most testcases by Adrian Prantl!
llvm-svn: 298165
Since bitcode uses VBR encoding, large numbers are more expensive than
small ones. Instead of emitting a UINT_MAX sentinel after each sequence
of state-change pairs, emit the size of the sequence as a prefix.
This should have no functionality change besides saving bits from the
encoding.
llvm-svn: 297770
Change ASTFileSignature from a random 32-bit number to the hash of the
PCM content.
- Move definition ASTFileSignature to Basic/Module.h so Module and
ASTSourceDescriptor can use it.
- Change the signature from uint64_t to std::array<uint32_t,5>.
- Stop using (saving/reading) the size and modification time of PCM
files when there is a valid SIGNATURE.
- Add UNHASHED_CONTROL_BLOCK, and use it to store the SIGNATURE record
and other records that shouldn't affect the hash. Because implicit
modules reuses the same file for multiple levels of -Werror, this
includes DIAGNOSTIC_OPTIONS and DIAG_PRAGMA_MAPPINGS.
This helps to solve a PCH + implicit Modules dependency issue: PCH files
are handled by the external build system, whereas implicit modules are
handled by internal compiler build system. This prevents invalidating a
PCH when the compiler overwrites a PCM file with the same content
(modulo the diagnostic differences).
Design and original patch by Manman Ren!
llvm-svn: 297655
Essentially, as a base class constructor does not construct virtual bases, such
a constructor for an abstract class does not need the corresponding base class
construction to be valid, and likewise for destructors.
This creates an awkward situation: clang will sometimes generate references to
the complete object and deleting destructors for an abstract class (it puts
them in the construction vtable for a derived class). But we can't generate a
"correct" version of these because we can't generate references to base class
constructors any more (if they're template specializations, say, we might not
have instantiated them and can't assume any other TU will emit a copy).
Fortunately, we don't need to, since no correct program can ever invoke them,
so instead emit symbols that just trap.
We should stop emitting references to these symbols, but still need to emit
definitions for compatibility.
llvm-svn: 296275
Reserve a spot for ODR hash in CXXRecordDecl and in its modules storage.
Default the hash value to 0 for all classes.
Differential Revision: https://reviews.llvm.org/D21675
llvm-svn: 295533
A slightly weaker form of ODR checking than previous attempts, but hopefully
won't break the modules build bot. Future work will be needed to catch all
cases.
When objects are imported for modules, there is a chance that a name collision
will cause an ODR violation. Previously, only a small number of such
violations were detected. This patch provides a stronger check based on
AST nodes.
The information needed to uniquely identify an object is taken from the AST and
put into a one-dimensional byte stream. This stream is then hashed to give
a value to represent the object, which is stored with the other object data
in the module.
When modules are loaded, and Decl's are merged, the hash values of the two
Decl's are compared. Only Decl's with matched hash values will be merged.
Mismatch hashes will generate a module error, and if possible, point to the
first difference between the two objects.
The transform from AST to byte stream is a modified depth first algorithm.
Due to references between some AST nodes, a pure depth first algorithm could
generate loops. For Stmt nodes, a straight depth first processing occurs.
For Type and Decl nodes, they are replaced with an index number and only on
first visit will these nodes be processed. As an optimization, boolean
values are saved and stored together in reverse order at the end of the
byte stream to lower the ammount of data that needs to be hashed.
Compile time impact was measured at 1.5-2.0% during module building, and
negligible during builds without module building.
Differential Revision: https://reviews.llvm.org/D21675
llvm-svn: 295421
Recommit r293585 that was reverted in r293611 with new fixes. The previous
issue was determined to be an overly aggressive AST visitor from forward
declared objects. The visitor will now only deeply visit certain Decl's and
only do a shallow information extraction from all other Decl's.
When objects are imported for modules, there is a chance that a name collision
will cause an ODR violation. Previously, only a small number of such
violations were detected. This patch provides a stronger check based on
AST nodes.
The information needed to uniquely identify an object is taken from the AST and
put into a one-dimensional byte stream. This stream is then hashed to give
a value to represent the object, which is stored with the other object data
in the module.
When modules are loaded, and Decl's are merged, the hash values of the two
Decl's are compared. Only Decl's with matched hash values will be merged.
Mismatch hashes will generate a module error, and if possible, point to the
first difference between the two objects.
The transform from AST to byte stream is a modified depth first algorithm.
Due to references between some AST nodes, a pure depth first algorithm could
generate loops. For Stmt nodes, a straight depth first processing occurs.
For Type and Decl nodes, they are replaced with an index number and only on
first visit will these nodes be processed. As an optimization, boolean
values are saved and stored together in reverse order at the end of the
byte stream to lower the ammount of data that needs to be hashed.
Compile time impact was measured at 1.5-2.0% during module building, and
negligible during builds without module building.
Differential Revision: https://reviews.llvm.org/D21675
llvm-svn: 295284
Following up on r291465 after a regression in r276159. When we use
-fmodule-name=X while building a PCH, modular headers in X will be
textually included and the compiler knows that we are not building
module X, so don't serialize such headers in the PCH as being part of a
module, because at this point they are not.
This was causing subtle bugs and malformed AST crashes, for instance,
when using the PCH in subsequent compiler invocation with -fmodules, the
HFI for a modular header would map to the PCH, which would force a
module load of and unexistent module ID.
rdar://problem/30171164
llvm-svn: 294361
We model deduction-guides as functions with a new kind of name that identifies
the template whose deduction they guide; the bulk of this patch is adding the
new name kind. This gives us a clean way to attach an extensible list of guides
to a class template in a way that doesn't require any special handling in AST
files etc (and we're going to need these functions we come to performing
deduction).
llvm-svn: 294266
The Module::WithCodegen flag was only being set when the module was
parsed from a ModuleMap. Instead set it late, in the ASTWriter to match
the layer where the MODULAR_CODEGEN_DECLs list is determined (the
WithCodegen flag essentially means "are this module's decls in
MODULAR_CODEGEN_DECLs").
When simultaneous emission of AST file and modular object is implemented
this may need to change - the Module::WithCodegen flag will need to be
set earlier, and ideally the MODULAR_CODEGEN_DECLs gathering will
consult this flag (that's not possible right now since Decls destined
for an AST File don't have a Module - only if they're /read/ from a
Module is that true - I expect that would need to change as well).
llvm-svn: 293692
We're seeing what we believe are false positives. (It's hard to tell with the
available diagnostics, and I'm not sure how to reduce them yet).
I'll send Richard reproduction details offline.
djasper/chandlerc suggested this should be a warning for now, to make rolling it
out feasible.
llvm-svn: 293611
When objects are imported for modules, there is a chance that a name collision
will cause an ODR violation. Previously, only a small number of such
violations were detected. This patch provides a stronger check based on
AST nodes.
The information needed to uniquely identify an object is taked from the AST and
put into a one-dimensional byte stream. This stream is then hashed to give
a value to represent the object, which is stored with the other object data
in the module.
When modules are loaded, and Decl's are merged, the hash values of the two
Decl's are compared. Only Decl's with matched hash values will be merged.
Mismatch hashes will generate a module error, and if possible, point to the
first difference between the two objects.
The transform from AST to byte stream is a modified depth first algorithm.
Due to references between some AST nodes, a pure depth first algorithm could
generate loops. For Stmt nodes, a straight depth first processing occurs.
For Type and Decl nodes, they are replaced with an index number and only on
first visit will these nodes be processed. As an optimization, boolean
values are saved and stored together in reverse order at the end of the
byte stream to lower the ammount of data that needs to be hashed.
Compile time impact was measured at 1.5-2.0% during module building, and
negligible during builds without module building.
Differential Revision: https://reviews.llvm.org/D21675
llvm-svn: 293585
First pass at generating weak definitions of inline functions from module files
(& skipping (-O0) or emitting available_externally (optimizations)
definitions where those modules are used).
External functions defined in modules are emitted into the modular
object file as well (this may turn an existing ODR violation (if that
module were imported into multiple translations) into valid/linkable
code).
Internal symbols (static functions, for example) are not correctly
supported yet. The symbol will be produced, internal, in the modular
object - unreferenceable from the users.
Reviewers: rsmith
Differential Revision: https://reviews.llvm.org/D28845
llvm-svn: 293456
Hide the pointer indirection in ModuleManager::begin, ModuleManager::end,
ModuleManager::rbegin, and ModuleManager::rend. Besides tidying up the call
sites, this is preparation for making ownership of ModuleFile explicit.
llvm-svn: 293394
This change adds a new type node, DeducedTemplateSpecializationType, to
represent a type template name that has been used as a type. This is modeled
around AutoType, and shares a common base class for representing a deduced
placeholder type.
We allow deduced class template types in a few more places than the standard
does: in conditions and for-range-declarators, and in new-type-ids. This is
consistent with GCC and with discussion on the core reflector. This patch
does not yet support deduced class template types being named in typename
specifiers.
llvm-svn: 293207
Rather than storing a single flat list of SourceLocations where the diagnostic
state changes (in source order), we now store a separate list for each FileID
in which there is a diagnostic state transition. (State for other files is
built and cached lazily, on demand.) This has two consequences:
1) We can now sensibly support modules, and properly track the diagnostic state
for modular headers (this matters when, for instance, triggering instantiation
of a template defined within a module triggers diagnostics).
2) It's much faster than the old approach, since we can now just do a binary
search on the offsets within the FileID rather than needing to call
isBeforeInTranslationUnit to determine source order (which is surprisingly
slow). For some pathological (but real world) files, this reduces total
compilation time by more than 10%.
For now, the diagnostic state points for modules are loaded eagerly. It seems
feasible to defer this until diagnostic state information for one of the
module's files is needed, but that's not part of this patch.
llvm-svn: 293123
D28684 changed llvm::zlib to return Error instead of Status.
It was accepted and committed in r292214, but then reverted in r292217
because I missed that clang code also needs to be updated.
Patch do that.
D28684 recommitted again as r292226
Differential revision: https://reviews.llvm.org/D28807
llvm-svn: 292227
Summary:
We do not currently track the source locations for exception specifications such
that their source range can be queried through the AST. This leads to trying to
write more complex code to determine the source range for uses like FixItHints
(see D18575 for an example). In addition to use within tools like clang-tidy, I
think this information may become more important to track as exception
specifications become more integrated into the type system.
Patch by Don Hinton.
Reviewers: rsmith
Subscribers: malcolm.parsons, sbarzowski, alexfh, hintonda, cfe-commits
Differential Revision: https://reviews.llvm.org/D20428
llvm-svn: 291771
In r276159, we started to say that a module X is defined in a pch if we specify
-fmodule-name when building the pch. This caused a regression that reports
module X is defined in both pch and pcm if we generate the pch with
-fmodule-name=X and then in a separate clang invocation, we include the pch and
also import X.pcm.
This patch adds an option CompilingPCH similar to CompilingModule. When we use
-fmodule-name=X while building a pch, modular headers in X will be textually
included and the compiler knows that we are not building module X, so we don't
put module X in SUBMODULE_DEFINITION of the pch.
Differential Revision: http://reviews.llvm.org/D28415
llvm-svn: 291465
Richard Smith