Leverage the InMemoryModuleCache to invalidate a module the first time
it fails to import (and to lock a module as soon as it's built or
imported successfully). For implicit module builds, this optimizes
importing deep graphs where the leaf module is out-of-date; see example
near the end of the commit message.
Previously the cache finalized ("locked in") all modules imported so far
when starting a new module build. This was sufficient to prevent
loading two versions of the same module, but was somewhat arbitrary and
hard to reason about.
Now the cache explicitly tracks module state, where each module must be
one of:
- Unknown: module not in the cache (yet).
- Tentative: module in the cache, but not yet fully imported.
- ToBuild: module found on disk could not be imported; need to build.
- Final: module in the cache has been successfully built or imported.
Preventing repeated failed imports avoids variation in builds based on
shifting filesystem state. Now it's guaranteed that a module is loaded
from disk exactly once. It now seems safe to remove
FileManager::invalidateCache, but I'm leaving that for a later commit.
The new, precise logic uncovered a pre-existing problem in the cache:
the map key is the module filename, and different contexts use different
filenames for the same PCM file. (In particular, the test
Modules/relative-import-path.c does not build without this commit.
r223577 started using a relative path to describe a module's base
directory when importing it within another module. As a result, the
module cache sees an absolute path when (a) building the module or
importing it at the top-level, and a relative path when (b) importing
the module underneath another one.)
The "obvious" fix is to resolve paths using FileManager::getVirtualFile
and change the map key for the cache to a FileEntry, but some contexts
(particularly related to ASTUnit) have a shorter lifetime for their
FileManager than the InMemoryModuleCache. This is worth pursuing
further in a later commit; perhaps by tying together the FileManager and
InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a
VFS layer.
For now, use the PCM's base directory as-written for constructing the
filename to check the ModuleCache.
Example
=======
To understand the build optimization, first consider the build of a
module graph TU -> A -> B -> C -> D with an empty cache:
TU builds A'
A' builds B'
B' builds C'
C' builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
If we build TU again, where A, B, C, and D are in the cache and D is
out-of-date, we would previously get this build:
TU imports A
imports B
imports C
imports D (out-of-date)
TU builds A'
A' imports B
imports C
imports D (out-of-date)
builds B'
B' imports C
imports D (out-of-date)
builds C'
C' imports D (out-of-date)
builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
After this commit, we'll immediateley invalidate A, B, C, and D when we
first observe that D is out-of-date, giving this build:
TU imports A
imports B
imports C
imports D (out-of-date)
TU builds A' // The same graph as an empty cache.
A' builds B'
B' builds C'
C' builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
The new build matches what we'd naively expect, pretty closely matching
the original build with the empty cache.
rdar://problem/48545366
llvm-svn: 355778
Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module build
more explicitly, but this is split out because it's mostly mechanical.
Because of the move to Serialization we can no longer abuse the
Preprocessor to forward it to the ASTReader. Besides the rename and
file move, that means Preprocessor::Preprocessor has one fewer parameter
and ASTReader::ASTReader has one more.
llvm-svn: 355777
In 64 bit MSVC environment size_t is defined as unsigned long long.
In single source language like HIP, data layout should be consistent
in device and host compilation, therefore copy data layout controlling
fields from Aux target for AMDGPU target.
Differential Revision: https://reviews.llvm.org/D56318
llvm-svn: 352620
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
When debugging a boost build with a modified
version of Clang, I discovered that the PTH implementation
stores TokenKind in 8 bits. However, we currently have 368
TokenKinds.
The result is that the value gets truncated and the wrong token
gets picked up when including PTH files. It seems that this will
go wrong every time someone uses a token that uses the 9th bit.
Upon asking on IRC, it was brought up that this was a highly
experimental features that was considered a failure. I discovered
via googling that BoostBuild (mostly Boost.Math) is the only user of
this
feature, using the CC1 flag directly. I believe that this can be
transferred over to normal PCH with minimal effort:
https://github.com/boostorg/build/issues/367
Based on advice on IRC and research showing that this is a nearly
completely unused feature, this patch removes it entirely.
Note: I considered leaving the build-flags in place and making them
emit an error/warning, however since I've basically identified and
warned the only user, it seemed better to just remove them.
Differential Revision: https://reviews.llvm.org/D54547
Change-Id: If32744275ef1f585357bd6c1c813d96973c4d8d9
llvm-svn: 348266
The current version only emits the below error for a module (attempted to be loaded) from the `prebuilt-module-path`:
```
error: module file blabla.pcm cannot be loaded due to a configuration mismatch with the current compilation [-Wmodule-file-config-mismatch]
```
With this change, if the prebuilt module is used, we allow the proper diagnostic behind the configuration mismatch to be shown.
```
error: POSIX thread support was disabled in PCH file but is currently enabled
error: module file blabla.pcm cannot be loaded due to a configuration mismatch with the current compilation [-Wmodule-file-config-mismatch]
```
(A few lines later an error is emitted anyways, so there is no reason not to complain for configuration mismatches if a config mismatch is found and kills the build.)
Reviewed By: dblaikie
Tags: #clang
Differential Revision: https://reviews.llvm.org/D53334
llvm-svn: 346439
This reverts commit r345963. We have a path forward now.
Original commit message:
The driver accidentally stopped passing the input filenames on to -cc1
in this mode due to confusion over what action was being requested.
This change also fixes a couple of crashes I encountered when passing
multiple files to such a -cc1 invocation.
llvm-svn: 346130
This reverts commit r345803 and r345915 (a follow-up fix to r345803).
Reason: r345803 blocks our internal integrate because of the new
warnings showing up in too many places. The fix is actually correct,
we will reland it after figuring out how to integrate properly.
llvm-svn: 345963
This patch should not introduce any behavior changes. It consists of
mostly one of two changes:
1. Replacing fall through comments with the LLVM_FALLTHROUGH macro
2. Inserting 'break' before falling through into a case block consisting
of only 'break'.
We were already using this warning with GCC, but its warning behaves
slightly differently. In this patch, the following differences are
relevant:
1. GCC recognizes comments that say "fall through" as annotations, clang
doesn't
2. GCC doesn't warn on "case N: foo(); default: break;", clang does
3. GCC doesn't warn when the case contains a switch, but falls through
the outer case.
I will enable the warning separately in a follow-up patch so that it can
be cleanly reverted if necessary.
Reviewers: alexfh, rsmith, lattner, rtrieu, EricWF, bollu
Differential Revision: https://reviews.llvm.org/D53950
llvm-svn: 345882
-fsyntax-only.
The driver accidentally stopped passing the input filenames on to -cc1
in this mode due to confusion over what action was being requested.
This change also fixes a couple of crashes I encountered when passing
multiple files to such a -cc1 invocation.
llvm-svn: 345803
We haven't supported compiling ObjC1 for a long time (and never will again), so
there isn't any reason to keep these separate. This patch replaces
LangOpts::ObjC1 and LangOpts::ObjC2 with LangOpts::ObjC.
Differential revision: https://reviews.llvm.org/D53547
llvm-svn: 345637
This patch moves the virtual file system form clang to llvm so it can be
used by more projects.
Concretely the patch:
- Moves VirtualFileSystem.{h|cpp} from clang/Basic to llvm/Support.
- Moves the corresponding unit test from clang to llvm.
- Moves the vfs namespace from clang::vfs to llvm::vfs.
- Formats the lines affected by this change, mostly this is the result of
the added llvm namespace.
RFC on the mailing list:
http://lists.llvm.org/pipermail/llvm-dev/2018-October/126657.html
Differential revision: https://reviews.llvm.org/D52783
llvm-svn: 344140
Summary:
Most callers I can find are using only `getName()`. Type is used by the
recursive iterator.
Now we don't have to call stat() on every listed file (on most platforms).
Exceptions are e.g. Solaris where readdir() doesn't include type information.
On those platforms we'll still stat() - see D51918.
The result is significantly faster (stat() can be slow).
My motivation: this may allow us to improve clang IO on large TUs with long
include search paths. Caching readdir() results may allow us to skip many stat()
and open() operations on nonexistent files.
Reviewers: bkramer
Subscribers: fedor.sergeev, cfe-commits
Differential Revision: https://reviews.llvm.org/D51921
llvm-svn: 342232
LLVM triple normalization is handling "unknown" and empty components
differently; for example given "x86_64-unknown-linux-gnu" and
"x86_64-linux-gnu" which should be equivalent, triple normalization
returns "x86_64-unknown-linux-gnu" and "x86_64--linux-gnu". autoconf's
config.sub returns "x86_64-unknown-linux-gnu" for both
"x86_64-linux-gnu" and "x86_64-unknown-linux-gnu". This changes the
triple normalization to behave the same way, replacing empty triple
components with "unknown".
This addresses PR37129.
Differential Revision: https://reviews.llvm.org/D50219
llvm-svn: 339294
Summary:
Reproducer and errors:
https://bugs.llvm.org/show_bug.cgi?id=37878
lookupModule was falling back to loadSubdirectoryModuleMaps when it couldn't
find ModuleName in (proper) search paths. This was causing iteration over all
files in the search path subdirectories for example "/usr/include/foobar" in
bugzilla case.
Users don't expect Clang to load modulemaps in subdirectories implicitly, and
also the disk access is not cheap.
if (AllowExtraModuleMapSearch) true with ObjC with @import ModuleName.
Reviewers: rsmith, aprantl, bruno
Subscribers: cfe-commits, teemperor, v.g.vassilev
Differential Revision: https://reviews.llvm.org/D48367
llvm-svn: 337430
Summary:
Reproducer and errors:
https://bugs.llvm.org/show_bug.cgi?id=37878
lookupModule was falling back to loadSubdirectoryModuleMaps when it couldn't
find ModuleName in (proper) search paths. This was causing iteration over all
files in the search path subdirectories for example "/usr/include/foobar" in
bugzilla case.
Users don't expect Clang to load modulemaps in subdirectories implicitly, and
also the disk access is not cheap.
if (AllowExtraModuleMapSearch) true with ObjC with @import ModuleName.
Reviewers: rsmith, aprantl, bruno
Subscribers: cfe-commits, teemperor, v.g.vassilev
Differential Revision: https://reviews.llvm.org/D48367
llvm-svn: 336660
Implement support for MS-style PCH through headers.
This enables support for /Yc and /Yu where the through header is either
on the command line or included in the source. It replaces the current
support the requires the header also be specified with /FI.
This change adds a -cc1 option -pch-through-header that is used to either
start or stop compilation during PCH create or use.
When creating a PCH, the compilation ends after compilation of the through
header.
When using a PCH, tokens are skipped until after the through header is seen.
Patch By: mikerice
Differential Revision: https://reviews.llvm.org/D46652
llvm-svn: 336379
Unless the user uses -Wno-module-file-config-mismatch (or -Wno-error=...),
allow the AST reader to produce errors describing the nature of the config
mismatch.
llvm-svn: 333220
This is similar to the LLVM change https://reviews.llvm.org/D46290.
We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.
Patch produced by
for i in $(git grep -l '\@brief'); do perl -pi -e 's/\@brief //g' $i & done
for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done
Differential Revision: https://reviews.llvm.org/D46320
llvm-svn: 331834
This replicates 'cl.exe' behavior and allows for both preprocessor output and
dependency information to be extraced with a single compiler invocation.
This is especially useful for compiler caching with tools like Mozilla's sccache.
See: https://github.com/mozilla/sccache/issues/246
Patch By: fxb
Differential Revision: https://reviews.llvm.org/D46394
llvm-svn: 331533
A @import targeting a top level module from a private module map file
(@import Foo_Private), would fail if there's any submodule declaration
around (module Foo.SomeSub) in the same private module map.
This happens because compileModuleImpl, when building Foo_Private, will
start with the private module map and will not parse the public one,
which leads to unsuccessful parsing of Foo.SomeSub, since top level Foo
was never parsed.
Declaring other submodules in the private module map is not common and
should usually be avoided, but it shouldn't fail to build. Canonicalize
compileModuleImpl to always look at the public module first, so that all
necessary information is available when parsing the private one.
rdar://problem/39822328
llvm-svn: 331322
framework module SomeKitCore {
...
export_as SomeKit
}
Given the module above, while generting autolink information during
codegen, clang should to emit '-framework SomeKitCore' only if SomeKit
was not imported in the relevant TU, otherwise it should use '-framework
SomeKit' instead.
rdar://problem/38269782
llvm-svn: 330152
The current support of the feature produces only 2 lines in report:
-Some general Code Generation Time;
-Total time of Backend Consumer actions.
This patch extends Clang time report with new lines related to Preprocessor, Include Filea Search, Parsing, etc.
Differential Revision: https://reviews.llvm.org/D43578
llvm-svn: 329684
Found via codespell -q 3 -I ../clang-whitelist.txt
Where whitelist consists of:
archtype
cas
classs
checkk
compres
definit
frome
iff
inteval
ith
lod
methode
nd
optin
ot
pres
statics
te
thru
Patch by luzpaz! (This is a subset of D44188 that applies cleanly with a few
files that have dubious fixes reverted.)
Differential revision: https://reviews.llvm.org/D44188
llvm-svn: 329399
This make -ivfsoverlay behave more like other fatal errors (e.g. missing
-include file) by skipping the missing file instead of bailing out of
the whole compilation. This makes it possible for libclang to still
provide some functionallity as well as to correctly produce the fatal
error diagnostic (previously we lost the diagnostic in libclang since
there was no TU to tie it to).
rdar://33385423
llvm-svn: 328337
I'm not sure /why/ this is causing issues for libclang, but it is.
Unbreak the buildbots since it's already consumed an hour of my time.
llvm-svn: 328286
When skipping building the module for a private framework module,
LangOpts.CurrentModule isn't enough for implict modules builds; for
instance, in case a private module is built while building a public one,
LangOpts.CurrentModule doesn't reflect the -fmodule-name being passed
down, but instead the module name which triggered the build.
Store the actual -fmodule-name in LangOpts.ModuleName and actually
check a name was provided during compiler invocation in order to
skip building the private module.
rdar://problem/38434694
llvm-svn: 328053
In case `@import Foo.Private` fails because the submodule doesn't exist,
look for `Foo_Private` (if available) and build/load that module
instead. In that process emit a warning and tell the user about the
assumption.
The intention here is to assist all existing private modules owners
(in ObjC and Swift) to migrate to the new `Foo_Private` syntax.
rdar://problem/36023940
llvm-svn: 321342
Move the logic for determining the `wchar_t` type information into the
driver. Rather than passing the single bit of information of
`-fshort-wchar` indicate to the frontend the desired type of `wchar_t`
through a new `-cc1` option of `-fwchar-type` and indicate the
signedness through `-f{,no-}signed-wchar`. This replicates the current
logic which was spread throughout Basic into the
`RenderCharacterOptions`.
Most of the changes to the tests are to ensure that the frontend uses
the correct type. Add a new test set under `test/Driver/wchar_t.c` to
ensure that we calculate the proper types for the various cases.
llvm-svn: 315126
Summary:
The CompilerInstance should create its default VFS from its CompilerInvocation. Right now the
user has to manually create the VFS before creating the FileManager even though
`-ivfsoverlay file.yaml` was passed via the CompilerInvocation (which is exactly how we worked
around this issue in `FrontendAction.cpp` so far).
This patch uses the invocation's VFS by default and also tests this behavior now from the
point of view of a program that uses the clang API.
Reviewers: benlangmuir, v.g.vassilev
Reviewed By: v.g.vassilev
Subscribers: mgorny, cfe-commits, v.g.vassilev
Differential Revision: https://reviews.llvm.org/D37416
llvm-svn: 313049
This fixes a possible crash on certain kinds of corrupted AST file, but
checking in an AST file corrupted in just the right way will be a maintenance
nightmare because the format changes frequently.
llvm-svn: 312851
Summary:
That is, instead of "1 error generated", we now say "1 error generated
when compiling for sm_35".
This (partially) solves a usability foogtun wherein e.g. users call a
function that's only defined on sm_60 when compiling for sm_35, and they
get an unhelpful error message.
Reviewers: tra
Subscribers: sanjoy, cfe-commits
Differential Revision: https://reviews.llvm.org/D37548
llvm-svn: 312736
Extend the -fmodule-file option to support the [<name>=]<file> value format.
If the name is omitted, then the old semantics is preserved (the module file
is loaded whether needed or not). If the name is specified, then the mapping
is treated as just another prebuilt module search mechanism, similar to
-fprebuilt-module-path, and the module file is only loaded if actually used
(e.g., via import). With one exception: this mapping also overrides module
file references embedded in other modules (which can be useful if module files
are moved/renamed as often happens during remote compilation).
This override semantics requires some extra work: we now store the module name
in addition to the file name in the serialized AST representation.
Reviewed By: rsmith
Differential Revision: https://reviews.llvm.org/D35020
llvm-svn: 312220
Extend the -fmodule-file option to support the [<name>=]<file> value format.
If the name is omitted, then the old semantics is preserved (the module file
is loaded whether needed or not). If the name is specified, then the mapping
is treated as just another prebuilt module search mechanism, similar to
-fprebuilt-module-path, and the module file is only loaded if actually used
(e.g., via import). With one exception: this mapping also overrides module
file references embedded in other modules (which can be useful if module files
are moved/renamed as often happens during remote compilation).
This override semantics requires some extra work: we now store the module name
in addition to the file name in the serialized AST representation.
Reviewed By: rsmith
Differential Revision: https://reviews.llvm.org/D35020
llvm-svn: 312105
If a TS module name has more than one component (e.g., foo.bar) then we
erroneously activated the submodule semantics when encountering a module
declaration in the module implementation unit (e.g., 'module foo.bar;').
Reviewed By: rsmith
Differential Revision: https://reviews.llvm.org/D35678
llvm-svn: 312007
This helps some tools that do things based on the output's extension.
For example, we got reports from users on Windows that have a tool that scan a
build output dir (but skip .obj files). The tool would keep the "foo.obj-12345"
file open, and then when clang tried to rename the temp file to the final
output filename, that would fail. By making the tempfile end in ".obj.tmp",
tools like this could now have a rule to ignore .tmp files.
This is a less ambitious reland of https://reviews.llvm.org/D36238https://reviews.llvm.org/D36413
llvm-svn: 310376
This helps some tools that do things based on the output's extension.
For example, we got reports from users on Windows that have a tool that scan a
build output dir (but skip .obj files). The tool would keep the "foo.obj-12345"
file open, and then when clang tried to rename the temp file to the final
output filename, that would fail. By making the tempfile end in ".obj", tools
like this will now skip the temp files as well.
https://reviews.llvm.org/D36238
llvm-svn: 309984
We use this when running a preprocessor-only action on an AST file in order to
avoid paying the runtime cost of loading the extra information.
llvm-svn: 306760
Summary: Device offloading requires the specification of an additional flag containing the triple of the //other// architecture the code is being compiled on if such an architecture exists. If compiling for the host, the auxiliary triple flag will contain the triple describing the device and vice versa.
Reviewers: arpith-jacob, sfantao, caomhin, carlo.bertolli, ABataev, Hahnfeld, jlebar, hfinkel, tstellar
Reviewed By: Hahnfeld
Subscribers: rengolin, cfe-commits
Differential Revision: https://reviews.llvm.org/D29339
llvm-svn: 306689
Duncan P. N. Exon Smith