Avoid use-after-frees when FrontendAction::BeginSourceFile is called
twice on the same CompilerInstance by sinking
CompilerInstance::KnownModules into ModuleMap. On the way, rename the
map to CachedModuleLoads. I considered (but rejected) merging this with
ModuleMap::Modules, since that only has top-level modules and this map
includes submodules.
This is an alternative to https://reviews.llvm.org/D58497. Thanks to
nemanjai for the detailed analysis of the problem!
Remove one instance of a hardcoded output stream in
CompilerInstance::ExecuteAction. There are still other cases of output
being hard-coded to standard streams in ExecuteCompilerInvocation, but
this patch covers the case when no flags like -version or -help are
passed, namely the "X warnings and Y errors generated." diagnostic.
Differential Revision: https://reviews.llvm.org/D53768
llvm-svn: 375442
Summary:
When files often get touched during builds, the mtime based validation
leads to different problems in implicit modules builds, even when the
content doesn't actually change:
- Modules only: module invalidation due to out of date files. Usually causing rebuild traffic.
- Modules + PCH: build failures because clang cannot rebuild a module if it comes from building a PCH.
- PCH: build failures because clang cannot rebuild a PCH in case one of the input headers has different mtime.
This patch proposes hashing the content of input files (headers and
module maps), which is performed during serialization time. When looking
at input files for validation, clang only computes the hash in case
there's a mtime mismatch.
I've tested a couple of different hash algorithms availble in LLVM in
face of building modules+pch for `#import <Cocoa/Cocoa.h>`:
- `hash_code`: performace diff within the noise, total module cache increased by 0.07%.
- `SHA1`: 5% slowdown. Haven't done real size measurements, but it'd be BLOCK_ID+20 bytes per input file, instead of BLOCK_ID+8 bytes from `hash_code`.
- `MD5`: 3% slowdown. Like above, but BLOCK_ID+16 bytes per input file.
Given the numbers above, the patch uses `hash_code`. The patch also
improves invalidation error msgs to point out which type of problem the
user is facing: "mtime", "size" or "content".
rdar://problem/29320105
Reviewers: dexonsmith, arphaman, rsmith, aprantl
Subscribers: jkorous, cfe-commits, ributzka
Tags: #clang
Differential Revision: https://reviews.llvm.org/D67249
> llvm-svn: 374841
llvm-svn: 374895
Summary:
When files often get touched during builds, the mtime based validation
leads to different problems in implicit modules builds, even when the
content doesn't actually change:
- Modules only: module invalidation due to out of date files. Usually causing rebuild traffic.
- Modules + PCH: build failures because clang cannot rebuild a module if it comes from building a PCH.
- PCH: build failures because clang cannot rebuild a PCH in case one of the input headers has different mtime.
This patch proposes hashing the content of input files (headers and
module maps), which is performed during serialization time. When looking
at input files for validation, clang only computes the hash in case
there's a mtime mismatch.
I've tested a couple of different hash algorithms availble in LLVM in
face of building modules+pch for `#import <Cocoa/Cocoa.h>`:
- `hash_code`: performace diff within the noise, total module cache increased by 0.07%.
- `SHA1`: 5% slowdown. Haven't done real size measurements, but it'd be BLOCK_ID+20 bytes per input file, instead of BLOCK_ID+8 bytes from `hash_code`.
- `MD5`: 3% slowdown. Like above, but BLOCK_ID+16 bytes per input file.
Given the numbers above, the patch uses `hash_code`. The patch also
improves invalidation error msgs to point out which type of problem the
user is facing: "mtime", "size" or "content".
rdar://problem/29320105
Reviewers: dexonsmith, arphaman, rsmith, aprantl
Subscribers: jkorous, cfe-commits, ributzka
Tags: #clang
Differential Revision: https://reviews.llvm.org/D67249
llvm-svn: 374841
`FileManager::getFileRef` is a modern API which we expect to convert to
over time. We should modernize the error handling as well, using
`llvm::Expected` instead of `llvm::ErrorOr`, to help clients that care
about errors to ensure nothing is missed.
However, not all clients care. I've also added another path for those
that don't:
- `FileEntryRef` is now copy- and move-assignable (using a pointer
instead of a reference).
- `FileManager::getOptionalFileRef` returns an `llvm::Optional` instead
of `llvm::Expected`.
- Added an `llvm::expectedToOptional` utility in case this is useful
elsewhere.
https://reviews.llvm.org/D66705
llvm-svn: 369943
Summary:
Clang performs various recursive operations (such as template instantiation),
and may use non-trivial amounts of stack space in each recursive step (for
instance, due to recursive AST walks). While we try to keep the stack space
used by such steps to a minimum and we have explicit limits on the number of
such steps we perform, it's impractical to guarantee that we won't blow out the
stack on deeply recursive template instantiations on complex ASTs, even with
only a moderately high instantiation depth limit.
The user experience in these cases is generally terrible: we crash with
no hint of what went wrong. Under this patch, we attempt to do better:
* Detect when the stack is nearly exhausted, and produce a warning with a
nice template instantiation backtrace, telling the user that we might
run slowly or crash.
* For cases where we're forced to trigger recursive template
instantiation in arbitrarily-deeply-nested contexts, check whether
we're nearly out of stack space and allocate a new stack (by spawning
a new thread) after producing the warning.
Reviewers: rnk, aaron.ballman
Subscribers: mgorny, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66361
llvm-svn: 369940
when the FileManager is reused across invocations
This commit introduces a parallel API to FileManager's getFile: getFileEntryRef, which returns
a reference to the FileEntry, and the name that was used to access the file. In the case of
a VFS with 'use-external-names', the FileEntyRef contains the external name of the file,
not the filename that was used to access it.
The new API is adopted only in the HeaderSearch and Preprocessor for include file lookup, so that the
accessed path can be propagated to SourceManager's FileInfo. SourceManager's FileInfo now can report this accessed path, using
the new getName method. This API is then adopted in the dependency collector, which now correctly reports dependencies when a file
is included both using a symlink and a real path in the case when the FileManager is reused across multiple Preprocessor invocations.
Note that this patch does not fix all dependency collector issues, as the same problem is still present in other cases when dependencies
are obtained using FileSkipped, InclusionDirective, and HasInclude. This will be fixed in follow-up commits.
Differential Revision: https://reviews.llvm.org/D65907
llvm-svn: 369680
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.
Differential revision: https://reviews.llvm.org/D66259
llvm-svn: 368942
This patch is a prerequisite for using LangStandard from Driver in
https://reviews.llvm.org/D64793.
It moves LangStandard* and InputKind::Language to Basic. It is mostly
mechanical, with only a few changes of note:
- enum Language has been changed into enum class Language : uint8_t to
avoid a clash between OpenCL in enum Language and OpenCL in enum
LangFeatures and not to increase the size of class InputKind.
- Now that getLangStandardForName, which is currently unused, also checks
both canonical and alias names, I've introduced a helper getLangKind
which factors out a code pattern already used 3 times.
The patch has been tested on x86_64-pc-solaris2.11, sparcv9-sun-solaris2.11,
and x86_64-pc-linux-gnu.
There's a companion patch for lldb which uses LangStandard.h
(https://reviews.llvm.org/D65717).
While polly includes isl which in turn uses InputKind::C, that part of the
code isn't even built inside the llvm tree. I've posted a patch to allow
for both InputKind::C and Language::C upstream
(https://groups.google.com/forum/#!topic/isl-development/6oEvNWOSQFE).
Differential Revision: https://reviews.llvm.org/D65562
llvm-svn: 367864
Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.
Signed-off-by: Harlan Haskins <harlan@apple.com>
llvm-svn: 367616
The bitstream reader handles errors poorly. This has two effects:
* Bugs in file handling (especially modules) manifest as an "unexpected end of
file" crash
* Users of clang as a library end up aborting because the code unconditionally
calls `report_fatal_error`
The bitstream reader should be more resilient and return Expected / Error as
soon as an error is encountered, not way late like it does now. This patch
starts doing so and adopting the error handling where I think it makes sense.
There's plenty more to do: this patch propagates errors to be minimally useful,
and follow-ups will propagate them further and improve diagnostics.
https://bugs.llvm.org/show_bug.cgi?id=42311
<rdar://problem/33159405>
Differential Revision: https://reviews.llvm.org/D63518
llvm-svn: 364464
Make DependencyFileGenerator a DependencyCollector as it was intended when
DependencyCollector was introduced. The missing PPCallbacks overrides are added to
the DependencyCollector as well.
This change will allow clang-scan-deps to access the produced dependencies without
writing them out to .d files to disk, so that it will be able collate them and
report them to the user.
Differential Revision: https://reviews.llvm.org/D63290
llvm-svn: 363840
in the compiler
The function SetUpDiagnosticLog that was called from createDiagnostics didn't
handle the case where the diagnostics engine didn't own the diagnostics consumer.
This is a potential problem for a clang tool, in particular some of the follow-up
patches for clang-scan-deps will need this fix.
Differential Revision: https://reviews.llvm.org/D63101
llvm-svn: 363009
Summary:
It's never set to true. Its only effect would be to set stdout to binary mode.
Hopefully we have better ways of doing this by now :-)
Reviewers: hokein
Subscribers: jkorous, arphaman, kadircet, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D60871
llvm-svn: 358696
This change adds hierarchical "time trace" profiling blocks that can be visualized in Chrome, in a "flame chart" style. Each profiling block can have a "detail" string that for example indicates the file being processed, template name being instantiated, function being optimized etc.
This is taken from GitHub PR: https://github.com/aras-p/llvm-project-20170507/pull/2
Patch by Aras Pranckevičius.
Differential Revision: https://reviews.llvm.org/D58675
llvm-svn: 357340
FileManager constructs a VFS in its constructor if it isn't passed one,
and there's no way to reset it. Make that contract clear by returning a
reference from its accessor.
https://reviews.llvm.org/D59388
llvm-svn: 357038
Remove CompilerInstance::VirtualFileSystem and
CompilerInstance::setVirtualFileSystem, instead relying on the VFS in
the FileManager. CompilerInstance and its clients already went to some
trouble to make these match. Now they are guaranteed to match.
As part of this, I added a VFS parameter (defaults to nullptr) to
CompilerInstance::createFileManager, to avoid repeating construction
logic in clients that just wanted to customize the VFS.
https://reviews.llvm.org/D59377
llvm-svn: 357037
Leverage the InMemoryModuleCache to invalidate a module the first time
it fails to import (and to lock a module as soon as it's built or
imported successfully). For implicit module builds, this optimizes
importing deep graphs where the leaf module is out-of-date; see example
near the end of the commit message.
Previously the cache finalized ("locked in") all modules imported so far
when starting a new module build. This was sufficient to prevent
loading two versions of the same module, but was somewhat arbitrary and
hard to reason about.
Now the cache explicitly tracks module state, where each module must be
one of:
- Unknown: module not in the cache (yet).
- Tentative: module in the cache, but not yet fully imported.
- ToBuild: module found on disk could not be imported; need to build.
- Final: module in the cache has been successfully built or imported.
Preventing repeated failed imports avoids variation in builds based on
shifting filesystem state. Now it's guaranteed that a module is loaded
from disk exactly once. It now seems safe to remove
FileManager::invalidateCache, but I'm leaving that for a later commit.
The new, precise logic uncovered a pre-existing problem in the cache:
the map key is the module filename, and different contexts use different
filenames for the same PCM file. (In particular, the test
Modules/relative-import-path.c does not build without this commit.
r223577 started using a relative path to describe a module's base
directory when importing it within another module. As a result, the
module cache sees an absolute path when (a) building the module or
importing it at the top-level, and a relative path when (b) importing
the module underneath another one.)
The "obvious" fix is to resolve paths using FileManager::getVirtualFile
and change the map key for the cache to a FileEntry, but some contexts
(particularly related to ASTUnit) have a shorter lifetime for their
FileManager than the InMemoryModuleCache. This is worth pursuing
further in a later commit; perhaps by tying together the FileManager and
InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a
VFS layer.
For now, use the PCM's base directory as-written for constructing the
filename to check the ModuleCache.
Example
=======
To understand the build optimization, first consider the build of a
module graph TU -> A -> B -> C -> D with an empty cache:
TU builds A'
A' builds B'
B' builds C'
C' builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
If we build TU again, where A, B, C, and D are in the cache and D is
out-of-date, we would previously get this build:
TU imports A
imports B
imports C
imports D (out-of-date)
TU builds A'
A' imports B
imports C
imports D (out-of-date)
builds B'
B' imports C
imports D (out-of-date)
builds C'
C' imports D (out-of-date)
builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
After this commit, we'll immediateley invalidate A, B, C, and D when we
first observe that D is out-of-date, giving this build:
TU imports A
imports B
imports C
imports D (out-of-date)
TU builds A' // The same graph as an empty cache.
A' builds B'
B' builds C'
C' builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
The new build matches what we'd naively expect, pretty closely matching
the original build with the empty cache.
rdar://problem/48545366
llvm-svn: 355778
Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module build
more explicitly, but this is split out because it's mostly mechanical.
Because of the move to Serialization we can no longer abuse the
Preprocessor to forward it to the ASTReader. Besides the rename and
file move, that means Preprocessor::Preprocessor has one fewer parameter
and ASTReader::ASTReader has one more.
llvm-svn: 355777
In 64 bit MSVC environment size_t is defined as unsigned long long.
In single source language like HIP, data layout should be consistent
in device and host compilation, therefore copy data layout controlling
fields from Aux target for AMDGPU target.
Differential Revision: https://reviews.llvm.org/D56318
llvm-svn: 352620
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
When debugging a boost build with a modified
version of Clang, I discovered that the PTH implementation
stores TokenKind in 8 bits. However, we currently have 368
TokenKinds.
The result is that the value gets truncated and the wrong token
gets picked up when including PTH files. It seems that this will
go wrong every time someone uses a token that uses the 9th bit.
Upon asking on IRC, it was brought up that this was a highly
experimental features that was considered a failure. I discovered
via googling that BoostBuild (mostly Boost.Math) is the only user of
this
feature, using the CC1 flag directly. I believe that this can be
transferred over to normal PCH with minimal effort:
https://github.com/boostorg/build/issues/367
Based on advice on IRC and research showing that this is a nearly
completely unused feature, this patch removes it entirely.
Note: I considered leaving the build-flags in place and making them
emit an error/warning, however since I've basically identified and
warned the only user, it seemed better to just remove them.
Differential Revision: https://reviews.llvm.org/D54547
Change-Id: If32744275ef1f585357bd6c1c813d96973c4d8d9
llvm-svn: 348266
The current version only emits the below error for a module (attempted to be loaded) from the `prebuilt-module-path`:
```
error: module file blabla.pcm cannot be loaded due to a configuration mismatch with the current compilation [-Wmodule-file-config-mismatch]
```
With this change, if the prebuilt module is used, we allow the proper diagnostic behind the configuration mismatch to be shown.
```
error: POSIX thread support was disabled in PCH file but is currently enabled
error: module file blabla.pcm cannot be loaded due to a configuration mismatch with the current compilation [-Wmodule-file-config-mismatch]
```
(A few lines later an error is emitted anyways, so there is no reason not to complain for configuration mismatches if a config mismatch is found and kills the build.)
Reviewed By: dblaikie
Tags: #clang
Differential Revision: https://reviews.llvm.org/D53334
llvm-svn: 346439
This reverts commit r345963. We have a path forward now.
Original commit message:
The driver accidentally stopped passing the input filenames on to -cc1
in this mode due to confusion over what action was being requested.
This change also fixes a couple of crashes I encountered when passing
multiple files to such a -cc1 invocation.
llvm-svn: 346130
This reverts commit r345803 and r345915 (a follow-up fix to r345803).
Reason: r345803 blocks our internal integrate because of the new
warnings showing up in too many places. The fix is actually correct,
we will reland it after figuring out how to integrate properly.
llvm-svn: 345963
This patch should not introduce any behavior changes. It consists of
mostly one of two changes:
1. Replacing fall through comments with the LLVM_FALLTHROUGH macro
2. Inserting 'break' before falling through into a case block consisting
of only 'break'.
We were already using this warning with GCC, but its warning behaves
slightly differently. In this patch, the following differences are
relevant:
1. GCC recognizes comments that say "fall through" as annotations, clang
doesn't
2. GCC doesn't warn on "case N: foo(); default: break;", clang does
3. GCC doesn't warn when the case contains a switch, but falls through
the outer case.
I will enable the warning separately in a follow-up patch so that it can
be cleanly reverted if necessary.
Reviewers: alexfh, rsmith, lattner, rtrieu, EricWF, bollu
Differential Revision: https://reviews.llvm.org/D53950
llvm-svn: 345882
-fsyntax-only.
The driver accidentally stopped passing the input filenames on to -cc1
in this mode due to confusion over what action was being requested.
This change also fixes a couple of crashes I encountered when passing
multiple files to such a -cc1 invocation.
llvm-svn: 345803
We haven't supported compiling ObjC1 for a long time (and never will again), so
there isn't any reason to keep these separate. This patch replaces
LangOpts::ObjC1 and LangOpts::ObjC2 with LangOpts::ObjC.
Differential revision: https://reviews.llvm.org/D53547
llvm-svn: 345637
This patch moves the virtual file system form clang to llvm so it can be
used by more projects.
Concretely the patch:
- Moves VirtualFileSystem.{h|cpp} from clang/Basic to llvm/Support.
- Moves the corresponding unit test from clang to llvm.
- Moves the vfs namespace from clang::vfs to llvm::vfs.
- Formats the lines affected by this change, mostly this is the result of
the added llvm namespace.
RFC on the mailing list:
http://lists.llvm.org/pipermail/llvm-dev/2018-October/126657.html
Differential revision: https://reviews.llvm.org/D52783
llvm-svn: 344140
Summary:
Most callers I can find are using only `getName()`. Type is used by the
recursive iterator.
Now we don't have to call stat() on every listed file (on most platforms).
Exceptions are e.g. Solaris where readdir() doesn't include type information.
On those platforms we'll still stat() - see D51918.
The result is significantly faster (stat() can be slow).
My motivation: this may allow us to improve clang IO on large TUs with long
include search paths. Caching readdir() results may allow us to skip many stat()
and open() operations on nonexistent files.
Reviewers: bkramer
Subscribers: fedor.sergeev, cfe-commits
Differential Revision: https://reviews.llvm.org/D51921
llvm-svn: 342232
LLVM triple normalization is handling "unknown" and empty components
differently; for example given "x86_64-unknown-linux-gnu" and
"x86_64-linux-gnu" which should be equivalent, triple normalization
returns "x86_64-unknown-linux-gnu" and "x86_64--linux-gnu". autoconf's
config.sub returns "x86_64-unknown-linux-gnu" for both
"x86_64-linux-gnu" and "x86_64-unknown-linux-gnu". This changes the
triple normalization to behave the same way, replacing empty triple
components with "unknown".
This addresses PR37129.
Differential Revision: https://reviews.llvm.org/D50219
llvm-svn: 339294
Summary:
Reproducer and errors:
https://bugs.llvm.org/show_bug.cgi?id=37878
lookupModule was falling back to loadSubdirectoryModuleMaps when it couldn't
find ModuleName in (proper) search paths. This was causing iteration over all
files in the search path subdirectories for example "/usr/include/foobar" in
bugzilla case.
Users don't expect Clang to load modulemaps in subdirectories implicitly, and
also the disk access is not cheap.
if (AllowExtraModuleMapSearch) true with ObjC with @import ModuleName.
Reviewers: rsmith, aprantl, bruno
Subscribers: cfe-commits, teemperor, v.g.vassilev
Differential Revision: https://reviews.llvm.org/D48367
llvm-svn: 337430
Summary:
Reproducer and errors:
https://bugs.llvm.org/show_bug.cgi?id=37878
lookupModule was falling back to loadSubdirectoryModuleMaps when it couldn't
find ModuleName in (proper) search paths. This was causing iteration over all
files in the search path subdirectories for example "/usr/include/foobar" in
bugzilla case.
Users don't expect Clang to load modulemaps in subdirectories implicitly, and
also the disk access is not cheap.
if (AllowExtraModuleMapSearch) true with ObjC with @import ModuleName.
Reviewers: rsmith, aprantl, bruno
Subscribers: cfe-commits, teemperor, v.g.vassilev
Differential Revision: https://reviews.llvm.org/D48367
llvm-svn: 336660
Implement support for MS-style PCH through headers.
This enables support for /Yc and /Yu where the through header is either
on the command line or included in the source. It replaces the current
support the requires the header also be specified with /FI.
This change adds a -cc1 option -pch-through-header that is used to either
start or stop compilation during PCH create or use.
When creating a PCH, the compilation ends after compilation of the through
header.
When using a PCH, tokens are skipped until after the through header is seen.
Patch By: mikerice
Differential Revision: https://reviews.llvm.org/D46652
llvm-svn: 336379
Unless the user uses -Wno-module-file-config-mismatch (or -Wno-error=...),
allow the AST reader to produce errors describing the nature of the config
mismatch.
llvm-svn: 333220
This is similar to the LLVM change https://reviews.llvm.org/D46290.
We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.
Patch produced by
for i in $(git grep -l '\@brief'); do perl -pi -e 's/\@brief //g' $i & done
for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done
Differential Revision: https://reviews.llvm.org/D46320
llvm-svn: 331834
This replicates 'cl.exe' behavior and allows for both preprocessor output and
dependency information to be extraced with a single compiler invocation.
This is especially useful for compiler caching with tools like Mozilla's sccache.
See: https://github.com/mozilla/sccache/issues/246
Patch By: fxb
Differential Revision: https://reviews.llvm.org/D46394
llvm-svn: 331533
A @import targeting a top level module from a private module map file
(@import Foo_Private), would fail if there's any submodule declaration
around (module Foo.SomeSub) in the same private module map.
This happens because compileModuleImpl, when building Foo_Private, will
start with the private module map and will not parse the public one,
which leads to unsuccessful parsing of Foo.SomeSub, since top level Foo
was never parsed.
Declaring other submodules in the private module map is not common and
should usually be avoided, but it shouldn't fail to build. Canonicalize
compileModuleImpl to always look at the public module first, so that all
necessary information is available when parsing the private one.
rdar://problem/39822328
llvm-svn: 331322
framework module SomeKitCore {
...
export_as SomeKit
}
Given the module above, while generting autolink information during
codegen, clang should to emit '-framework SomeKitCore' only if SomeKit
was not imported in the relevant TU, otherwise it should use '-framework
SomeKit' instead.
rdar://problem/38269782
llvm-svn: 330152
Duncan P. N. Exon Smith