Requires making the llvm::MemoryBuffer* stored by SourceManager const,
which in turn requires making the accessors for that return const
llvm::MemoryBuffer*s and updating all call sites.
The original motivation for this was to use it and fix the TODO in
CodeGenAction.cpp's ConvertBackendLocation() by using the UnownedTag
version of createFileID, and since llvm::SourceMgr* hands out a const
llvm::MemoryBuffer* this is required. I'm not sure if fixing the TODO
this way actually works, but this seems like a good change on its own
anyways.
No intended behavior change.
Differential Revision: https://reviews.llvm.org/D60247
llvm-svn: 357724
This change adds hierarchical "time trace" profiling blocks that can be visualized in Chrome, in a "flame chart" style. Each profiling block can have a "detail" string that for example indicates the file being processed, template name being instantiated, function being optimized etc.
This is taken from GitHub PR: https://github.com/aras-p/llvm-project-20170507/pull/2
Patch by Aras Pranckevičius.
Differential Revision: https://reviews.llvm.org/D58675
llvm-svn: 357340
FileManager constructs a VFS in its constructor if it isn't passed one,
and there's no way to reset it. Make that contract clear by returning a
reference from its accessor.
https://reviews.llvm.org/D59388
llvm-svn: 357038
Remove CompilerInstance::VirtualFileSystem and
CompilerInstance::setVirtualFileSystem, instead relying on the VFS in
the FileManager. CompilerInstance and its clients already went to some
trouble to make these match. Now they are guaranteed to match.
As part of this, I added a VFS parameter (defaults to nullptr) to
CompilerInstance::createFileManager, to avoid repeating construction
logic in clients that just wanted to customize the VFS.
https://reviews.llvm.org/D59377
llvm-svn: 357037
__pragma(execution_character_set(push, "UTF-8")) is used in
TraceLoggingProvider.h. This commit implements a no-op handler for
compatability, similar to how the flag -fexec_charset is handled.
Patch by Matt Gardner!
Differential Revision: https://reviews.llvm.org/D58530
llvm-svn: 356185
Currently we have -Rpass for filtering the remarks that are displayed as
diagnostics, but when using -fsave-optimization-record, there is no way
to filter the remarks while generating them.
This adds support for filtering remarks by passes using a regex.
Ex: `clang -fsave-optimization-record -foptimization-record-passes=inline`
will only emit the remarks coming from the pass `inline`.
This adds:
* `-fsave-optimization-record` to the driver
* `-opt-record-passes` to cc1
* `-lto-pass-remarks-filter` to the LTOCodeGenerator
* `--opt-remarks-passes` to lld
* `-pass-remarks-filter` to llc, opt, llvm-lto, llvm-lto2
* `-opt-remarks-passes` to gold-plugin
Differential Revision: https://reviews.llvm.org/D59268
Original llvm-svn: 355964
llvm-svn: 355984
Currently we have -Rpass for filtering the remarks that are displayed as
diagnostics, but when using -fsave-optimization-record, there is no way
to filter the remarks while generating them.
This adds support for filtering remarks by passes using a regex.
Ex: `clang -fsave-optimization-record -foptimization-record-passes=inline`
will only emit the remarks coming from the pass `inline`.
This adds:
* `-fsave-optimization-record` to the driver
* `-opt-record-passes` to cc1
* `-lto-pass-remarks-filter` to the LTOCodeGenerator
* `--opt-remarks-passes` to lld
* `-pass-remarks-filter` to llc, opt, llvm-lto, llvm-lto2
* `-opt-remarks-passes` to gold-plugin
Differential Revision: https://reviews.llvm.org/D59268
llvm-svn: 355964
Add an option to cache the generated PCH in the ModuleCache when
emitting it. This protects clients that build PCHs and read them in the
same process, allowing them to avoid race conditions between parallel
jobs the same way that Clang's implicit module build system does.
rdar://problem/48740787
llvm-svn: 355950
Leverage the InMemoryModuleCache to invalidate a module the first time
it fails to import (and to lock a module as soon as it's built or
imported successfully). For implicit module builds, this optimizes
importing deep graphs where the leaf module is out-of-date; see example
near the end of the commit message.
Previously the cache finalized ("locked in") all modules imported so far
when starting a new module build. This was sufficient to prevent
loading two versions of the same module, but was somewhat arbitrary and
hard to reason about.
Now the cache explicitly tracks module state, where each module must be
one of:
- Unknown: module not in the cache (yet).
- Tentative: module in the cache, but not yet fully imported.
- ToBuild: module found on disk could not be imported; need to build.
- Final: module in the cache has been successfully built or imported.
Preventing repeated failed imports avoids variation in builds based on
shifting filesystem state. Now it's guaranteed that a module is loaded
from disk exactly once. It now seems safe to remove
FileManager::invalidateCache, but I'm leaving that for a later commit.
The new, precise logic uncovered a pre-existing problem in the cache:
the map key is the module filename, and different contexts use different
filenames for the same PCM file. (In particular, the test
Modules/relative-import-path.c does not build without this commit.
r223577 started using a relative path to describe a module's base
directory when importing it within another module. As a result, the
module cache sees an absolute path when (a) building the module or
importing it at the top-level, and a relative path when (b) importing
the module underneath another one.)
The "obvious" fix is to resolve paths using FileManager::getVirtualFile
and change the map key for the cache to a FileEntry, but some contexts
(particularly related to ASTUnit) have a shorter lifetime for their
FileManager than the InMemoryModuleCache. This is worth pursuing
further in a later commit; perhaps by tying together the FileManager and
InMemoryModuleCache lifetime, or moving the in-memory PCM storage into a
VFS layer.
For now, use the PCM's base directory as-written for constructing the
filename to check the ModuleCache.
Example
=======
To understand the build optimization, first consider the build of a
module graph TU -> A -> B -> C -> D with an empty cache:
TU builds A'
A' builds B'
B' builds C'
C' builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
If we build TU again, where A, B, C, and D are in the cache and D is
out-of-date, we would previously get this build:
TU imports A
imports B
imports C
imports D (out-of-date)
TU builds A'
A' imports B
imports C
imports D (out-of-date)
builds B'
B' imports C
imports D (out-of-date)
builds C'
C' imports D (out-of-date)
builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
After this commit, we'll immediateley invalidate A, B, C, and D when we
first observe that D is out-of-date, giving this build:
TU imports A
imports B
imports C
imports D (out-of-date)
TU builds A' // The same graph as an empty cache.
A' builds B'
B' builds C'
C' builds D'
imports D'
B' imports C'
imports D'
A' imports B'
imports C'
imports D'
TU imports A'
imports B'
imports C'
imports D'
The new build matches what we'd naively expect, pretty closely matching
the original build with the empty cache.
rdar://problem/48545366
llvm-svn: 355778
Change MemoryBufferCache to InMemoryModuleCache, moving it from Basic to
Serialization. Another patch will start using it to manage module build
more explicitly, but this is split out because it's mostly mechanical.
Because of the move to Serialization we can no longer abuse the
Preprocessor to forward it to the ASTReader. Besides the rename and
file move, that means Preprocessor::Preprocessor has one fewer parameter
and ASTReader::ASTReader has one more.
llvm-svn: 355777
Summary:
FileData was only ever used as a container for the values in
llvm::vfs::Status, so they might as well be consolidated.
The `InPCH` member was also always set to false, and unused.
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D58924
llvm-svn: 355368
Part 1 of CSPGO change in Clang. This includes changes in clang options
and calls to llvm PassManager. Tests will be committed in part2.
This change needs the PassManager change in llvm.
Differential Revision: https://reviews.llvm.org/D54176
llvm-svn: 355331
enum SanitizerOrdinal has reached maximum capacity, this change extends the capacity to 128 sanitizer checks.
This can eventually allow us to add gcc 8's options "-fsanitize=pointer-substract" and "-fsanitize=pointer-compare".
This is a recommit of r354873 but with a fix for unqualified lookup error in lldb cmake build bot.
Fixes: https://llvm.org/PR39425
Differential Revision: https://reviews.llvm.org/D57914
llvm-svn: 355190
enum SanitizerOrdinal has reached maximum capacity, this change extends the capacity to 128 sanitizer checks.
This can eventually allow us to add gcc 8's options "-fsanitize=pointer-substract" and "-fsanitize=pointer-compare".
Fixes: https://llvm.org/PR39425
Differential Revision: https://reviews.llvm.org/D57914
llvm-svn: 354873
We unconditionally predefine these macros. However, they may be used to
determine if the type is supported. In that case, there are unnecessary
failures to compile the code.
This is the proposed fix for https://bugs.llvm.org/show_bug.cgi?id=40559
Differential revision: https://reviews.llvm.org/D57577
llvm-svn: 354512
A faster way to reduce the values in teams reductions was found, the
codegen is updated to use this faster algorithm and new runtime functions.
llvm-svn: 354479
This adds ACLE-defined macros to test for code being compiled in the ROPI and
RWPI position-independence modes.
Differential revision: https://reviews.llvm.org/D23610
llvm-svn: 354265
Fixed diagnostic emission for the exceptions support in case of the
compilation of OpenMP code for the devices. From now on, it uses delayed
diagnostics mechanism, previously used for CUDA only. It allow to
diagnose not allowed used of exceptions only in functions that are going
to be codegen'ed.
llvm-svn: 353542
It is important to delay the emission of the diagnostic messages for the
functions unless it is proved that the function is going to be used on
the device side. It is required to support compilation with some of the
target-specific system headers.
llvm-svn: 353540
Valid OpenCL C code should still compile in C++ mode.
This change enables extensions and OpenCL types.
Differential Revision: https://reviews.llvm.org/D57824
llvm-svn: 353431
When a framework with the same name is available at multiple framework
search paths, we use the first matching location. If a framework at this
location doesn't have all the headers, it can be confusing for
developers because they see only an error `'Foo/Foo.h' file not found`,
can find the complete framework with required header, and don't know the
incomplete framework was used instead.
Add a note explaining a framework without required header was found.
Also mention framework directory path to make it easier to find the
incomplete framework.
rdar://problem/39246514
Reviewers: arphaman, erik.pilkington, jkorous
Reviewed By: jkorous
Subscribers: jkorous, dexonsmith, cfe-commits
Differential Revision: https://reviews.llvm.org/D56561
llvm-svn: 353231
Summary:
This adds support for new-PM plugin loading to clang. The option
`-fpass-plugin=` may be used to specify a dynamic shared object file
that adheres to the PassPlugin API.
Tested: created simple plugin that registers an EP callback; with optimization level > 0, the pass is run as expected.
Committed on behalf of Marco Elver
Differential Revision: https://reviews.llvm.org/D56935
llvm-svn: 352972
- fixes the test on macOS with LLVM_ENABLE_PIC=OFF
- together with D57343, gets the test to pass on Windows
- makes it run everywhere (it seems to just pass on Linux)
The main change is to pull out the resource directory computation into a
function shared by all 3 places that do it. In CIndexer.cpp, this now works no
matter if libclang is in lib/ or bin/ or statically linked to a binary in bin/.
Differential Revision: https://reviews.llvm.org/D57345
llvm-svn: 352803
In 64 bit MSVC environment size_t is defined as unsigned long long.
In single source language like HIP, data layout should be consistent
in device and host compilation, therefore copy data layout controlling
fields from Aux target for AMDGPU target.
Differential Revision: https://reviews.llvm.org/D56318
llvm-svn: 352620
Use the real_path implementation from llvm::sys::fs::real_path instead
of having a custom implementation in the ModuleDependencyCollector.
Differential revision: https://reviews.llvm.org/D57411
llvm-svn: 352605
Introduce an option to request global visibility settings be applied to
declarations without a definition or an explicit visibility, rather than
the existing behavior of giving these default visibility. When the
visibility of all or most extern definitions are known this allows for
the same optimisations -fvisibility permits without updating source code
to annotate all declarations.
Differential Revision: https://reviews.llvm.org/D56868
llvm-svn: 352391
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
Summary:
Adds a new -f[no]split-lto-unit flag that is disabled by default to
control module splitting during ThinLTO. It is automatically enabled
for -fsanitize=cfi and -fwhole-program-vtables.
The new EnableSplitLTOUnit codegen flag is passed down to llvm
via a new module flag of the same name.
Depends on D53890.
Reviewers: pcc
Subscribers: ormris, mehdi_amini, inglorion, eraman, steven_wu, dexonsmith, cfe-commits, llvm-commits
Differential Revision: https://reviews.llvm.org/D53891
llvm-svn: 350949
Summary:
P1353R0, adopted in San Diego, specified an implementation feature test macro for destroying delete (P0722R3).
The implementation of the feature (https://reviews.llvm.org/rL315662) is not guarded behind a flag, so the macro is not conditional on language version.
Reviewers: rsmith
Reviewed By: rsmith
Subscribers: cfe-commits
Differential Revision: https://reviews.llvm.org/D55741
llvm-svn: 350934
Summary: Introduce a compiler flag for cases when the user knows that the collapsed loop counter can be safely represented using at most 32 bits. This will prevent the emission of expensive mathematical operations (such as the div operation) on the iteration variable using 64 bits where 32 bit operations are sufficient.
Reviewers: ABataev, caomhin
Reviewed By: ABataev
Subscribers: hfinkel, kkwli0, guansong, cfe-commits
Differential Revision: https://reviews.llvm.org/D55928
llvm-svn: 350758
Moves the code added in r350340 around a bit, to hopefully make the existing
plugin tests pass when clang is built with examples enabled.
llvm-svn: 350451
The problem is similar to D55986 but for threads: a process with the
interceptor hwasan library loaded might have some threads started by
instrumented libraries and some by uninstrumented libraries, and we
need to be able to run instrumented code on the latter.
The solution is to perform per-thread initialization lazily. If a
function needs to access shadow memory or add itself to the per-thread
ring buffer its prologue checks to see whether the value in the
sanitizer TLS slot is null, and if so it calls __hwasan_thread_enter
and reloads from the TLS slot. The runtime does the same thing if it
needs to access this data structure.
This change means that the code generator needs to know whether we
are targeting the interceptor runtime, since we don't want to pay
the cost of lazy initialization when targeting a platform with native
hwasan support. A flag -fsanitize-hwaddress-abi={interceptor,platform}
has been introduced for selecting the runtime ABI to target. The
default ABI is set to interceptor since it's assumed that it will
be more common that users will be compiling application code than
platform code.
Because we can no longer assume that the TLS slot is initialized,
the pthread_create interceptor is no longer necessary, so it has
been removed.
Ideally, lazy initialization should only cost one instruction in the
hot path, but at present the call may cause us to spill arguments
to the stack, which means more instructions in the hot path (or
theoretically in the cold path if the spills are moved with shrink
wrapping). With an appropriately chosen calling convention for
the per-thread initialization function (TODO) the hot path should
always need just one instruction and the cold path should need two
instructions with no spilling required.
Differential Revision: https://reviews.llvm.org/D56038
llvm-svn: 350429
-plugin already prints an error if the name of an unknown plugin is passed.
-add-plugin used to silently ignore that, now it errors too.
Differential Revision: https://reviews.llvm.org/D56273
llvm-svn: 350340
Since CallExpr::setNumArgs has been removed, it is now possible to store the
callee expression and the argument expressions of CallExpr in a trailing array.
This saves one pointer per CallExpr, CXXOperatorCallExpr, CXXMemberCallExpr,
CUDAKernelCallExpr and UserDefinedLiteral.
Given that CallExpr is used as a base of the above classes we cannot use
llvm::TrailingObjects. Instead we store the offset in bytes from the this pointer
to the start of the trailing objects and manually do the casts + arithmetic.
Some notes:
1.) I did not try to fit the number of arguments in the bit-fields of Stmt.
This leaves some space for future additions and avoid the discussion about
whether x bits are sufficient to hold the number of arguments.
2.) It would be perfectly possible to recompute the offset to the trailing
objects before accessing the trailing objects. However the trailing objects
are frequently accessed and benchmarks show that it is slightly faster to
just load the offset from the bit-fields. Additionally, because of 1),
we have plenty of space in the bit-fields of Stmt.
Differential Revision: https://reviews.llvm.org/D55771
Reviewed By: rjmccall
llvm-svn: 349910
All of the other constructors already take a reference to the AST context.
This avoids calling Decl::getASTContext in most cases. Additionally move
the definition of the constructor from Expr.h to Expr.cpp since it is calling
DeclRefExpr::computeDependence. NFC.
llvm-svn: 349901
Summary:
Add an option to initialize automatic variables with either a pattern or with
zeroes. The default is still that automatic variables are uninitialized. Also
add attributes to request uninitialized on a per-variable basis, mainly to disable
initialization of large stack arrays when deemed too expensive.
This isn't meant to change the semantics of C and C++. Rather, it's meant to be
a last-resort when programmers inadvertently have some undefined behavior in
their code. This patch aims to make undefined behavior hurt less, which
security-minded people will be very happy about. Notably, this means that
there's no inadvertent information leak when:
- The compiler re-uses stack slots, and a value is used uninitialized.
- The compiler re-uses a register, and a value is used uninitialized.
- Stack structs / arrays / unions with padding are copied.
This patch only addresses stack and register information leaks. There's many
more infoleaks that we could address, and much more undefined behavior that
could be tamed. Let's keep this patch focused, and I'm happy to address related
issues elsewhere.
To keep the patch simple, only some `undef` is removed for now, see
`replaceUndef`. The padding-related infoleaks are therefore not all gone yet.
This will be addressed in a follow-up, mainly because addressing padding-related
leaks should be a stand-alone option which is implied by variable
initialization.
There are three options when it comes to automatic variable initialization:
0. Uninitialized
This is C and C++'s default. It's not changing. Depending on code
generation, a programmer who runs into undefined behavior by using an
uninialized automatic variable may observe any previous value (including
program secrets), or any value which the compiler saw fit to materialize on
the stack or in a register (this could be to synthesize an immediate, to
refer to code or data locations, to generate cookies, etc).
1. Pattern initialization
This is the recommended initialization approach. Pattern initialization's
goal is to initialize automatic variables with values which will likely
transform logic bugs into crashes down the line, are easily recognizable in
a crash dump, without being values which programmers can rely on for useful
program semantics. At the same time, pattern initialization tries to
generate code which will optimize well. You'll find the following details in
`patternFor`:
- Integers are initialized with repeated 0xAA bytes (infinite scream).
- Vectors of integers are also initialized with infinite scream.
- Pointers are initialized with infinite scream on 64-bit platforms because
it's an unmappable pointer value on architectures I'm aware of. Pointers
are initialize to 0x000000AA (small scream) on 32-bit platforms because
32-bit platforms don't consistently offer unmappable pages. When they do
it's usually the zero page. As people try this out, I expect that we'll
want to allow different platforms to customize this, let's do so later.
- Vectors of pointers are initialized the same way pointers are.
- Floating point values and vectors are initialized with a negative quiet
NaN with repeated 0xFF payload (e.g. 0xffffffff and 0xffffffffffffffff).
NaNs are nice (here, anways) because they propagate on arithmetic, making
it more likely that entire computations become NaN when a single
uninitialized value sneaks in.
- Arrays are initialized to their homogeneous elements' initialization
value, repeated. Stack-based Variable-Length Arrays (VLAs) are
runtime-initialized to the allocated size (no effort is made for negative
size, but zero-sized VLAs are untouched even if technically undefined).
- Structs are initialized to their heterogeneous element's initialization
values. Zero-size structs are initialized as 0xAA since they're allocated
a single byte.
- Unions are initialized using the initialization for the largest member of
the union.
Expect the values used for pattern initialization to change over time, as we
refine heuristics (both for performance and security). The goal is truly to
avoid injecting semantics into undefined behavior, and we should be
comfortable changing these values when there's a worthwhile point in doing
so.
Why so much infinite scream? Repeated byte patterns tend to be easy to
synthesize on most architectures, and otherwise memset is usually very
efficient. For values which aren't entirely repeated byte patterns, LLVM
will often generate code which does memset + a few stores.
2. Zero initialization
Zero initialize all values. This has the unfortunate side-effect of
providing semantics to otherwise undefined behavior, programs therefore
might start to rely on this behavior, and that's sad. However, some
programmers believe that pattern initialization is too expensive for them,
and data might show that they're right. The only way to make these
programmers wrong is to offer zero-initialization as an option, figure out
where they are right, and optimize the compiler into submission. Until the
compiler provides acceptable performance for all security-minded code, zero
initialization is a useful (if blunt) tool.
I've been asked for a fourth initialization option: user-provided byte value.
This might be useful, and can easily be added later.
Why is an out-of band initialization mecanism desired? We could instead use
-Wuninitialized! Indeed we could, but then we're forcing the programmer to
provide semantics for something which doesn't actually have any (it's
uninitialized!). It's then unclear whether `int derp = 0;` lends meaning to `0`,
or whether it's just there to shut that warning up. It's also way easier to use
a compiler flag than it is to manually and intelligently initialize all values
in a program.
Why not just rely on static analysis? Because it cannot reason about all dynamic
code paths effectively, and it has false positives. It's a great tool, could get
even better, but it's simply incapable of catching all uses of uninitialized
values.
Why not just rely on memory sanitizer? Because it's not universally available,
has a 3x performance cost, and shouldn't be deployed in production. Again, it's
a great tool, it'll find the dynamic uses of uninitialized variables that your
test coverage hits, but it won't find the ones that you encounter in production.
What's the performance like? Not too bad! Previous publications [0] have cited
2.7 to 4.5% averages. We've commmitted a few patches over the last few months to
address specific regressions, both in code size and performance. In all cases,
the optimizations are generally useful, but variable initialization benefits
from them a lot more than regular code does. We've got a handful of other
optimizations in mind, but the code is in good enough shape and has found enough
latent issues that it's a good time to get the change reviewed, checked in, and
have others kick the tires. We'll continue reducing overheads as we try this out
on diverse codebases.
Is it a good idea? Security-minded folks think so, and apparently so does the
Microsoft Visual Studio team [1] who say "Between 2017 and mid 2018, this
feature would have killed 49 MSRC cases that involved uninitialized struct data
leaking across a trust boundary. It would have also mitigated a number of bugs
involving uninitialized struct data being used directly.". They seem to use pure
zero initialization, and claim to have taken the overheads down to within noise.
Don't just trust Microsoft though, here's another relevant person asking for
this [2]. It's been proposed for GCC [3] and LLVM [4] before.
What are the caveats? A few!
- Variables declared in unreachable code, and used later, aren't initialized.
This goto, Duff's device, other objectionable uses of switch. This should
instead be a hard-error in any serious codebase.
- Volatile stack variables are still weird. That's pre-existing, it's really
the language's fault and this patch keeps it weird. We should deprecate
volatile [5].
- As noted above, padding isn't fully handled yet.
I don't think these caveats make the patch untenable because they can be
addressed separately.
Should this be on by default? Maybe, in some circumstances. It's a conversation
we can have when we've tried it out sufficiently, and we're confident that we've
eliminated enough of the overheads that most codebases would want to opt-in.
Let's keep our precious undefined behavior until that point in time.
How do I use it:
1. On the command-line:
-ftrivial-auto-var-init=uninitialized (the default)
-ftrivial-auto-var-init=pattern
-ftrivial-auto-var-init=zero -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang
2. Using an attribute:
int dont_initialize_me __attribute((uninitialized));
[0]: https://users.elis.ugent.be/~jsartor/researchDocs/OOPSLA2011Zero-submit.pdf
[1]: https://twitter.com/JosephBialek/status/1062774315098112001
[2]: https://outflux.net/slides/2018/lss/danger.pdf
[3]: https://gcc.gnu.org/ml/gcc-patches/2014-06/msg00615.html
[4]: 776a0955ef
[5]: http://wg21.link/p1152
I've also posted an RFC to cfe-dev: http://lists.llvm.org/pipermail/cfe-dev/2018-November/060172.html
<rdar://problem/39131435>
Reviewers: pcc, kcc, rsmith
Subscribers: JDevlieghere, jkorous, dexonsmith, cfe-commits
Differential Revision: https://reviews.llvm.org/D54604
llvm-svn: 349442
Balazs Keri