A missing "break" in the initial implementation had us adding a
spurious "/usr/include" to the header search list. Later someone
introduced LLVM_FALLTHROUGH to prevent a warning. Replace this with
the correct "break" and make sure the extra directory isn't added to
the PS4 header search list.
Previously, if a `#pragma clang assume_nonnull begin` was at the
end of a premable with a `#pragma clang assume_nonnull end` at the
end of the main file, clang would diagnose an unterminated begin in
the preamble and an unbalanced end in the main file.
With this change, those errors no longer occur and the case above is
now properly handled. I've added a corresponding test to clangd,
which makes use of preambles, in order to verify this works as
expected.
Differential Revision: https://reviews.llvm.org/D122179
in filesystems
It is simpler to search for module unit by -fprebuilt-module-path
option. However, the separator ':' of partitions is not friendly.
According to the discussion in https://reviews.llvm.org/D118586, I think
we get consensus to use '-' as the separator instead. The '-' is the
choice of GCC too.
Previously I thought it would be better to add an option. But I feel it
is over-engineering now. Another reason here is that there are too many
options for modules (for clang module mainly) now. Given it is not bad
to use '-' when searching, I think it is acceptable to not add an
option.
Reviewed By: iains
Differential Revision: https://reviews.llvm.org/D120874
This is the first in a series of patches that introduce C++20 importable
header units.
These differ from clang header modules in that:
(a) they are identifiable by an internal name
(b) they represent the top level source for a single header - although
that might include or import other headers.
We name importable header units with the path by which they are specified
(although that need not be the absolute path for the file).
So "foo/bar.h" would have a name "foo/bar.h". Header units are made a
separate module type so that we can deal with diagnosing places where they
are permitted but a named module is not.
Differential Revision: https://reviews.llvm.org/D121095
`HeaderSearch` currently assumes `LookupFileCache` is eventually populated in `LookupFile`. However, that's not always the case with `-fms-compatibility` and its early returns.
This patch adds a defensive check that the iterator pulled out of the cache is actually valid before using it.
(This bug was introduced in D119721. Before that, the cache was initialized to `0` - essentially the `search_dir_begin()` iterator.)
Reviewed By: dexonsmith, erichkeane
Differential Revision: https://reviews.llvm.org/D122237
FLT_EVAL_METHOD tells the user the precision at which, temporary results
are evaluated but when fast-math is enabled, the numeric values are not
guaranteed to match the source semantics, so the eval-method is
meaningless.
For example, the expression `x + y + z` has as source semantics `(x + y)
+ z`. FLT_EVAL_METHOD is telling the user at which precision `(x + y)`
is evaluated. With fast-math enable the compiler can choose to
evaluate the expression as `(y + z) + x`.
The correct behavior is to set the FLT_EVAL_METHOD to `-1` to tell the
user that the precision of the intermediate values is unknow. This
patch is doing that.
Differential Revision: https://reviews.llvm.org/D121122
The iterator is not needed after the loop body anymore, meaning we can use more terse range-based for loop.
Depends on D121295.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D121685
To reduce the number of modules we build in explicit builds (which use strict context hash), we prune unused header search paths. This essentially merges parts of the dependency graph.
Determining whether a search path was used to discover a module (through implicit module maps) proved to be somewhat complicated. Initial support landed in D102923, while D113676 attempts to fix some bugs.
However, now that we don't use implicit module maps in explicit builds (since D120465), we don't need to consider such search paths as used anymore. Modules are no longer discovered through the header search mechanism, so we can drop such search paths (provided they are not needed for other reasons).
This patch removes whatever support for detecting such usage we had, since it's buggy and not required anymore.
Depends on D120465.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D121295
WG14 adopted N2775 (http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2775.pdf)
at our Feb 2022 meeting. This paper adds a literal suffix for
bit-precise types that automatically sizes the bit-precise type to be
the smallest possible legal _BitInt type that can represent the literal
value. The suffix chosen is wb (for a signed bit-precise type) which
can be combined with the u suffix (for an unsigned bit-precise type).
The preprocessor continues to operate as-if all integer types were
intmax_t/uintmax_t, including bit-precise integer types. It is a
constraint violation if the bit-precise literal is too large to fit
within that type in the context of the preprocessor (when still using
a pp-number preprocessing token), but it is not a constraint violation
in other circumstances. This allows you to make bit-precise integer
literals that are wider than what the preprocessor currently supports
in order to initialize variables, etc.
Given that there is only one external user of Lexer::getLangOpts
we can remove getter entirely without much pain.
Differential Revision: https://reviews.llvm.org/D120404
clang has support for lazy headers in module maps - if size and/or
modtime and provided in the cppmap file, headers are only resolved when
an include directive for a file with that size/modtime is encoutered.
Before this change, the lazy resolution was all-or-nothing per module.
That means as soon as even one file in that module potentially matched
an include, all lazy files in that module were resolved. With this
change, only files with matching size/modtime will be resolved.
The goal is to avoid unnecessary stat() calls on non-included files,
which is especially valuable on networked file systems, with higher
latency.
Differential Revision: https://reviews.llvm.org/D120569
This change can be seen as code cleanup but motivation is more performance related.
While browsing perf reports captured during Linux build we can notice unusual portion of instructions executed in std::vector<std::string> copy constructor like:
0.59% 0.58% clang-14 clang-14 [.] std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >,
std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::vector
or even:
1.42% 0.26% clang clang-14 [.] clang::LangOptions::LangOptions
|
--1.16%--clang::LangOptions::LangOptions
|
--0.74%--std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >,
std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::vector
After more digging we can see that relevant LangOptions std::vector members (*Files, ModuleFeatures and NoBuiltinFuncs)
are constructed when Lexer::LangOpts field is initialized on list:
Lexer::Lexer(..., const LangOptions &langOpts, ...)
: ..., LangOpts(langOpts),
Since LangOptions copy constructor is called by Lexer(..., const LangOptions &LangOpts,...) and local Lexer objects are created thousands times
(in Lexer::getRawToken, Preprocessor::EnterSourceFile and more) during single module processing in frontend it makes std::vector copy constructors surprisingly hot.
Unfortunately even though in current Lexer implementation mentioned std::vector members are unused and most of time empty,
no compiler is smart enough to optimize their std::vector copy constructors out (take a look at test assembly): https://godbolt.org/z/hdoxPfMYY even with LTO enabled.
However there is simple way to fix this. Since Lexer doesn't access *Files, ModuleFeatures, NoBuiltinFuncs and any other LangOptions fields (but only LangOptionsBase)
we can simply get rid of redundant copy constructor assembly by changing LangOpts type to more appropriate const LangOptions reference: https://godbolt.org/z/fP7de9176
Additionally we need to store LineComment outside LangOpts because it's written in SkipLineComment function.
Also FormatTokenLexer need to be adjusted a bit to avoid lifetime issues related to passing local LangOpts reference to Lexer.
After this change I can see more than 1% speedup in some of my microbenchmarks when using Clang release binary built with LTO.
For Linux build gains are not so significant but still nice at the level of -0.4%/-0.5% instructions drop.
Differential Revision: https://reviews.llvm.org/D120334
Before this change without any good reason Lexer::LangOpts is sometimes accessed by getter and another time read directly in Lexer functions.
Since getLangOpts is a bit more verbose prefer direct access to LangOpts member when possible.
Differential Revision: https://reviews.llvm.org/D120333
This patch replaces a lot of index-based loops with iterators and ranges.
Depends on D117566.
Reviewed By: ahoppen
Differential Revision: https://reviews.llvm.org/D119722
This patch starts using the new iterator type in `LookupFileCacheInfo`.
Depends on D117566.
Reviewed By: ahoppen
Differential Revision: https://reviews.llvm.org/D119721
The `const DirectoryLookup *` out-parameter of `{HeaderSearch,Preprocessor}::LookupFile()` is assigned the most recently used search directory, which callers use to implement `#include_next`.
From the function signature it's not obvious the `const DirectoryLookup *` is being used as an iterator. This patch introduces `ConstSearchDirIterator` to make that affordance obvious. This would've prevented a bug that occurred after initially landing D116750.
Reviewed By: ahoppen
Differential Revision: https://reviews.llvm.org/D117566
This patch addresses a FIXME and de-duplicates some `#include_next` logic
Depends on D119714.
Reviewed By: ahoppen
Differential Revision: https://reviews.llvm.org/D119716
The minimizer strips out single-line comments (introduced by `//`). This sequence of characters can also appear in `#include` or `#import` directives where they play the role of path separators. We already avoid stripping this character sequence for `#include` but not for `#import` (which has the same semantics). This patch makes it so `#import <A//A.h>` is not affected by minimization. Previously, we would incorrectly reduce it into `#import <A`.
Reviewed By: arphaman
Differential Revision: https://reviews.llvm.org/D119226
The minimizer tries to squash multi-line macro definitions into single line. For that to work, contents of each line need to be separated by a space. Since we always strip leading whitespace on lines of a macro definition, the code currently tries to preserve exactly one space that appeared before the backslash.
This means the following code:
```
#define FOO(BAR) \
#BAR \
baz
```
gets minimized into:
```
#define FOO(BAR) #BAR baz
```
However, if there are no spaces before the backslash on line 2:
```
#define FOO(BAR) \
#BAR\
baz
```
no space can be preserved, leading to (most likely) malformed macro definition:
```
#define FOO(BAR) #BARbaz
```
This patch makes sure we always put exactly one space at the end of line ending with a backslash.
Reviewed By: arphaman
Differential Revision: https://reviews.llvm.org/D119231
`Preprocessor` exposes the search directory iterator via `GetCurDirLookup()` getter, which is only used in two static functions.
To simplify reasoning about search directory iterators/references and to simplify the `Preprocessor` API, this patch makes the two static functions private member functions and removes the getter entirely.
Depends D119708.
Reviewed By: ahoppen, dexonsmith
Differential Revision: https://reviews.llvm.org/D119714
The purpose of the `FileNotFound` preprocessor callback was to add the ability to recover from failed header lookups. This was to support downstream project.
However, injecting additional search path while performing header search can invalidate currently used iterators/references to `DirectoryLookup` in `Preprocessor` and `HeaderSearch`.
The downstream project ended up maintaining a separate patch to further tweak the functionality. Since we don't have any upstream users nor open source downstream users, I'd like to remove this callback for good to prevent future misuse. I doubt there are any actual downstream users, since the functionality is definitely broken at the moment.
Reviewed By: ahoppen
Differential Revision: https://reviews.llvm.org/D119708
Recently we observed high memory pressure caused by clang during some parallel builds.
We discovered that we have several projects that have a large number of #define directives
in their TUs (on the order of millions), which caused huge memory consumption in clang due
to a lot of allocations for MacroInfo. We would like to reduce the memory overhead of
clang for a single #define to reduce the memory overhead for these files, to allow us to
reduce the memory pressure on the system during highly parallel builds. This change achieves
that by removing the SmallVector in MacroInfo and instead storing the tokens in an array
allocated using the bump pointer allocator, after all tokens are lexed.
The added unit test with 1000000 #define directives illustrates the problem. Prior to this
change, on arm64 macOS, clang's PP bump pointer allocator allocated 272007616 bytes, and
used roughly 272 bytes per #define. After this change, clang's PP bump pointer allocator
allocates 120002016 bytes, and uses only roughly 120 bytes per #define.
For an example test file that we have internally with 7.8 million #define directives, this
change produces the following improvement on arm64 macOS: Persistent allocation footprint for
this test case file as it's being compiled to LLVM IR went down 22% from 5.28 GB to 4.07 GB
and the total allocations went down 14% from 8.26 GB to 7.05 GB. Furthermore, this change
reduced the total number of allocations made by the system for this clang invocation from
1454853 to 133663, an order of magnitude improvement.
The recommit fixes the LLDB build failure.
Differential Revision: https://reviews.llvm.org/D117348
Recently we observed high memory pressure caused by clang during some parallel builds.
We discovered that we have several projects that have a large number of #define directives
in their TUs (on the order of millions), which caused huge memory consumption in clang due
to a lot of allocations for MacroInfo. We would like to reduce the memory overhead of
clang for a single #define to reduce the memory overhead for these files, to allow us to
reduce the memory pressure on the system during highly parallel builds. This change achieves
that by removing the SmallVector in MacroInfo and instead storing the tokens in an array
allocated using the bump pointer allocator, after all tokens are lexed.
The added unit test with 1000000 #define directives illustrates the problem. Prior to this
change, on arm64 macOS, clang's PP bump pointer allocator allocated 272007616 bytes, and
used roughly 272 bytes per #define. After this change, clang's PP bump pointer allocator
allocates 120002016 bytes, and uses only roughly 120 bytes per #define.
For an example test file that we have internally with 7.8 million #define directives, this
change produces the following improvement on arm64 macOS: Persistent allocation footprint for
this test case file as it's being compiled to LLVM IR went down 22% from 5.28 GB to 4.07 GB
and the total allocations went down 14% from 8.26 GB to 7.05 GB. Furthermore, this change
reduced the total number of allocations made by the system for this clang invocation from
1454853 to 133663, an order of magnitude improvement.
Differential Revision: https://reviews.llvm.org/D117348
An impilt used of C++ module without prebuild path may cause crash.
For example:
```
// ./dir1/C.cppm
export module C;
// ./dir2/B.cppm
export module B;
import C;
// ./A.cpp
import B;
import C;
```
When we compile A.cpp without the prebuild path of C.pcm, the compiler
will crash.
```
clang++ -std=c++20 --precompile -c ./dir1/C.cppm -o ./dir1/C.pcm
clang++ -std=c++20 --precompile -fprebuilt-module-path=./dir2 -c
./dir2/B.cppm -o ./dir2/B.pcm
clang++ -std=c++20 -fprebuilt-module-path=./dir2 A.cpp
```
The prebuilt path of module C is cached when import module B, and in the
function HeaderSearch::getCachedModuleFileName, the compiler try to get
the filename by modulemap without check if modulemap exists, and there
is no modulemap in C++ module.
Reviewed By: ChuanqiXu
Differential review: https://reviews.llvm.org/D119426
Previously, the Framework name was only set if the file
came from a header mapped framework; now we'll always
set the framework name if the file is in a framework.
Differential Revision: https://reviews.llvm.org/D117830
The lexer can attempt to lex a _Pragma and crash with an out of bounds string access when it's
lexing a _Pragma whose string token is an invalid buffer, e.g. when a module header file from which the macro
expansion for that token was deleted from the file system.
Differential Revision: https://reviews.llvm.org/D116052
Normally there are heruistics in lexer to treat `//*` specially in
language modes that don't have line comments (to emit `/`). Unfortunately this
only applied to the first occurence of a line comment inside the file, as the
subsequent line comments were treated as if language had support for them.
This unfortunately only holds in normal lexing mode, as in raw mode all
occurences of line comments received this treatment, which created discrepancies
when comparing expanded and spelled tokens.
The proper fix would be to just make sure we treat all the line comments with a
subsequent `*` the same way, but it would imply breaking some code that's
accepted by clang today. So instead we introduce the same bug into raw lexing
mode.
Fixes https://github.com/clangd/clangd/issues/1003.
Differential Revision: https://reviews.llvm.org/D118471
This patch replaces the exact include count of each file in `HeaderFileInfo` with a set of included files in `Preprocessor`.
The number of includes isn't a property of a header file but rather a preprocessor state. The exact number of includes is not used anywhere except statistic tracking.
Reviewed By: vsapsai
Differential Revision: https://reviews.llvm.org/D114095
The `{HeaderSearch,Preprocessor}::LookupFile()` functions take an out-parameter `const DirectoryLookup *&`. Most callers end up creating a `const DirectoryLookup *` variable that's otherwise unused.
This patch changes the out-parameter from reference to a pointer, making it possible to simply pass `nullptr` to the function without the ceremony.
Reviewed By: ahoppen
Differential Revision: https://reviews.llvm.org/D117312
In D116750, the `clangFrontend` library was added as a dependency of `LexTests` in order to make `clang::ApplyHeaderSearchOptions()` available. This increased the number of TUs the test depends on.
This patch moves the function into `clangLex` and removes dependency of `LexTests` on `clangFrontend`.
Reviewed By: thakis
Differential Revision: https://reviews.llvm.org/D117024
The elements of `SearchPath::SearchDirs` are being referenced to by their indices. This proved to be error-prone: `HeaderSearch::SearchDirToHSEntry` was accidentally not being updated in `HeaderSearch::AddSearchPath()`. This patch fixes that by referencing `SearchPath::SearchDirs` elements by their address instead, which is stable thanks to the bump-ptr-allocation strategy.
Reviewed By: ahoppen
Differential Revision: https://reviews.llvm.org/D116750
Clang will now search through the framework includes to identify
the framework include path to a file, and then suggest a framework
style include spelling for the file.
Differential Revision: https://reviews.llvm.org/D115183
According to [module.unit]p7.2.3, a declaration within a linkage-specification
should be attached to the global module.
This let user to forward declare types across modules.
Reviewed by: rsmith, aaron.ballman
Differential Revision: https://reviews.llvm.org/D110215
This patch refactors the code that checks whether a file has just been included for the first time.
The `HeaderSearch::FirstTimeLexingFile` function is removed and the information is threaded to the original call site from `HeaderSearch::ShouldEnterIncludeFile`. This will make it possible to avoid tracking the number of includes in a follow up patch.
Depends on D114092.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D114093
When using clangd, it's possible to trigger assertions in
NumericLiteralParser and CharLiteralParser when switching git branches.
This commit removes the initial asserts on invalid input and replaces
those asserts with the error handling mechanism from those respective
classes instead. This allows clangd to gracefully recover without
crashing.
See https://github.com/clangd/clangd/issues/888 for more information
on the clangd crashes.
Some time back I extended GCC's '# NNN' line marker semantics.
Specifically popping to a blank filename will restore the filename to
that of the popped-to include. Restore to line 5 of including file
(escaped BOL #'s to avoid git eliding them):
\# 5 "" 2
Added documentation for this line control extension.
This was useful in developing modules tests, but turned out to also be
useful with machine-generated source code. Specifically, a generated
include file that itself includes fragments from elsewhere. The
ability to pop to the generated include file -- with its full path
prefix -- is useful for diagnostic & debug purposes. For instance
something like:
// Machine generated -- DO NOT EDIT
Type Var = {
\# 7 "encoded.dsl" 1 // push to snippet-container
{snippet, of, code}
\# 6 " 2 // Restore to machined-generated source
,
};
// user-code
...
\#include "dsl.h"
...
That pop to "" will restore the filename to '..includepath../dsl.h',
which is better than restoring to plain "dsl.h".
Differential Revision: https://reviews.llvm.org/D113425
Paul Robinson