Summary:
Currently, `buildAST[WithArgs]` either succeeds or fails. This patch adds
support for the caller to pass a `DiagnosticConsumer` to receive all relevant
diagnostics.
Reviewers: gribozavr
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D74763
The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, so that parallel computations such as LLD/ThinLTO can use all hardware threads in the system. Before this patch, on Windows, a maximum of 64 hardware threads could be used at most, in some cases dispatched only on one CPU socket.
== Background ==
Windows doesn't have a flat cpu_set_t like Linux. Instead, it projects hardware CPUs (or NUMA nodes) to applications through a concept of "processor groups". A "processor" is the smallest unit of execution on a CPU, that is, an hyper-thread if SMT is active; a core otherwise. There's a limit of 32-bit processors on older 32-bit versions of Windows, which later was raised to 64-processors with 64-bit versions of Windows. This limit comes from the affinity mask, which historically is represented by the sizeof(void*). Consequently, the concept of "processor groups" was introduced for dealing with systems with more than 64 hyper-threads.
By default, the Windows OS assigns only one "processor group" to each starting application, in a round-robin manner. If the application wants to use more processors, it needs to programmatically enable it, by assigning threads to other "processor groups". This also means that affinity cannot cross "processor group" boundaries; one can only specify a "preferred" group on start-up, but the application is free to allocate more groups if it wants to.
This creates a peculiar situation, where newer CPUs like the AMD EPYC 7702P (64-cores, 128-hyperthreads) are projected by the OS as two (2) "processor groups". This means that by default, an application can only use half of the cores. This situation could only get worse in the years to come, as dies with more cores will appear on the market.
== The problem ==
The heavyweight_hardware_concurrency() API was introduced so that only *one hardware thread per core* was used. Once that API returns, that original intention is lost, only the number of threads is retained. Consider a situation, on Windows, where the system has 2 CPU sockets, 18 cores each, each core having 2 hyper-threads, for a total of 72 hyper-threads. Both heavyweight_hardware_concurrency() and hardware_concurrency() currently return 36, because on Windows they are simply wrappers over std:🧵:hardware_concurrency() -- which can only return processors from the current "processor group".
== The changes in this patch ==
To solve this situation, we capture (and retain) the initial intention until the point of usage, through a new ThreadPoolStrategy class. The number of threads to use is deferred as late as possible, until the moment where the std::threads are created (ThreadPool in the case of ThinLTO).
When using hardware_concurrency(), setting ThreadCount to 0 now means to use all the possible hardware CPU (SMT) threads. Providing a ThreadCount above to the maximum number of threads will have no effect, the maximum will be used instead.
The heavyweight_hardware_concurrency() is similar to hardware_concurrency(), except that only one thread per hardware *core* will be used.
When LLVM_ENABLE_THREADS is OFF, the threading APIs will always return 1, to ensure any caller loops will be exercised at least once.
Differential Revision: https://reviews.llvm.org/D71775
DynTypedNode and ASTNodeKind are implemented as part of the clang AST
library, which uses the main clang namespace. There doesn't seem to be a
need for this extra level of namespacing.
I left behind aliases in the ast_type_traits namespace for out of tree
clients of these APIs. To provide aliases for the enumerators, I used
this pattern:
namespace ast_type_traits {
constexpr TraversalKind TK_AsIs = ::clang::TK_AsIs;
}
I think the typedefs will be useful for migration, but we might be able
to drop these enumerator aliases.
Reviewed By: aaron.ballman
Differential Revision: https://reviews.llvm.org/D74499
Summary:
When renaming a class with template constructors, we are missing the
occurrences of the template constructors, because getUSRsForDeclaration doesn't
give USRs of the templated constructors (they are not in the normal `ctors()`
method).
Reviewers: kbobyrev
Subscribers: jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D74216
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
The only part of ASTContext.h that requires most AST types to be
complete is the parent map. Nothing in Clang proper uses the ParentMap,
so split it out into its own class. Make ASTContext own the
ParentMapContext so there is still a one-to-one relationship.
After this change, 562 fewer files depend on ASTTypeTraits.h, and 66
fewer depend on TypeLoc.h:
$ diff -u deps-before.txt deps-after.txt | \
grep '^[-+] ' | sort | uniq -c | sort -nr | less
562 - ../clang/include/clang/AST/ASTTypeTraits.h
340 + ../clang/include/clang/AST/ParentMapContext.h
66 - ../clang/include/clang/AST/TypeLocNodes.def
66 - ../clang/include/clang/AST/TypeLoc.h
15 - ../clang/include/clang/AST/TemplateBase.h
...
I computed deps-before.txt and deps-after.txt with `ninja -t deps`.
This removes a common and key dependency on TemplateBase.h and
TypeLoc.h.
This also has the effect of breaking the ParentMap RecursiveASTVisitor
instantiation into its own file, which roughly halves the compilation
time of ASTContext.cpp (29.75s -> 17.66s). The new file takes 13.8s to
compile.
I left behind forwarding methods for getParents(), but clients will need
to include a new header to make them work:
#include "clang/AST/ParentMapContext.h"
I noticed that this parent map functionality is unfortunately duplicated
in ParentMap.h, which only works for Stmt nodes.
Reviewed By: rsmith
Differential Revision: https://reviews.llvm.org/D71313
Differential Revision: https://reviews.llvm.org/D70268
This is a recommit of f978ea4983 with a fix for the PowerPC failure.
The issue was that:
* `CompilerInstance::ExecuteAction` calls
`getTarget().adjust(getLangOpts());`.
* `PPCTargetInfo::adjust` changes `LangOptions::HasAltivec`.
* This happens after the first few calls to `getModuleHash`.
There’s even a FIXME saying:
```
// FIXME: We shouldn't need to do this, the target should be immutable once
// created. This complexity should be lifted elsewhere.
```
This only showed up on PowerPC because it's one of the few targets that
almost always changes a hashed langopt.
I looked into addressing the fixme, but that would be a much larger
change, and it's not the only thing that happens in `ExecuteAction` that
can change the module context hash. Instead I changed the code to not
call `getModuleHash` until after it has been modified in `ExecuteAction`.
Summary: Currently, an attempt to rewrite source code inside a macro expansion succeeds, but results in empty text, rather than failing with an error. This patch restructures to the code to explicitly validate ranges before attempting to edit them.
Reviewers: gribozavr
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D72274
Add checks for some structural invariants when building and mutating
the syntax trees.
Fix a bug failing the invariants after mutations: the parent of nodes
added into the tree was null.
When they are free-standing, e.g. `struct X;` or `struct X {};`.
Although this complicates the common case (of free-standing class
declarations), this ensures the less common case (e.g. `struct X {} a;`)
are handled uniformly and produce similar syntax trees.
Summary:
Currently interpolation logic prefers -std over -x. But the latter is a
more strong signal, so this patch inverts the order and only makes use of -std
if -x didn't exist.
Fixes https://github.com/clangd/clangd/issues/185
Thanks @sammccall for tracking this down!
Reviewers: sammccall
Subscribers: ilya-biryukov, usaxena95, cfe-commits, sammccall
Tags: #clang
Differential Revision: https://reviews.llvm.org/D71727
This is both more efficient and avoids corner cases in
`SourceManager::isBeforeInTranslationUnit`.
The change is trivial and clearly a performance improvement on the hot
path of building the syntax tree, so sending without review.
Summary:
This patch adds facilities to mutate the syntax trees and produce
corresponding text replacements.
The public interface of the syntax library now includes facilities to:
1. perform type-safe modifications of syntax trees,
2. compute textual replacements to apply the modifications,
3. create syntax trees not backed by the source code.
For each of the three, we only add a few example transformations in this
patch to illustrate the idea, support for more kinds of nodes and
transformations will be done in follow-up patches.
The high-level mutation operations are implemented on top of operations
that allow to arbitrarily change the trees. They are considered to be
implementation details and are not available to the users of the
library.
Reviewers: sammccall, gribozavr2
Reviewed By: gribozavr2
Subscribers: merge_guards_bot, mgorny, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64573
Summary: Useful when positions are used to target nodes, with before/after ambiguity.
Reviewers: ilya-biryukov, kbobyrev
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D71356
Summary: Useful when positions are used to target nodes, with before/after ambiguity.
Reviewers: ilya-biryukov, kbobyrev
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D71356
Summary:
This is a follow-up to D70769 and D70222, which allows propagation of
current directory down to ExpandResponseFiles for handling of relative paths.
Previously clients had to mutate FS to achieve that, which is not thread-safe
and can even be thread-hostile in the case of real file system.
Reviewers: sammccall
Subscribers: hiraditya, cfe-commits, llvm-commits
Tags: #clang, #llvm
Differential Revision: https://reviews.llvm.org/D70857
Summary:
The exclusive-claim model is successful at resolving conflicts over tokens
between parent/child or siblings. However claims at the spelled-token
level do the wrong thing for macro expansions, where siblings can be
equally associated with the macro invocation.
Moreover, any model that only uses the endpoints in a range can fail when
a macro invocation occurs inside the node.
To address this, we use the existing TokenBuffer in more depth.
Claims are expressed in terms of expanded tokens, so there is no need to worry
about macros, includes etc.
Once we know which expanded tokens were claimed, they are mapped onto
spelled tokens for hit-testing.
This mapping is fairly flexible, currently the handling of macros is
pretty simple (map macro args onto spellings, other macro expansions onto the
macro name token).
This mapping is in principle token-by-token for correctness (though
there's some batching for performance).
The aggregation of the selection enum is now more principled as we need to be
able to aggregate several hit-test results together.
For simplicity i removed the ability to determine selectedness of TUDecl.
(That was originally implemented in 90a5bf92ff97b1, but doesn't seem to be very
important or worth the complexity any longer).
The expandedTokens(SourceLocation) helper could be added locally, but seems to
make sense on TokenBuffer.
Fixes https://github.com/clangd/clangd/issues/202
Fixes https://github.com/clangd/clangd/issues/126
Reviewers: hokein
Subscribers: MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits, ilya-biryukov
Tags: #clang
Differential Revision: https://reviews.llvm.org/D70512
Summary:
Also remove the temporary TopLevelDeclaration node and add
UnknownDeclaration to represent other unknown nodes.
See the follow-up change for building more top-level declarations.
Adding declarators is also pretty involved and will be done in another
follow-up patch.
Reviewers: gribozavr2
Reviewed By: gribozavr2
Subscribers: merge_guards_bot, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D70787
Fix the confusing naming of `CompilerInstance::ModuleManager`. This is
actually an instance of `ASTReader`, which contains an instance of
`ModuleManager`. I have to assume there was a point in the past where
they were just one class, but it's been pretty confusing for a while. I
think it's time to fix it.
The new name is `TheASTReader`; the annoying `The` prefix is so that we
don't shadow the `ASTReader` class. I tried out `ASTRdr` but that
seemed less clear, and this choice matches `ThePCHContainerOperations`
just a couple of declarations below.
Also rename `CompilerInstance::getModuleManager` and
`CompilerInstance::createModuleManager` to `*ASTReader`, making some
cases of `getModuleManager().getModuleManager()` a little more clear.
https://reviews.llvm.org/D70583
Summary:
Adds combinators `maybeDeref` and `maybeAddressOf` to provide a uniform way to handle
nodes which may be bound to either a pointer or a value (most often in the
context of member expressions). Such polymorphism is already supported by
`access`; these combinators extend it to more general uses.
Reviewers: gribozavr
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D70554
Summary:
Adds an optional parameter to `buildASTFromCodeWithArgs` that allows the user to
pass additional files that the main code needs to compile. This change makes
`buildASTFromCodeWithArgs` consistent with `runToolOnCodeWithArgs`.
Patch by Alexey Eremin.
Reviewers: gribozavr
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D70175
Summary:
Flags that generate output could result in failures when creating
syntax only actions. This patch introduces initial logic for filtering out
those. The first such flag is "save-temps", which saves intermediate
files(bitcode, assembly, etc.) into a specified directory.
Fixes https://github.com/clangd/clangd/issues/191
Reviewers: hokein
Subscribers: ilya-biryukov, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D70173
Summary:
By additional regex match, grouping of main include can be enabled in files that are not normally considered as a C/C++ source code.
For example, this might be useful in templated code, where template implementations are being held in *Impl.hpp files.
On the occassion, 'assume-filename' option description was reworded as it was misleading. It has nothing to do with `style=file` option and it does not influence sourced style filename.
Reviewers: rsmith, ioeric, krasimir, sylvestre.ledru, MyDeveloperDay
Reviewed By: MyDeveloperDay
Subscribers: MyDeveloperDay, cfe-commits
Patch by: furdyna
Tags: #clang
Differential Revision: https://reviews.llvm.org/D67750
Summary:
This revision introduces a new interface `MatchComputation` which generalizes
the `Stencil` interface and replaces the `std::function` interface of
`MatchConsumer`. With this revision, `Stencil` (as an abstraction) becomes just
one collection of implementations of
`MatchComputation<std::string>`. Correspondingly, we remove the `Stencil` class
entirely in favor of a simple type alias, deprecate `MatchConsumer` and change
all functions that accepted `MatchConsumer<std::string>` to use
`MatchComputation<std::string>` instead.
Reviewers: gribozavr
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D69802
Summary:
This should be NFC to clang-rename, by default the traversal scope is
TUDecl. Traversing the TUDecl in clangd is a performance cliff, we should
avoid it.
Reviewers: ilya-biryukov
Subscribers: kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D69892
Summary:
Currently, stencils are defined as a sequence of `StencilParts`. This
differentiation adds an unneeded layer of complexity to the definition of
Stencils. This change significantly simplifies the type structure: a stencil is
now conceptually any object implementing `StencilInterface` and `Stencil` is
just a thin wrapper for pointers to this interface.
To account for the sequencing that was supported by the old `Stencil` type, we
introduce a sequencing class that implements `StencilInterface`. That is,
sequences are just another kind of Stencil and no longer have any special
status.
Corresponding to this change in the type structure, we change the way `cat` is
used (and defined). `cat` bundles multiple features: it builds a stencil from a
sequence of subcomponents and admits multiple different types for its arguments,
while coercing them into the right type. Previously, `cat` was also used to
coerce a single `StencilPart` into a `Stencil`. With that distinction gone, many
uses of `cat` (e.g. in the tests) are unnecessary and have, therefore, been
removed.
Reviewers: gribozavr
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D69613
Summary:
Most of the statements mirror the ones provided by clang AST.
Major differences are:
- expressions are wrapped into 'ExpressionStatement' instead of being
a subclass of statement,
- semicolons are always consumed by the leaf expressions (return,
expression satement, etc),
- some clang statements are not handled yet, we wrap those into an
UnknownStatement class, which is not present in clang.
We also define an 'Expression' and 'UnknownExpression' classes in order
to produce 'ExpressionStatement' where needed. The actual implementation
of expressions is not yet ready, it will follow later.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D63835
Yitzhak Mandelbaum