Summary:
We want to be sure they don't cause AST rebuilds or evict items from the cache.
D77847 is going to start sending spurious no-op changes (in case the preamble
was invalidated), this is cheap enough but we shouldn't regress that in future.
Reviewers: kadircet
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, jfb, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D78048
Summary:
This assertion was bad. It will show up once we start running preamble
thread async. Think about the following case:
- Update 1
builds a preamble, and an AST. Caches the AST.
- Update 2
Invalidates the cache, preamble hasn't changed.
- Update 3
Invalidates the cache, preamble hasn't changed
- Read
builds AST using preamble v1, and caches it.
preamble for v2 gets build, cache isn't invalidated since preamble is same.
generateDiags tries to reuse cached AST but latest version is 3 not 2, so
assertion fails.
Reviewers: sammccall
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D77664
Summary:
FileInputs are only written by ASTWorker thread, therefore it is safe
to read them without the lock inside that thread. It can still be read by other
threads through ASTWorker::getCurrentCompileCommand though.
This patch also gets rid of the smart pointer wrapping FileInputs as there is
never mutliple owners.
Reviewers: sammccall
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D77309
Summary:
This is another step for out-of-order preamble builds. To keep the
diagnostic behavior same, we only build ASTs either with "usable" preambles,
the ones that are fully applicable to a given ParseInput, or after building a
new preamble. Which is the same behaviour as what we do today. ASTs
built in the latter is called golden ASTs.
Reviewers: sammccall
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76725
Summary:
TUStatus api had a single thread in mind. This introudces a section
action to represent state of the preamble thread. In the file status extension,
we keep old behavior almost the same. We only prepend current task with a
`parsing includes` if preamble thread is working. We omit the idle thread in the
output unless both threads are idle.
Reviewers: sammccall
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76304
Summary:
First step to enable deferred preamble builds. Not intending to land it
alone, will have follow-ups that will implement full deferred build
functionality and will land after all of them are ready.
Reviewers: sammccall
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76125
Summary:
This helps us prevent races when scheduler (or any other thread) tries
to read a request while it's still running.
Reviewers: sammccall
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D75927
Summary:
This ties to an LSP feature (diagnostic versioning) but really a lot
of the value is in being able to log what's happening with file versions
and queues more descriptively and clearly.
As such it's fairly invasive, for a logging patch :-\
Key decisions:
- at the LSP layer, we don't reqire the client to provide versions (LSP
makes it mandatory but we never enforced it). If not provided,
versions start at 0 and increment. DraftStore handles this.
- don't propagate magically using contexts, but rather manually:
addDocument -> ParseInputs -> (ParsedAST, Preamble, various callbacks)
Context-propagation would hide the versions from ClangdServer, which
would make producing good log messages hard
- within ClangdServer, treat versions as opaque and unordered.
std::string is a convenient type for this, and allows richer versions
for embedders. They're "mandatory" but "null" is a reasonable default.
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D75582
Summary:
Otherwise they can force us to build lots of snapshots that we don't need.
Particularly, try to do this for operations that are frequently
generated by editors without explicit user interaction, and where
editing the file makes the result less useful. (Code action
enumeration is a good example).
https://github.com/clangd/clangd/issues/298
This doesn't return the "right" LSP error code (ContentModified) to the client,
we need to teach the cancellation API to distinguish between different causes.
Reviewers: kadircet
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, jfb, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D75602
The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, so that parallel computations such as LLD/ThinLTO can use all hardware threads in the system. Before this patch, on Windows, a maximum of 64 hardware threads could be used at most, in some cases dispatched only on one CPU socket.
== Background ==
Windows doesn't have a flat cpu_set_t like Linux. Instead, it projects hardware CPUs (or NUMA nodes) to applications through a concept of "processor groups". A "processor" is the smallest unit of execution on a CPU, that is, an hyper-thread if SMT is active; a core otherwise. There's a limit of 32-bit processors on older 32-bit versions of Windows, which later was raised to 64-processors with 64-bit versions of Windows. This limit comes from the affinity mask, which historically is represented by the sizeof(void*). Consequently, the concept of "processor groups" was introduced for dealing with systems with more than 64 hyper-threads.
By default, the Windows OS assigns only one "processor group" to each starting application, in a round-robin manner. If the application wants to use more processors, it needs to programmatically enable it, by assigning threads to other "processor groups". This also means that affinity cannot cross "processor group" boundaries; one can only specify a "preferred" group on start-up, but the application is free to allocate more groups if it wants to.
This creates a peculiar situation, where newer CPUs like the AMD EPYC 7702P (64-cores, 128-hyperthreads) are projected by the OS as two (2) "processor groups". This means that by default, an application can only use half of the cores. This situation could only get worse in the years to come, as dies with more cores will appear on the market.
== The problem ==
The heavyweight_hardware_concurrency() API was introduced so that only *one hardware thread per core* was used. Once that API returns, that original intention is lost, only the number of threads is retained. Consider a situation, on Windows, where the system has 2 CPU sockets, 18 cores each, each core having 2 hyper-threads, for a total of 72 hyper-threads. Both heavyweight_hardware_concurrency() and hardware_concurrency() currently return 36, because on Windows they are simply wrappers over std:🧵:hardware_concurrency() -- which can only return processors from the current "processor group".
== The changes in this patch ==
To solve this situation, we capture (and retain) the initial intention until the point of usage, through a new ThreadPoolStrategy class. The number of threads to use is deferred as late as possible, until the moment where the std::threads are created (ThreadPool in the case of ThinLTO).
When using hardware_concurrency(), setting ThreadCount to 0 now means to use all the possible hardware CPU (SMT) threads. Providing a ThreadCount above to the maximum number of threads will have no effect, the maximum will be used instead.
The heavyweight_hardware_concurrency() is similar to hardware_concurrency(), except that only one thread per hardware *core* will be used.
When LLVM_ENABLE_THREADS is OFF, the threading APIs will always return 1, to ensure any caller loops will be exercised at least once.
Differential Revision: https://reviews.llvm.org/D71775
Summary:
- This option forces a preamble rebuild to handle the odd case
of a missing header file being added
Reviewers: sammccall
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, jfb, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D73916
Summary:
Currently we delay AST rebuilds by 500ms after each edit, to wait for
further edits. This is a win if a rebuild takes 5s, and a loss if it
takes 50ms.
This patch sets debouncepolicy = clamp(min, ratio * rebuild_time, max).
However it sets min = max = 500ms so there's no policy change or actual
customizability - will do that in a separate patch.
See https://github.com/clangd/clangd/issues/275
Reviewers: hokein
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D73873
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
Summary:
This is the initial version. The cross-file rename is purely based on
the index.
It is hidden under a command-line flag, and only available for a small set
of symbols.
Reviewers: ilya-biryukov, sammccall
Subscribers: merge_guards_bot, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D69263
This much better reflects what is (now) in this header.
Maybe a rename to ParsedTU would be an improvement, but that's a much
more invasive change and life is too short.
ClangdUnit is dead, long live ClangdUnitTests!
llvm-svn: 370862
Summary:
Those errors are exposed at the first character of a file,
for a lack of a better place.
Previously, all errors were stored inside the AST and report
accordingly. However, errors in command-line argument parsing could
result in failure to produce the AST, so we need an alternative ways to
report those errors.
We take the following approach in this patch:
- buildCompilerInvocation() now requires an explicit DiagnosticConsumer.
- TUScheduler and TestTU now collect the diagnostics produced when
parsing command line arguments.
If pasing of the AST failed, diagnostics are reported via a new
ParsingCallbacks::onFailedAST method.
If parsing of the AST succeeded, any errors produced during
command-line parsing are stored alongside the AST inside the
ParsedAST instance and reported as previously by calling the
ParsingCallbacks::onMainAST method;
- The client code that uses ClangdServer's DiagnosticConsumer
does not need to change, it will receive new diagnostics in the
onDiagnosticsReady() callback
Errors produced when parsing command-line arguments are collected using
the same StoreDiags class that is used to collect all other errors. They
are recognized by their location being invalid. IIUC, the location is
invalid as there is no source manager at this point, it is created at a
later stage.
Although technically we might also get diagnostics that mention the
command-line arguments FileID with after the source manager was created
(and they have valid source locations), we choose to not handle those
and they are dropped as not coming from the main file. AFAICT, those
diagnostics should always be notes, therefore it's safe to drop them
without loosing too much information.
Reviewers: kadircet
Reviewed By: kadircet
Subscribers: nridge, javed.absar, MaskRay, jkorous, arphaman, cfe-commits, gribozavr
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66759
llvm-svn: 370177
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.
Differential revision: https://reviews.llvm.org/D66259
llvm-svn: 368944
Summary:
clangd supports a -j option to limit the amount of threads to use for parsing
TUs. However, when using -background-index (the default in later versions of
clangd), the parallelism used by clangd defaults to the hardware_parallelisn,
i.e. number of physical cores.
On shared hardware environments, with large projects, this can significantly
affect performance with no way to tune it down.
This change makes the -j parameter apply equally to parsing and background
index. It's not perfect, because the total number of threads is 2x the -j value,
which may still be unexpected. But at least this change allows users to prevent
clangd using all CPU cores.
Reviewers: kadircet, sammccall
Reviewed By: sammccall
Subscribers: javed.absar, jfb, sammccall, ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66031
llvm-svn: 368498
Summary:
By exposing a callback that can guard code publishing results of
'onMainAST' callback in the same manner we guard diagnostics.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: javed.absar, MaskRay, jkorous, arphaman, kadircet, hokein, jvikstrom, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64985
llvm-svn: 366577
Summary:
In practice, opening Foo.h will still often result in Foo.cpp making the
second index build instead of the first, as the rebuild policy doesn't
know to wait.
Reviewers: kadircet
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D64575
llvm-svn: 365888
Summary:
This makes addDocument non-blocking and would also allow code completion
(in fallback mode) to run when worker waits for the compile command.
Reviewers: sammccall, ilya-biryukov
Reviewed By: ilya-biryukov
Subscribers: javed.absar, MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D60607
llvm-svn: 358400
Summary:
When calling TUScehduler::runWithPreamble (e.g. in code compleiton), allow
entering a fallback mode when compile command or preamble is not ready, instead of
waiting. This allows clangd to perform naive code completion e.g. using identifiers
in the current file or symbols in the index.
This patch simply returns empty result for code completion in fallback mode. Identifier-based
plus more advanced index-based completion will be added in followup patches.
Reviewers: ilya-biryukov, sammccall
Reviewed By: sammccall
Subscribers: sammccall, javed.absar, MaskRay, jkorous, arphaman, kadircet, jdoerfert, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D59811
llvm-svn: 357916
Summary:
This enables include insertion by adding canonical includes into
preambledata.
Reviewers: ioeric, ilya-biryukov
Subscribers: javed.absar, MaskRay, jkorous, arphaman, cfe-commits
Differential Revision: https://reviews.llvm.org/D57508
llvm-svn: 353054
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
The new guideline is to qualify with 'llvm::' explicitly both in
'.h' and '.cpp' files. This simplifies moving the code between
header and source files and is easier to keep consistent.
llvm-svn: 350531
Sam McCall