This commit adds an optimization to clang-scan-deps and clang's preprocessor that skips excluded preprocessor
blocks by bumping the lexer pointer, and not lexing the tokens until reaching appropriate #else/#endif directive.
The skip positions and lexer offsets are computed when the file is minimized, directly from the minimized tokens.
On an 18-core iMacPro with macOS Catalina Beta I got 10-15% speed-up from this optimization when running clang-scan-deps on
the compilation database for a recent LLVM and Clang (3511 files).
Differential Revision: https://reviews.llvm.org/D67127
llvm-svn: 371656
Summary:
This significantly reduces the time required to run clangd tests, by
~10%.
Should also have an effect on other tests that run command-line parsing
multiple times inside a single invocation.
Reviewers: gribozavr, sammccall
Reviewed By: sammccall
Subscribers: kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D67163
llvm-svn: 370908
There's no need to purge symlinked entries in the FileManager,
as the new FileEntryRef API allows us to compute dependencies more
accurately when the FileManager is reused.
llvm-svn: 370493
to report the dependencies to the client
This will allow the scanner to report modular dependencies to the consumer.
This will also allow the scanner to accept regular cc1 clang invocations, e.g.
in an implementation of a libclang C API for clang-scan-deps, that I will add
follow-up patches for in the future.
llvm-svn: 370425
the dependency scanner on a single worker thread
This behavior can be controlled using the new `-reuse-filemanager` clang-scan-deps
option. By default the file manager is reused.
The added test/ClangScanDeps/symlink.cpp is able to pass with
the reused filemanager after the related FileEntryRef changes
landed earlier. The test test/ClangScanDeps/subframework_header_dir_symlink.m
still fails when the file manager is reused (I run the FileCheck with not to
make it PASS). I will address this in a follow-up patch that improves
the DirectoryEntry name modelling in the FileManager.
llvm-svn: 370420
Summary:
Returns the first token in every mapping where the token is an identifier.
This API is required to be able to highlight macro expansions in clangd.
Reviewers: hokein, ilya-biryukov
Subscribers: kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66470
llvm-svn: 369385
The change in r368681 contains a (probably unintentional) behavioral change for
rewrite rules with a single matcher. Previously, the single matcher would not
need to be bound (`joinCaseMatchers` returned it directly), even though a final
DynTypeMatcher was created and bound by `buildMatcher`. With the new change, a
single matcher will be bound, in addition to the final binding (which is now in
`buildMatchers`, but happens roughly at the same point in the overall flow).
This patch simply duplicates the "final matcher" trick: it creates an extra
DynTypedMatcher for each rewrite rule case matcher, and unconditionally makes it
bindable. This is probably not the right long-term fix, but it does allow
existing code to continue to work with this interface.
Subscribers: cfe-commits, gribozavr, ymandel
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66273
llvm-svn: 368958
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.
Differential revision: https://reviews.llvm.org/D66259
llvm-svn: 368942
Summary:
- Moved the SourceExtraction header from lib to include so that it can be used in clangd.
Reviewers: arphaman
Subscribers: ilya-biryukov, dexonsmith, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D65878
llvm-svn: 368743
Summary:
This patch removes an (artificial) limitation of `applyFirst`, which requires
that all of the rules' matchers can be grouped together in a single `anyOf()`.
This change generalizes the code to group the matchers into separate `anyOf`s
based on compatibility. Correspondingly, `buildMatcher` is changed to
`buildMatchers`, to allow for returning a set of matchers rather than just one.
Reviewers: gribozavr
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D65877
llvm-svn: 368681
D65628 requires a flag to specify the number of threads for a clang-doc step. It would be good to use ExecutorConcurrency after exposing it instead of creating a new one that has the same purpose.
Differential Revision: https://reviews.llvm.org/D65833
llvm-svn: 368196
This commit implements the fast dependency scanning mode in clang-scan-deps: the
preprocessing is done on files that are minimized using the dependency directives source minimizer.
A shared file system cache is used to ensure that the file system requests and source minimization
is performed only once. The cache assumes that the underlying filesystem won't change during the course
of the scan (or if it will, it will not affect the output), and it can't be evicted. This means that the
service and workers can be used for a single run of a dependency scanner, and can't be reused across multiple,
incremental runs. This is something that we'll most likely support in the future though.
Note that the driver still utilizes the underlying real filesystem.
This commit is also still missing the fast skipped PP block skipping optimization that I mentioned at EuroLLVM talk.
Additionally, the file manager is still not reused by the threads as well.
Differential Revision: https://reviews.llvm.org/D63907
llvm-svn: 368086
Summary:
While useful as a sentinel value when iterating over tokens, having
'eof' in the tree, seems to do more harm than good.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: javed.absar, kristof.beyls, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64576
llvm-svn: 368062
This patch is a prerequisite for using LangStandard from Driver in
https://reviews.llvm.org/D64793.
It moves LangStandard* and InputKind::Language to Basic. It is mostly
mechanical, with only a few changes of note:
- enum Language has been changed into enum class Language : uint8_t to
avoid a clash between OpenCL in enum Language and OpenCL in enum
LangFeatures and not to increase the size of class InputKind.
- Now that getLangStandardForName, which is currently unused, also checks
both canonical and alias names, I've introduced a helper getLangKind
which factors out a code pattern already used 3 times.
The patch has been tested on x86_64-pc-solaris2.11, sparcv9-sun-solaris2.11,
and x86_64-pc-linux-gnu.
There's a companion patch for lldb which uses LangStandard.h
(https://reviews.llvm.org/D65717).
While polly includes isl which in turn uses InputKind::C, that part of the
code isn't even built inside the llvm tree. I've posted a patch to allow
for both InputKind::C and Language::C upstream
(https://groups.google.com/forum/#!topic/isl-development/6oEvNWOSQFE).
Differential Revision: https://reviews.llvm.org/D65562
llvm-svn: 367864
Update the callers of FileManager::getFile and FileManager::getDirectory to handle the new llvm::ErrorOr-returning methods.
Signed-off-by: Harlan Haskins <harlan@apple.com>
llvm-svn: 367616
Summary:
Currently, Transformer rejects any changes to source locations inside macro
expansions. This change relaxes that constraint to allow rewrites when the
entirety of the expansion is replaced, since that can be mapped to replacing the
entirety of the expansion range in the file source. This change makes
Transformer consistent with the handling of edit ranges in `clang::edit::Commit`
(which is used, for example, for applying `FixItHint`s from diagnostics).
Reviewers: ilya-biryukov
Subscribers: gribozavr, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64518
llvm-svn: 366473
Summary:
Adds the function `getRangeForEdit` to validate that a given source range is
editable and, if needed, translate it into a range in the source file (for
example, if it's sourced in macro expansions).
Reviewers: ilya-biryukov
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64924
llvm-svn: 366469
Summary:
It's common to use compiler wrappers by setting CC="gomacc clang++".
This results in both args appearing in compile_commands.json, and clang's driver
can't handle this.
This patch attempts to recognize this pattern (by looking for well-known
wrappers) and dropping argv0 in this case.
It conservatively ignores other cases for now:
- wrappers with unknown names
- wrappers that accept -flags
- wrappers where the compiler to use is implied (usually cc or gcc)
This is done at the JSONCompilationDatabase level rather than somewhere more
fundamental, as (hopefully) this isn't a general conceptual problem, but a messy
aspect of the nature of the ecosystem around compile_commands.json.
i.e. compilation databases more tightly tied to the build system should not have
this problem.
Reviewers: phosek, klimek
Subscribers: llvm-commits
Tags: #llvm
Differential Revision: https://reviews.llvm.org/D64297
llvm-svn: 365887
Summary:
A tooling-focused alternative to the AST. This commit focuses on the
memory-management strategy and the structure of the AST.
More to follow later:
- Operations to mutate the syntax trees and corresponding textual
replacements.
- Mapping between clang AST nodes and syntax tree nodes.
- More node types corresponding to the language constructs.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: llvm-commits, mgorny, cfe-commits
Tags: #clang, #llvm
Differential Revision: https://reviews.llvm.org/D61637
........
Fixes buildbots which were crashing on SyntaxTests.exe
llvm-svn: 365465
Summary:
A tooling-focused alternative to the AST. This commit focuses on the
memory-management strategy and the structure of the AST.
More to follow later:
- Operations to mutate the syntax trees and corresponding textual
replacements.
- Mapping between clang AST nodes and syntax tree nodes.
- More node types corresponding to the language constructs.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: llvm-commits, mgorny, cfe-commits
Tags: #clang, #llvm
Differential Revision: https://reviews.llvm.org/D61637
llvm-svn: 365355
Summary:
This revision allows users to specify the insertion of an included directive (at
the top of the file being rewritten) as part of a rewrite rule. These
directives are bundled with `RewriteRule` cases, so that different cases can
potentially result in different include actions.
Reviewers: ilya-biryukov, gribozavr
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D63892
llvm-svn: 364917
thread worker code and better error handling
This commit extracts out the code that will powers the fast scanning
worker into a new file in a new DependencyScanning library. The error
and output handling is improved so that the clients can gather
errors/results from the worker directly.
Differential Revision: https://reviews.llvm.org/D63681
llvm-svn: 364474
Summary:
Wraps JSON compilation database with a target and mode adding database
wrapper. So that driver can correctly figure out which toolchain to use.
Note that clients that wants to make use of this target discovery mechanism
needs to link in TargetsInfos and initialize them at startup.
Reviewers: ilya-biryukov
Subscribers: mgorny, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D63755
llvm-svn: 364386
Summary:
Previously, we performed rename for all kinds of symbols (local, global).
This patch narrows the scope by only renaming symbols not being used
outside of the main file (with index asisitance). Renaming global
symbols is not supported at the moment (return an error).
Reviewers: sammccall
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D63426
llvm-svn: 364283
Summary:
This change makes sure we have a single mapping for each macro expansion,
even if the result of expansion was empty.
To achieve that, we take information from PPCallbacks::MacroExpands into
account. Previously we relied only on source locations of expanded tokens.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D62953
llvm-svn: 364236
Summary: Used in clangd for a code tweak that expands a macro.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D62954
llvm-svn: 363698
Summary:
The `before` and `after` selectors allow users to specify a zero-length range --
a point -- at the relevant location in an AST-node's source. Point ranges can
be useful, for example, to insert a change using an API that takes a range to be
modified (e.g. `tooling::change()`).
Reviewers: ilya-biryukov
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D62419
llvm-svn: 361955
Summary:
Conceptually, a single-case RewriteRule has a matcher, edit(s) and an (optional)
explanation. `makeRule` previously only took the matcher and edit(s). This
change adds (optional) support for the explanation.
Reviewers: ilya-biryukov
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D62390
llvm-svn: 361643
Summary:
RangeSelector had a number of cases of capturing a StringRef in a lambda, which
lead to dangling references. This change converts all uses in the API of
`StringRef` to `std::string` to avoid this problem. `std::string` in the API is
a reasonable choice, because the combinators are always storing the string
beyond the life of the combinator construction.
Reviewers: ilya-biryukov, gribozavr
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D62328
llvm-svn: 361514
Using a static function as a template parameter gets a bogus compile-time
error with Visual Studio 2017, prior to version 15.8. Our current
minimum-version requirement is a particular update to VS2015, and we
assume all Visual Studio 2017 versions are usable. This patch makes the
code buildable with older versions of VS2017, and can be reverted after
we upgrade the minimum version sometime in the future.
Description of the Microsoft bug:
https://developercommunity.visualstudio.com/content/problem/25334/error-code-c2971-when-specifying-a-function-as-the.html
Differential Revision: https://reviews.llvm.org/D62202
llvm-svn: 361502
Alex Lorenz