Commit Graph

911 Commits

Author SHA1 Message Date
Eduardo Caldas 23657d9cc3 [SyntaxTree] Add reverse links to syntax Nodes.
Rationale:
Children of a syntax tree had forward links only, because there was no
need for reverse links.

This need appeared when we started mutating the syntax tree.
On a forward list, to remove a target node in O(1) we need a pointer to the node before the target. If we don't have this "before" pointer, we have to find it, and that requires O(n).
So in order to remove a syntax node from a tree, we would similarly need to find the node before to then remove. This is both not ergonomic nor does it have a good complexity.

Differential Revision: https://reviews.llvm.org/D90240
2020-11-05 09:33:53 +00:00
Sam McCall dd6f7ee05e [Syntax] DeclaratorList is a List
I think this was just an oversight.

Differential Revision: https://reviews.llvm.org/D90541
2020-11-03 03:29:06 +01:00
Sam McCall d4934eb5f8 [Syntax] Add iterators over children of syntax trees.
This gives us slightly nicer syntax (foreach) for idioms currently expressed
as a loop, and the option to use range algorithms where it makes sense
(e.g. llvm::all_of et al encapsulate the needed flow control in a useful way).

It's also a building block for iteration over filtered views (e.g. iterate over
all Stmt children, with the right type):
for (const Statement &S : filter<Statement>(N.children()))
  ...

I realize the recent direction has been mostly towards strongly-typed
node-specific facilities, but I think it's important we have convenient
generic facilities too.

Differential Revision: https://reviews.llvm.org/D90023
2020-10-28 12:37:57 +01:00
Kirill Bobyrev 5ad6bbacf0
[clangd] Start using SyntaxTrees for folding ranges feature
This is an initial attempt to start using Syntax Trees in clangd while improving state of folding ranges feature and experimenting with Syntax Tree capabilities.

Reviewed By: sammccall

Differential Revision: https://reviews.llvm.org/D88553
2020-10-27 16:47:35 +01:00
Nathan Ridge 2756e2ee0b [libTooling] Recognize sccache as a compiler wrapper in compilation database commands
sccache is a compiler caching tool similar to ccache.

Differential Revision: https://reviews.llvm.org/D88790
2020-10-27 01:46:32 -04:00
Duncan P. N. Exon Smith cf593d224c SourceManager: getFileEntryRefForID => getNonBuiltinFilenameForID, NFC
`SourceManager::getFileEntryRefForID`'s remaining callers just want the
filename component, which is coming from the `FileInfo`. Replace the API
with `getNonBuiltinFilenameForID`, which also removes another use of
`FileEntryRef::FileEntryRef` outside of `FileManager`.

Both callers are collecting file dependencies, and one of them relied on
this API to filter out built-ins (as exposed by
clang/test/ClangScanDeps/modules-full.cpp). It seems nice to continue
providing that service.

Differential Revision: https://reviews.llvm.org/D89508
2020-10-22 22:20:50 -04:00
Yitzhak Mandelbaum 6f8f5cb77e [libTooling] Add function to Transformer to create a no-op edit.
This functionality is commonly needed in clang tidy checks (based on
transformer) that only print warnings, without suggesting any edits. The no-op
edit allows the user to associate a diagnostic message with a source location.

Differential Revision: https://reviews.llvm.org/D89961
2020-10-22 21:29:03 +00:00
Mikhail Maltsev 7819411837 [clang] Use SourceLocation as key in hash maps, NFCI
The patch adjusts the existing `llvm::DenseMap<unsigned, T>` and
`llvm::DenseSet<unsigned>` objects that store source locations, so
that they use `SourceLocation` directly instead of `unsigned`.

This patch relies on the `DenseMapInfo` trait added in D89719.

It also replaces the construction of `SourceLocation` objects from
the constants -1 and -2 with calls to the trait's methods `getEmptyKey`
and `getTombstoneKey` where appropriate.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D69840
2020-10-20 16:24:09 +01:00
Haojian Wu 84048e234f [format] foo.<name>.h should be the main-header for foo.<name>.cc
This fixes a regression introduced in https://reviews.llvm.org/D88640.

Differential Revision: https://reviews.llvm.org/D89783
2020-10-20 13:27:02 +02:00
Alex Richardson a2214757e2 Move clang/Tooling/Core/Lookup.h to clang/Tooling/Refactoring/Lookup.h
This allows removing the clangAST dependency from libclangToolingCore and
therefore allows clang-format to be built without depending on clangAST.
Before 1166 files had to be compiled for clang-format, now only 796.

Reviewed By: bkramer

Differential Revision: https://reviews.llvm.org/D89708
2020-10-20 10:13:28 +01:00
Duncan P. N. Exon Smith b3eff6b7bb Lexer: Update the Lexer to use MemoryBufferRef, NFC
Update `Lexer` / `Lexer::Lexer` to use `MemoryBufferRef` instead of
`MemoryBuffer*`. Callers that were acquiring a `MemoryBuffer*` via
`SourceManager::getBuffer` were updated, such that if they checked
`Invalid` they use `getBufferOrNone` and otherwise `getBufferOrFake`.

Differential Revision: https://reviews.llvm.org/D89398
2020-10-19 19:10:21 -04:00
Haojian Wu 1e32df2f91 [clang-rename] Fix rename on variable templates.
This patch adds support for renaming variable templates.

Differential Revision: https://reviews.llvm.org/D89300
2020-10-19 09:44:59 +02:00
Haojian Wu 45a15dc682 [clang-rename] Fix rename on function template specializations.
previously, we missed to rename occurrences to explicit function template
specilizations.

Differential Revision: https://reviews.llvm.org/D89221
2020-10-19 09:32:17 +02:00
Yitzhak Mandelbaum 65cb4fdd69 [libTooling] Change `after` range-selector to operate only on source ranges
Currently, `after` fails when applied to locations in macro arguments.  This
change projects the subrange into a file source range and then applies `after`.

Differential Revision: https://reviews.llvm.org/D89468
2020-10-15 20:58:30 +00:00
Haojian Wu e6b4179c5d Fix a broken build for gcc <= 7.1
we need add a "this->" inside the lambda body to workaround it. Rewrite
it to normal for-range loop.
2020-10-14 15:14:59 +02:00
Eduardo Caldas 6fbad9bf30 [SyntaxTree][NFC] Nit on `replaceChildRangeLowLevel` 2020-10-14 09:40:37 +00:00
Eduardo Caldas 72732acade [SyntaxTree] Bug fix in `MutationsImpl::addAfter`.
* Add assertions to other `MutationsImpl` member functions
* `findPrevious` is a free function

Differential Revision: https://reviews.llvm.org/D89314
2020-10-14 09:22:01 +00:00
Eduardo Caldas 4178f8f2f0 [SyntaxTree] Improve safety of `replaceChildRangeLowLevel`
* Add assertions for other preconditions.
* If nothing is modified, don't mark it.

Differential Revision: https://reviews.llvm.org/D89303
2020-10-14 09:18:32 +00:00
Haojian Wu 27c691cf62 [clang-rename] Simplify the code of handling class paritial specializations, NFC.
Instead of collecting all specializations and doing a post-filterin, we
can just get all targeted specializations from getPartialSpecializationsizations.

Differential Revision: https://reviews.llvm.org/D89220
2020-10-14 09:57:55 +02:00
Duncan P. N. Exon Smith 69feac12d0 Lex: Avoid MemoryBuffer* key in ExcludedPreprocessorDirectiveSkipMapping, NFC
This is a prep patch for changing SourceManager to return
`Optional<MemoryBufferRef>` instead of `MemoryBuffer`. With that change the
address of the MemoryBuffer will be gone, so instead use the start of the
buffer as the key for this map.

No functionality change intended, as it's expected that the pointer identity
matches between the buffers and the buffer data.

Radar-Id: rdar://70139990
Differential Revision: https://reviews.llvm.org/D89136
2020-10-12 17:39:01 -04:00
Duncan P. N. Exon Smith d07b290e4b DependencyScanning: pull factory function into MinimizedVFS, NFC
Avoid need for getBufferPtr API, simplifying another patch. No
functionality change.
2020-10-12 17:25:10 -04:00
Haojian Wu c1b209cc61 [Format] Don't treat compound extension headers (foo.proto.h) as foo.cc main-file header.
We receive internal bugs about this false positives after D86597.

Differential Revision: https://reviews.llvm.org/D88640.
2020-10-01 19:57:57 +02:00
Eduardo Caldas 5011d43108 Migrate Declarators to use the List API
After this change all nodes that have a delimited-list are using the
`List` API.

Implementation details:
Let's look at a declaration with multiple declarators:
`int a, b;`
To generate a declarator list node we need to have the range of
declarators: `a, b`:
However, the `ClangAST` actually stores them as separate declarations:
`int a   ;`
`int    b;`
We solve that by appropriately marking the declarators on each separate
declaration in the `ClangAST` and then for the final declarator `int
b`, shrinking its range to fit to the already marked declarators.

Differential Revision: https://reviews.llvm.org/D88403
2020-10-01 13:56:31 +00:00
Eduardo Caldas 66bcb14312 [SyntaxTree][Synthesis] Fix: `deepCopy` -> `deepCopyExpandingMacros`.
There can be Macros that are tagged with `modifiable`. Thus verifying
`canModifyAllDescendants` is not sufficient to avoid macros when deep
copying.

We think the `TokenBuffer` could inform us whether a `Token` comes from
a macro. We'll look into that when we can surface this information
easily, for instance in unit tests for `ComputeReplacements`.

Differential Revision: https://reviews.llvm.org/D88034
2020-09-22 09:15:21 +00:00
Eduardo Caldas af582c9b0f [SyntaxTree] Test `findFirstLeaf` and `findLastLeaf`
* Introduce `TreeTest.cpp` to unit test `Tree.h`
* Add `generateAllTreesWithShape` to generating test cases
* Add tests for `findFirstLeaf` and `findLastLeaf`
* Fix implementations of `findFirstLeaf` and `findLastLeaf` that had
been broken when empty `Tree` were present.

Differential Revision: https://reviews.llvm.org/D87779
2020-09-22 06:47:36 +00:00
Eduardo Caldas 1dc7836aed [SyntaxTree][Nit] Take `ArrayRef` instead of `std::vector` as argument for `createTree`
I also assured that there are no other functions unnecessarily using std::vector as argument.

Differential Revision: https://reviews.llvm.org/D88024
2020-09-22 06:47:36 +00:00
Eduardo Caldas 87f0b51d68 [SyntaxTree][NFC] follow naming convention + remove auto on empty vector declaration
Differential Revision: https://reviews.llvm.org/D88004
2020-09-21 11:45:15 +00:00
Eduardo Caldas 4a5cc389c5 [SyntaxTree][Synthesis] Implement `deepCopy`
Differential Revision: https://reviews.llvm.org/D87749
2020-09-21 09:27:15 +00:00
Simon Pilgrim a735d6eae2 JSONCompilationDatabase.cpp - cleanup key parsing error checks. NFCI.
Merge the key + sequence/value checks with the key handling code.

Reduces the number of key string comparisons and avoids a number of clang static analyzer null dereference warnings.
2020-09-20 13:55:29 +01:00
Vitaly Buka 55edf7039e [NFC] clang-format one line 2020-09-17 18:03:55 -07:00
Vitaly Buka 03358becbf [NFC][Lsan] Fix zero-sized array compilation error 2020-09-17 17:59:52 -07:00
Eduardo Caldas 1e19165bd8 [SyntaxTree][Synthesis] Fix allocation in `createTree` for more general use
Prior to this change `createTree` could not create arbitrary syntax
trees. Now it dispatches to the constructor of the concrete syntax tree
according to the `NodeKind` passed as argument. This allows reuse inside
the Synthesis API.  # Please enter the commit message for your changes.
Lines starting

Differential Revision: https://reviews.llvm.org/D87820
2020-09-17 16:09:35 +00:00
Eduardo Caldas ceb0128509 [SyntaxTree][List] `assertInvariants` for `List`s
Differential Revision: https://reviews.llvm.org/D87600
2020-09-14 10:36:11 +00:00
Eduardo Caldas 0f4cc64fd7 [SyntaxTree] Provide `List::classof`
Differential Revision: https://reviews.llvm.org/D87599
2020-09-14 10:35:41 +00:00
Eduardo Caldas 12232dc181 [SyntaxTree][List] Fix: `ParameterDeclarationList` is the `List` inside `ParametersAndQualifiers`
Differential Revision: https://reviews.llvm.org/D87598
2020-09-14 10:35:41 +00:00
Dmitri Gribenko e10df779f0 Fix clang Wrange-loop-analysis in BuildTree.cpp
Building on Mac OS with clang 12:

```
jhemphill@jhemphill-mbp build % clang --version
Apple clang version 12.0.0 (clang-1200.0.26.2)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
```

yields one warning:

```
/Users/jhemphill/oss/llvm-project/clang/lib/Tooling/Syntax/BuildTree.cpp:1126:22: warning: loop variable 'Arg' is always a copy because the range of type 'llvm::iterator_range<clang::Stmt::CastIterator<clang::Expr, clang::Expr *, clang::Stmt *> >' does not return a reference [-Wrange-loop-analysis]
    for (const auto &Arg : Args) {
                     ^
/Users/jhemphill/oss/llvm-project/clang/lib/Tooling/Syntax/BuildTree.cpp:1126:10: note: use non-reference type 'clang::Expr *'
    for (const auto &Arg : Args) {
```

It appears that `Arg` is an `Expr*`, passed by value rather than by const reference.

Reviewed By: eduucaldas, gribozavr2

Differential Revision: https://reviews.llvm.org/D87482
2020-09-12 01:07:54 +02:00
Eduardo Caldas 7c37b82f5b [SyntaxTree][Synthesis] Add support for Tree.
In a future patch
* Implement helper function to generate Trees for tests
* and test Tree methods, namely `findFirstLeaf` and `findLastLeaf`

Differential Revision: https://reviews.llvm.org/D87533
2020-09-11 20:37:23 +00:00
Eduardo Caldas 238ae4eee0 [SyntaxTree] Add const qualifiers, from [llvm-qualified-auto]
Differential Revision: https://reviews.llvm.org/D87522
2020-09-11 18:39:02 +00:00
Eduardo Caldas 515238d5b1 [SyntaxTree] Reduce visibility of `Arena::lexBuffer`.
Differential Revision: https://reviews.llvm.org/D87523
2020-09-11 18:32:38 +00:00
Eduardo Caldas 5d152127d4 [SyntaxTree][Synthesis] Add support for simple Leafs and test based on tree dump
Differential Revision: https://reviews.llvm.org/D87495
2020-09-11 18:22:00 +00:00
Eduardo Caldas 4c14ee61b7 [SyntaxTree] Rename functions to start with verb
According to LLVM coding standards:
https://llvm.org/docs/CodingStandards.html#name-types-functions-variables-and-enumerators-properly

Differential Revision: https://reviews.llvm.org/D87498
2020-09-11 14:54:18 +00:00
Yitzhak Mandelbaum a5cefd95cc [libTooling] Fix use of `char` in comparison.
Fixes Transformer's `Range` parser to handle `char` in a platform-independent way.

Differential Revision: https://reviews.llvm.org/D87409
2020-09-11 12:24:09 +00:00
Caroline Concatto 257b29715b [flang][driver] Add the new flang compiler and frontend drivers
Summary:

This is the first patch implementing the new Flang driver as outlined in [1],
[2] & [3]. It creates Flang driver (`flang-new`) and Flang frontend driver
(`flang-new -fc1`). These will be renamed as `flang` and `flang -fc1` once the
current Flang throwaway driver, `flang`, can be replaced with `flang-new`.

Currently only 2 options are supported: `-help` and `--version`.

`flang-new` is implemented in terms of libclangDriver, defaulting the driver
mode to `FlangMode` (added to libclangDriver in [4]). This ensures that the
driver runs in Flang mode regardless of the name of the binary inferred from
argv[0].

The design of the new Flang compiler and frontend drivers is inspired by it
counterparts in Clang [3]. Currently, the new Flang compiler and frontend
drivers re-use Clang libraries: clangBasic, clangDriver and clangFrontend.

To identify Flang options, this patch adds FlangOption/FC1Option enums.
Driver::printHelp is updated so that `flang-new` prints only Flang options.
The new Flang driver is disabled by default. To enable it, set
`-DBUILD_FLANG_NEW_DRIVER=ON` when configuring CMake and add clang to
`LLVM_ENABLE_PROJECTS` (e.g. -DLLVM_ENABLE_PROJECTS=“clang;flang;mlir”).

[1] “RFC: new Flang driver - next steps”
http://lists.llvm.org/pipermail/flang-dev/2020-July/000470.html
[2] “RFC: Adding a fortran mode to the clang driver for flang”
http://lists.llvm.org/pipermail/cfe-dev/2019-June/062669.html
[3] “RFC: refactoring libclangDriver/libclangFrontend to share with Flang”
http://lists.llvm.org/pipermail/cfe-dev/2020-July/066393.html
[4] https://reviews.llvm.org/rG6bf55804924d5a1d902925ad080b1a2b57c5c75c

co-authored-by: Andrzej Warzynski <andrzej.warzynski@arm.com>

Reviewed By: richard.barton.arm, sameeranjoshi

Differential Revision: https://reviews.llvm.org/D86089
2020-09-11 10:55:54 +01:00
Eduardo Caldas f5087d5c72 [SyntaxTree] Fix crash on functions with default arguments.
* Do not visit `CXXDefaultArgExpr`
* To build `CallArguments` nodes, just go through non-default arguments

Differential Revision: https://reviews.llvm.org/D87249
2020-09-08 09:49:30 +00:00
Eduardo Caldas 134455a07c [SyntaxTree] Ignore implicit `CXXFunctionalCastExpr` wrapping constructor
Differential Revision: https://reviews.llvm.org/D87229
2020-09-08 09:44:23 +00:00
Eduardo Caldas 46f4439dc9 [SyntaxTree] Ignore implicit leaf `CXXConstructExpr`
Differential Revision: https://reviews.llvm.org/D86700
2020-09-08 09:44:23 +00:00
Eduardo Caldas 2325d6b42f [SyntaxTree] Ignore implicit non-leaf `CXXConstructExpr`
Differential Revision: https://reviews.llvm.org/D86699
2020-09-08 09:44:23 +00:00
Shivanshu Goyal 2168dbf4cc getClangStripDependencyFileAdjuster(): Do not remove -M args when using MSVC cl driver
MSVC's cl.exe has a few command line arguments which start with -M such
as "-MD", "-MDd", "-MT", "-MTd", "-MP".
These arguments are not dependency file generation related, and these
arguments were being removed by getClangStripDependencyFileAdjuster()
which was wrong.

Differential revision: https://reviews.llvm.org/D86999
2020-09-08 10:21:18 +02:00
Yitzhak Mandelbaum d4f3903131 [libTooling] Provide overloads of `rewriteDescendants` that operate directly on an AST node.
The new overloads apply directly to a node, like the
`clang::ast_matchers::match` functions, Rather than generating an
`EditGenerator` combinator.

Differential Revision: https://reviews.llvm.org/D87031
2020-09-03 14:39:50 +00:00
Yitzhak Mandelbaum 6f0a3711bc [libTooling] Restore defaults for matchers in makeRule.
This patch restores the default traversal for Transformer's `makeRule` to
`TK_AsIs`. The implicit mode has proven problematic.

Differential Revision: https://reviews.llvm.org/D87048
2020-09-02 19:36:14 +00:00
Eduardo Caldas a1461953f4 [SyntaxTree] Add coverage for declarators and init-declarators 2020-08-28 12:19:38 +00:00
Eduardo Caldas 38bc0060e6 [SyntaxTree][NFC] Refactor function templates into functions taking base class
The refactored functions were
* `isReponsibleForCreatingDeclaration`
* `getQualifiedNameStart`

Differential Revision: https://reviews.llvm.org/D86719
2020-08-28 12:19:38 +00:00
Sam McCall 266825620c [Tooling][Format] Treat compound extensions (foo.bar.cc) as matching foo.h
Motivating use case is ".cu.cc" extensions used in some bazel projects.

Alternative is to work around this with IncludeIsMainRegex in styles.
I proposed this approach because it seems like a better default.

Differential Revision: https://reviews.llvm.org/D86597
2020-08-27 15:24:17 +02:00
Eduardo Caldas ac87a0b587 [SyntaxTree][NFC][Style] Functions start with lowercase
Differential Revision: https://reviews.llvm.org/D86682
2020-08-27 12:55:24 +00:00
Eduardo Caldas fda3fa822c [SyntaxTree][NFC] Append "get" to syntax Nodes accessor names
Differential Revision: https://reviews.llvm.org/D86679
2020-08-27 12:55:23 +00:00
Eduardo Caldas 718e550cd0 [SyntaxTree] Refactor `NodeRole`s
Previously a NodeRole would generally be prefixed with the `NodeKind`,
we remove this prefix, as it we redundant and made tests more noisy.

Differential Revision: https://reviews.llvm.org/D86636
2020-08-27 05:16:00 +00:00
Eduardo Caldas dc3d474327 [SyntaxTree] Migrate `ParamatersAndQualifiers` to use the new List API
Fix: Add missing `List::getTerminationKind()`, `List::canBeEmpty()`,
`List::getDelimiterTokenKind()` for `CallArguments`.

Differential Revision: https://reviews.llvm.org/D86600
2020-08-26 16:46:19 +00:00
Eduardo Caldas 2de2ca348d [SyntaxTree] Add support for `CallExpression`
* Generate `CallExpression` syntax node for all semantic nodes inheriting from
`CallExpr` with call-expression syntax - except `CUDAKernelCallExpr`.
* Implement all the accessors
* Arguments of `CallExpression` have their own syntax node which is based on
the `List` base API

Differential Revision: https://reviews.llvm.org/D86544
2020-08-26 07:03:49 +00:00
Eduardo Caldas c655d80815 [SyntaxTree] Extend the syntax tree dump to also cover `NodeRole`
We should see `NodeRole` information in the dump because that exposes how the
accessors will behave.

Functional changes in the dump:
* Surround Leaf tokens with `'`
* Append `Node` dumps with `NodeRole` information, except for unknown roles
* Append marks to `Node` dumps, instead of prepending

Non-functional changes:
* `::dumpTokens(llvm::raw_ostream, ArrayRef<syntax::Token>, const
SourceManager &SM)` always received as parameter a `syntax::Token *`
pointing to `Leaf::token()`. Changed the function to
`dumpLeaf(llvm::raw_ostream, syntax::Leaf *, const SourceManager&)`
* `dumpTree` acted on a Node, rename to `dumpNode`

Differential Revision: https://reviews.llvm.org/D85330
2020-08-25 06:34:40 +00:00
Eduardo Caldas 85c15f17cc [SyntaxTree] Add support for `this`
Differential Revision: https://reviews.llvm.org/D86298
2020-08-21 08:01:29 +00:00
Eduardo Caldas a4ef9e8643 [SyntaxTree] Unify logic for generating `id-expression` 2020-08-20 14:57:35 +00:00
Eduardo Caldas ba32915db2 [SyntaxTree] Add support for `MemberExpression`
Differential Revision: https://reviews.llvm.org/D86227
2020-08-20 14:57:35 +00:00
Benjamin Kramer b98e25b6d7 Make helpers static. NFC. 2020-08-19 16:00:03 +02:00
Eduardo Caldas ac37afa650 [SyntaxTree] Unbox operators into tokens for nodes generated from `CXXOperatorCallExpr`
For an user define `<`, `x < y` would yield the syntax tree:
```
BinaryOperatorExpression
|-IdExpression
| `-UnqualifiedId
|   `-x
|-IdExpression
| `-UnqualifiedId
|   `-<
`-IdExpression
  `-UnqualifiedId
    `-y
```
But there is no syntatic difference at call site between call site or
built-in `<`. As such they should generate the same syntax tree, namely:
```
BinaryOperatorExpression
|-IdExpression
| `-UnqualifiedId
|   `-x
|-<
`-IdExpression
  `-UnqualifiedId
    `-y
```

Differential Revision: https://reviews.llvm.org/D85750
2020-08-12 08:01:18 +00:00
Simon Pilgrim b9aaf32f46 Fix MSVC "not all control paths return a value" warning. NFC. 2020-08-11 18:13:03 +01:00
Yitzhak Mandelbaum d8c1f43dcc [libTooling] Move RewriteRule include edits to ASTEdit granularity.
Currently, changes to includes are applied to an entire rule. However,
include changes may be specific to particular edits within a rule (for example,
they may apply to one file but not another). Also, include changes may need to
carry metadata, just like other changes. So, we make include changes first-class
edits.

Reviewed By: tdl-g

Differential Revision: https://reviews.llvm.org/D85734
2020-08-11 16:47:14 +00:00
Yitzhak Mandelbaum 645dd1b3bf [libTooling] Cleanup and reorder `RewriteRule.h`.
This patch lifts `RootID` out of the `RewriteRule` class so that constructs
(e.g. inline functions) can that refer to the root id don't need to depend on
the `RewriteRule` class.

With this dependency, the patch is able to collect all `ASTEdit` helper function
declarations together with the class declaration, before the introduction of the
`RewriteRule` class. In the process, we also adjust some of the comments.

This patch is essentially a NFC.

Reviewed By: gribozavr2

Differential Revision: https://reviews.llvm.org/D85733
2020-08-11 16:35:36 +00:00
Kadir Cetinkaya ea8e71c3da
[clang][HeaderInsert] Do not treat defines with values as header guards
This was resulting in inserting headers at bogus locations, see
https://github.com/ycm-core/YouCompleteMe/issues/3736 for an example.

Differential Revision: https://reviews.llvm.org/D85590
2020-08-11 16:02:11 +02:00
Eduardo Caldas f9500cc487 [SyntaxTree] Expand support for `NestedNameSpecifier`
Summary:
We want NestedNameSpecifier syntax nodes to be generally supported, not
only for `DeclRefExpr` and `DependentScopedDeclRefExpr`.

To achieve this we:
* Use the `RecursiveASTVisitor`'s API to traverse
`NestedNameSpecifierLoc`s and automatically create its syntax nodes
* Add links from the `NestedNameSpecifierLoc`s to their syntax nodes.

In this way, from any semantic construct that has a `NestedNameSpecifier`,
we implicitly generate its syntax node via RAV and we can easily access
this syntax node via the links we added.
2020-08-10 15:47:20 +00:00
Eduardo Caldas fdbd599653 [SyntaxTree] Implement `NestedNameSpecifier` using the `List` base API 2020-08-10 13:43:21 +00:00
Eduardo Caldas a90c78ac52 [SyntaxTree] Implement the List construct.
We defined a List construct to help with the implementation of list-like
grammar rules. This is a first implementation of this API.

Differential Revision: https://reviews.llvm.org/D85295
2020-08-10 10:32:28 +00:00
Simon Pilgrim 090f9d5a55 Fix MSVC "not all control paths return a value" warning. NFC. 2020-08-08 19:12:11 +01:00
Eduardo Caldas 8abb5fb68f [SyntaxTree] Use simplified grammar rule for `NestedNameSpecifier` grammar nodes
This is our grammar rule for nested-name-specifiers:
globalbal-specifier:
  /*empty*/
simple-template-specifier:
  template_opt simple-template-id
name-specifier:
  global-specifier
  decltype-specifier
  identifier
  simple-template-specifier
nested-name-specifier:
  list(name-specifier, ::, non-empty, terminated)

It is a relaxed version of C++ [expr.prim.id] and quite simpler to map to our API.

TODO: refine name specifiers, `simple-template-name-specifier` and
decltype-name-specifier` are token soup for now.
2020-08-07 18:05:47 +00:00
Eduardo Caldas ba41a0f733 [SyntaxTree][NFC] remove redundant namespace-specifiers
Differential Revision: https://reviews.llvm.org/D85427
2020-08-07 08:45:29 +00:00
Bruno Ricci 19701458d4
[clang][nearly-NFC] Remove some superfluous uses of NamedDecl::getNameAsString
`OS << ND->getDeclName();` is equivalent to `OS << ND->getNameAsString();`
without the extra temporary string.

This is not quite a NFC since two uses of `getNameAsString` in a
diagnostic are replaced, which results in the named entity being
quoted with additional "'"s (ie: 'var' instead of var).
2020-08-05 13:54:37 +01:00
Eduardo Caldas 8ce15f7eeb [SyntaxTree] Fix crash on pointer to member function
Differential Revision: https://reviews.llvm.org/D85146
2020-08-04 14:31:12 +00:00
Eduardo Caldas 860cbbdd6b [SyntaxTree] Add support for `LiteralExpression`
We use inheritance to model the grammar's disjunction rule:
literal:
  integer-literal
  character-literal
  floating-point-literal
  string-literal
  boolean-literal
  pointer-literal
  user-defined-literal

Differential Revision: https://reviews.llvm.org/D85186
2020-08-04 14:05:09 +00:00
Kadir Cetinkaya 76c3ec814d
[clang][Tooling] Optimize addTargetAndMode in case of invalid modes
This skips searching for `target` related flags in the existing args if
we don't have a valid target to insert.

Depends on D85076

Differential Revision: https://reviews.llvm.org/D85077
2020-08-03 13:58:27 +02:00
Kadir Cetinkaya 87de54dbb6
[clang][Tooling] Fix addTargetAndModeForProgramName to use correct flag names
The logic was using incorrect flag versions. For example:
- `-target=` can't be a prefix, it must be `--target=`.
- `--driver-mode` can't appear on its own, value must be attached to it.

While fixing those, also changes the append logic to make use of new
`--target=X` format instead of the legacy `-target X` version.

In addition to that makes use of the OPTTable instead of hardcoded strings to
make sure helper also gets updated if clang's options are modified.

Differential Revision: https://reviews.llvm.org/D85076
2020-08-03 11:46:58 +02:00
Kadir Cetinkaya 1618828165
[clang][Syntax] syntax::Arena doesnt own TokenBuffer
Currently an Arena can only be built while consuming a TokenBuffer,
some users (like clangd) might want to share a TokenBuffer with multiple
compenents. This patch changes Arena's TokenBuffer member to be a reference so
that it can be created with read-only token buffers.

Differential Revision: https://reviews.llvm.org/D84973
2020-07-31 11:50:01 +02:00
Yitzhak Mandelbaum 04a21318b5 [libTooling] Add a `between` range-selector combinator.
Adds the `between` combinator and registers it with the parser. As a driveby, updates some deprecated names to their current versions.

Reviewed By: gribozavr2

Differential Revision: https://reviews.llvm.org/D84315
2020-07-28 17:26:12 +00:00
Yitzhak Mandelbaum c332a984ae [libTooling] Add an `EditGenerator` that applies a rule throughout a bound node.
The new combinator, `rewriteDescendants`, applies a rewrite rule to all
descendants of a specified bound node.  That rewrite rule can refer to nodes
bound by the parent, both in the matcher and in the edits.

Reviewed By: gribozavr2

Differential Revision: https://reviews.llvm.org/D84409
2020-07-24 14:38:17 +00:00
Yitzhak Mandelbaum cf42877812 [libTooling] Add assorted `EditGenerator` combinators.
Summary:
This patch adds various combinators that help in constructing `EditGenerator`s:
   * `noEdits`
   * `ifBound`, specialized to `ASTEdit`
   * `flatten` and `flattenVector` which allow for easy construction from a set
     of sub edits.
   * `shrinkTo`, which generates edits to shrink a given range to another that
     it encloses.

Reviewers: asoffer, gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D84310
2020-07-24 12:51:54 +00:00
Andy Soffer e5b3202b6f [libTooling] In Clang Transformer, change `Metadata` field to deferred evaluation.
`Metadata` is being changed from an `llvm::Any` to a `MatchConsumer<llvm::Any>`
so that it's evaluation can be be dependent on on `MatchResult`s passed in.

Reviewed By: ymandel, gribozavr2

Differential Revision: https://reviews.llvm.org/D83820
2020-07-21 18:05:49 +00:00
Logan Smith fc24d1eadd [clang][NFC] Add missing 'override's 2020-07-20 16:43:24 -07:00
Yitzhak Mandelbaum bd994b81d3 Revert "[libTooling] In Clang Transformer, change `Metadata` field to deferred evalutaion"
This reverts commit c0b8954ecb.

The commit has broken various builds. Reverting while I investigate the cause.
2020-07-20 21:24:58 +00:00
Yitzhak Mandelbaum c0b8954ecb [libTooling] In Clang Transformer, change `Metadata` field to deferred evalutaion
`Metadata` is being changed from an `llvm::Any` to a `MatchConsumer<llvm;:Any>`, so that it's evaluation can be be dependent on `MatchResult`s passed in.

Reviewed By: ymandel, gribozavr2

Differential Revision: https://reviews.llvm.org/D83820
2020-07-20 21:17:09 +00:00
Sam McCall f0ab336e74 [Syntax] expose API for expansions overlapping a spelled token range.
Summary:
This allows efficiently accessing all expansions (without iterating over each
token and searching), and also identifying tokens within a range that are
affected by the preprocessor (which is how clangd will use it).

Subscribers: ilya-biryukov, kadircet, usaxena95, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D84009
2020-07-20 14:48:12 +02:00
Aleksandr Platonov d19f0666bc [clang][Tooling] Try to avoid file system access if there is no record for the file in compile_commads.json
Summary:
If there is no record in compile_commands.json, we try to find suitable record with `MatchTrie.findEquivalent()` call.
This is very expensive operation with a lot of `llvm::sys::fs::equivalent()` calls in some cases.

This patch disables file symlinks for performance reasons.

Example scenario without this patch:
- compile_commands.json generated at clangd build (contains ~3000 files).
- it tooks more than 1 second to get compile command for newly created file in the root folder of LLVM project.
- we wait for 1 second every time when clangd requests compile command for this file (at file change).

Reviewers: sammccall, kadircet, hokein

Reviewed By: sammccall

Subscribers: chandlerc, djasper, klimek, ilya-biryukov, kadircet, usaxena95, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D83621
2020-07-17 18:49:14 +02:00
Michael Liao b8409c03ed Fix `-Wreturn-type` warning. NFC. 2020-07-11 16:20:41 -04:00
Eduardo Caldas a474d5bae4 Use FileRange::text instead of Lexer::getSpelling
* as we are using them only for integer and floating literals they have
the same behavior
* FileRange::text is simpler to call and is within the context of
syntax trees
2020-07-10 16:21:12 +00:00
Eduardo Caldas 1db5b348c4 Add kinded UDL for raw literal operator and numeric literal operator template 2020-07-10 16:21:11 +00:00
Eduardo Caldas f33c2c27a8 Fix crash on `user defined literals`
Summary:
Given an UserDefinedLiteral `1.2_w`:
Problem: Lexer generates one Token for the literal, but ClangAST
references two source locations
Fix: Ignore the operator and interpret it as the underlying literal.
e.g.: `1.2_w` token generates syntax node IntegerLiteral(1.2_w)

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82157
2020-07-10 16:21:11 +00:00
Simon Pilgrim 397c68202a Fix MSVC "not all control paths return a value" warning. NFC. 2020-07-09 11:06:39 +01:00
Eduardo Caldas ea8bba7e8d Fix crash on overloaded postfix unary operators due to invalid sloc
Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82954
2020-07-08 14:09:40 +00:00
Dmitri Gribenko 7349479f22 RecursiveASTVisitor: don't call WalkUp unnecessarily in post-order traversal
Summary:
How does RecursiveASTVisitor call the WalkUp callback for expressions?

* In pre-order traversal mode, RecursiveASTVisitor calls the WalkUp
  callback from the default implementation of Traverse callbacks.

* In post-order traversal mode when we don't have a DataRecursionQueue,
  RecursiveASTVisitor also calls the WalkUp callback from the default
  implementation of Traverse callbacks.

* However, in post-order traversal mode when we have a DataRecursionQueue,
  RecursiveASTVisitor calls the WalkUp callback from PostVisitStmt.

As a result, when the user overrides the Traverse callback, in pre-order
traversal mode they never get the corresponding WalkUp callback. However
in the post-order traversal mode the WalkUp callback is invoked or not
depending on whether the data recursion optimization could be applied.

I had to adjust the implementation of TraverseCXXForRangeStmt in the
syntax tree builder to call the WalkUp method directly, as it was
relying on this behavior. There is an existing test for this
functionality and it prompted me to make this extra fix.

In addition, I had to fix the default implementation implementation of
RecursiveASTVisitor::TraverseSynOrSemInitListExpr to call WalkUpFrom in
the same manner as the implementation generated by the DEF_TRAVERSE_STMT
macro. Without this fix, the InitListExprIsPostOrderNoQueueVisitedTwice
test was failing because WalkUpFromInitListExpr was never called.

Reviewers: eduucaldas, ymandel

Reviewed By: eduucaldas, ymandel

Subscribers: gribozavr2, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82486
2020-07-06 13:38:01 +02:00
Dmitri Gribenko 19eaff650c Revert RecursiveASTVisitor fixes.
This reverts commit 8bf4c40af8.
This reverts commit 7b0be962d6.
This reverts commit 94454442c3.

Some compilers on some buildbots didn't accept the specialization of
is_same_method_impl in a non-namespace scope.
2020-07-03 13:48:24 +02:00
Dmitri Gribenko 94454442c3 RecursiveASTVisitor: don't call WalkUp unnecessarily in post-order traversal
Summary:
How does RecursiveASTVisitor call the WalkUp callback for expressions?

* In pre-order traversal mode, RecursiveASTVisitor calls the WalkUp
  callback from the default implementation of Traverse callbacks.

* In post-order traversal mode when we don't have a DataRecursionQueue,
  RecursiveASTVisitor also calls the WalkUp callback from the default
  implementation of Traverse callbacks.

* However, in post-order traversal mode when we have a DataRecursionQueue,
  RecursiveASTVisitor calls the WalkUp callback from PostVisitStmt.

As a result, when the user overrides the Traverse callback, in pre-order
traversal mode they never get the corresponding WalkUp callback. However
in the post-order traversal mode the WalkUp callback is invoked or not
depending on whether the data recursion optimization could be applied.

I had to adjust the implementation of TraverseCXXForRangeStmt in the
syntax tree builder to call the WalkUp method directly, as it was
relying on this behavior. There is an existing test for this
functionality and it prompted me to make this extra fix.

In addition, I had to fix the default implementation implementation of
RecursiveASTVisitor::TraverseSynOrSemInitListExpr to call WalkUpFrom in
the same manner as the implementation generated by the DEF_TRAVERSE_STMT
macro. Without this fix, the InitListExprIsPostOrderNoQueueVisitedTwice
test was failing because WalkUpFromInitListExpr was never called.

Reviewers: eduucaldas, ymandel

Reviewed By: eduucaldas, ymandel

Subscribers: gribozavr2, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82486
2020-07-03 13:03:19 +02:00
Eduardo Caldas fdbd78333f Add parenthesized expression to SyntaxTree
Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82960
2020-07-02 06:28:41 +00:00
Gabriel Matute ecfa0b2418 [libTooling] Fix `maybeExtendRange` to support `CharRange`s.
Currently, `maybeExtendRange` takes a `CharSourceRange`, but only works
correctly for the `TokenRange` case. This change adds proper support for the
`CharRange` case.

Reviewed By: gribozavr2

Differential Revision: https://reviews.llvm.org/D82901
2020-07-01 20:40:48 +00:00
Andy Soffer 9945bd5911 Add Metadata to Transformer tooling
This change adds a Metadata field to ASTEdit, Edit, and AtomicChange so that
edits can have associated metadata and that metadata can be constructed with
Transformer-based RewriteRules. Metadata is ignored when applying edits to
source, but other consumers of AtomicChange can use this metadata to direct how
they want to consume each edit.

Reviewed By: ymandel, gribozavr2

Differential Revision: https://reviews.llvm.org/D82226
2020-06-30 15:03:07 +00:00
Yitzhak Mandelbaum 30deabf89f [libTooling] Improve error message from failure in selection Stencil
This patch improves the error message provided by the stencil that handles
source from a range selector.

Reviewed By: gribozavr2

Differential Revision: https://reviews.llvm.org/D82654
2020-06-26 16:17:28 +00:00
Yitzhak Mandelbaum 056a539e57 [libTooling] Rename overloaded `range` range selector.
Renames the overloaded `RangeSelector` combinator `range` to the more
descriptive `enclose` and `encloseNodes`. The old overloads are left in place
and marked deprected and will be deleted at a future time.

Reviewed By: tdl-g

Differential Revision: https://reviews.llvm.org/D82592
2020-06-26 14:23:25 +00:00
Eduardo Caldas 7b404b6d00 Add `FloatingLiteral` to SyntaxTree
Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82318
2020-06-25 17:05:08 +00:00
Eduardo Caldas 466e8b7ea6 Add StringLiteral to SyntaxTree
Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82360
2020-06-25 17:05:08 +00:00
Eduardo Caldas 221d7bbe49 Add `CharLiteral` to SyntaxTree
Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82312
2020-06-25 17:05:08 +00:00
Eduardo Caldas 7f7f856480 Add `BoolLiteralExpression` to SyntaxTree
Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82310
2020-06-25 15:37:53 +00:00
Yitzhak Mandelbaum 87340a2bf1 [libTooling] Delete deprecated `Stencil` combinators.
Summary: Deletes `text()` and `selection()` combinators, since they have been deprecated for months.

Reviewers: tdl-g

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82225
2020-06-24 16:45:24 +00:00
Valentin Clement d90443b1d9 [openmp] Base of tablegen generated OpenMP common declaration
Summary:
As discussed previously when landing patch for OpenMP in Flang, the idea is
to share common part of the OpenMP declaration between the different Frontend.
While doing this it was thought that moving to tablegen instead of Macros will also
give a cleaner and more powerful way of generating these declaration.
This first part of a future series of patches is setting up the base .td file for
DirectiveLanguage as well as the OpenMP version of it. The base file is meant to
be used by other directive language such as OpenACC.
In this first patch, the Directive and Clause enums are generated with tablegen
instead of the macros on OMPConstants.h. The next pacth will extend this
to other enum and move the Flang frontend to use it.

Reviewers: jdoerfert, DavidTruby, fghanim, ABataev, jdenny, hfinkel, jhuber6, kiranchandramohan, kiranktp

Reviewed By: jdoerfert, jdenny

Subscribers: arphaman, martong, cfe-commits, mgorny, yaxunl, hiraditya, guansong, jfb, sstefan1, aaron.ballman, llvm-commits

Tags: #llvm, #openmp, #clang

Differential Revision: https://reviews.llvm.org/D81736
2020-06-23 10:32:32 -04:00
Valentin Clement 8383ac6197 Revert commit 9e52530 because of dependencies issue
This reverts commit 9e525309fb.
2020-06-22 09:56:14 -04:00
Valentin Clement 9e525309fb [openmp] Base of tablegen generated OpenMP common declaration
Summary:
As discussed previously when landing patch for OpenMP in Flang, the idea is
to share common part of the OpenMP declaration between the different Frontend.
While doing this it was thought that moving to tablegen instead of Macros will also
give a cleaner and more powerful way of generating these declaration.
This first part of a future series of patches is setting up the base .td file for
DirectiveLanguage as well as the OpenMP version of it. The base file is meant to
be used by other directive language such as OpenACC.
In this first patch, the Directive and Clause enums are generated with tablegen
instead of the macros on OMPConstants.h. The next pacth will extend this
to other enum and move the Flang frontend to use it.

Reviewers: jdoerfert, DavidTruby, fghanim, ABataev, jdenny, hfinkel, jhuber6, kiranchandramohan, kiranktp

Reviewed By: jdoerfert, jdenny

Subscribers: cfe-commits, mgorny, yaxunl, hiraditya, guansong, jfb, sstefan1, aaron.ballman, llvm-commits

Tags: #llvm, #openmp, #clang

Differential Revision: https://reviews.llvm.org/D81736
2020-06-22 09:34:53 -04:00
Yitzhak Mandelbaum d81d69f1c0 [libTooling] Change Transformer's `cat` to handle some cases of text in macros.
Summary:
Currently, `cat` validates range selections before extracting the corresponding
source text. However, this means that any range inside a macro is rejected as an
error. This patch changes the implementation to first try to map the range to
something reasonable. This makes the behavior consistent with handling of ranges
used for selecting portions of the source to edit.

Also updates a clang-tidy lit-test for one of the checks which was affected by
this change.

Reviewers: gribozavr2, tdl-g

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D82126
2020-06-19 18:48:54 +00:00
Yitzhak Mandelbaum 9ca50e887d [libTooling] Add parser for string representation of `RangeSelector`.
This patch adds a parser for a `RangeSelector` written as a string. The format
is closely based on the way one would right the selector in C++. This should
enable use of `RangeSelector`s from tools like clang-query and web UIs.
2020-06-19 01:11:29 +00:00
Nathan James 8b0df1c1a9
[NFC] Refactor Registry loops to range for 2020-06-19 00:40:10 +01:00
Eduardo Caldas 1b2f6b4a08 Add support for DeclRefExpr in SyntaxTree, by generating IdExpressions
Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: hlopko, gribozavr2, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81168
2020-06-18 21:05:23 +02:00
Alexandre Ganea 89ea0b0520 [MC] Pass down argv0 & cc1 cmd-line to the back-end and store in MCTargetOptions
When targetting CodeView, the goal is to store argv0 & cc1 cmd-line in the emitted .OBJ, in order to allow a reproducer from the .OBJ alone.

This patch is to simplify https://reviews.llvm.org/D80833
2020-06-18 09:17:14 -04:00
Eduardo Caldas 42f6fec387 Propose naming principle for NodeRole and apply it
Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81157
2020-06-04 20:08:35 +02:00
Eduardo Caldas 62305f6db4 Rename arrow -> arrowToken for unified naming
Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81155
2020-06-04 16:12:16 +02:00
Eduardo Caldas 3b739690b0 Add support for IntegerLiteral in SyntaxTree
Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81135
2020-06-04 14:05:31 +02:00
Eduardo Caldas 007098d7e6 Add support for `nullptr` in SyntaxTrees
Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81092
2020-06-03 15:52:33 +02:00
Dmitri Gribenko b34b7691fa Syntax tree: ignore implicit expressions at the top level of statements
Summary:
I changed `markStmtChild` to ignore implicit expressions the same way as
`markExprChild` does it already. The test that I modified crashes
without this change.

Reviewers: hlopko, eduucaldas

Reviewed By: hlopko, eduucaldas

Subscribers: gribozavr2, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D81019
2020-06-03 10:58:12 +02:00
Eduardo Caldas 3a574a6cb3 Add support for Overloaded Binary Operators in SyntaxTree
Reviewers: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D80812
2020-05-29 20:03:59 +02:00
Yitzhak Mandelbaum ce5780b88c [libTooling] Fix Transformer to work with ambient traversal kinds.
Summary:
`RewriteRule`'s `applyFirst` was brittle with respect to the default setting of the
`TraversalKind`. This patch builds awareness of traversal kinds directly into
rewrite rules so that they are insensitive to any changes in defaults.

Reviewers: steveire, gribozavr

Subscribers: hokein, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D80606
2020-05-28 11:42:07 -04:00
Eduardo Caldas 461af57de7 Add support for UnaryOperator in SyntaxTree
Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D80624
2020-05-27 17:12:46 +02:00
Yitzhak Mandelbaum ff2743bf04 [libTooling] In Transformer, allow atomic changes to span multiple files.
Summary:
Currently, all changes returned by a single application of a rule must fit in
one atomic change and therefore must apply to one file. However, there are
patterns in which a single rule will want to modify multiple files; for example,
a header and implementation to change a declaration and its definition. This
patch relaxes Transformer, libTooling's interpreter of RewriteRules, to support
multiple changes.

Reviewers: gribozavr

Subscribers: mgrang, jfb, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D80239
2020-05-26 09:17:35 -04:00
Eduardo Caldas 3785eb83af Add support for binary operators in Syntax Trees
Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D80540
2020-05-26 12:25:58 +02:00
Sam McCall 48aa781ae4 [Tooling] Drop leading/trailing whitespace from compile_flags.txt lines
Summary:
These files tend to be hand-authored, and people get very confused.
I can't think of any reason that such whitespace would be intended.

Reviewers: kadircet

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D80160
2020-05-19 11:20:02 +02:00
Arthur Eubanks dab1326a5a Strip -showIncludes in getClangStripDependencyFileAdjuster()
Summary:
Follow-up to https://reviews.llvm.org/D78836.

Also consolidate some test cases.

Reviewers: thakis

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D78939
2020-04-27 11:20:08 -07:00
Arthur Eubanks 8000d506af [clangd] Strip /showIncludes in clangd compile commands
In command lines with /showIncludes, clangd would output includes to stdout, breaking clients.

Summary: Fixes https://github.com/clangd/clangd/issues/322.

Reviewers: sammccall, kadircet

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D78836
2020-04-26 18:57:39 -07:00
Georgii Rymar 1647ff6e27 [ADT/STLExtras.h] - Add llvm::is_sorted wrapper and update callers.
It can be used to avoid passing the begin and end of a range.
This makes the code shorter and it is consistent with another
wrappers we already have.

Differential revision: https://reviews.llvm.org/D78016
2020-04-14 14:11:02 +03:00
Christopher Tetreault 2a1418f242 Fix syntax error 2020-04-09 13:46:23 -07:00
ADRA 6ed88afd78 [CUDA] Accept -x cu to indicate language is CUDA, transfer CUDA language flag to header-file arguments
Summary:
* accept -x cu to indicate language is CUDA
* transfer CUDA language flag to header-file arguments

Differential Revision: https://reviews.llvm.org/D77451
2020-04-09 13:08:41 -07:00
Yitzhak Mandelbaum 5e5d366718 [libTooling] Simplify the representation of Transformer's RewriteRules.
Summary:
This revision simplifies the representation of edits in rewrite rules. The
simplified form is more general, allowing the user more flexibility in building
custom edit specifications.

The changes extend the API, without changing the signature of existing
functions. So this only risks breaking users that directly accessed the
`RewriteRule` struct.

Reviewers: gribozavr2

Subscribers: jfb, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D77419
2020-04-08 08:45:41 -04:00
Johannes Doerfert f9d558c871 [OpenMP] "UnFix" layering problem with FrontendOpenMP
This reverts commit 97aa593a83 as it
causes problems (PR45453) https://reviews.llvm.org/D77574#1966321.

This additionally adds an explicit reference to FrontendOpenMP to
clang-tidy where ASTMatchers is used.

This is hopefully just a temporary solution. The dependence on
`FrontendOpenMP` from `ASTMatchers` should be handled by CMake
implicitly, not us explicitly.

Reviewed By: aheejin

Differential Revision: https://reviews.llvm.org/D77666
2020-04-07 14:41:18 -05:00
Marcel Hlopko 1bf055c989 [Syntax] Add mapping from spelled to expanded tokens for TokenBuffer
Summary:
Same restrictions apply as in the other direction: macro arguments are
not supported yet, only full macro expansions can be mapped.

Taking over from https://reviews.llvm.org/D72581.

Reviewers: gribozavr2, sammccall

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D77209
2020-04-07 15:07:16 +02:00
Sam McCall d66afd6dde [Syntax] Merge overlapping top-level macros in TokenBuffer
Summary:
Our previous definition of "top-level" was too informal, and didn't
allow for overlapping macros that each directly produce expanded tokens.
See D77507 for previous discussion.

Fixes http://bugs.llvm.org/show_bug.cgi?id=45428

Reviewers: kadircet, vabridgers

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D77615
2020-04-07 15:06:04 +02:00
Sam McCall ec0b990895 [Syntax] Simplify TokenCollector::Builder, use captured expansion bounds. NFC
Summary:
The motivation here is fixing https://bugs.llvm.org/show_bug.cgi?id=45428, see
D77507. The fundamental problem is that a "top-level" expansion wasn't precisely
defined. Repairing this concept means that TokenBuffer's "top-level expansion"
may not correspond to a single macro expansion. Example:

```
M(2); // expands to 1+2
```

The expansions overlap, but neither expansion alone yields all the tokens.
We need a TokenBuffer::Mapping that corresponds to their union.

This is fairly easy to fix in CollectPPExpansions, but the current design of
TokenCollector::Builder needs a fix too as it relies on the macro's expansion
range rather than the captured expansion bounds. This fix is hard to make due
to the way code is reused within Builder. And honestly, I found that code pretty
hard to reason about too.

The new approach doesn't use the expansion range, but only the expansion
location: it assumes an expansion is the contiguous set of expanded tokens with
the same expansion location, which seems like a reasonable formalization of
the "top-level" notion.

And hopefully the control flow is easier to follow too, it's considerably
shorter even with more documentation.

Reviewers: kadircet

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D77614
2020-04-07 15:01:34 +02:00
Johannes Doerfert 97aa593a83 [OpenMP] Fix layering problem with FrontendOpenMP
Summary:
ASTMatchers is used in various places and it now exposes the
LLVMFrontendOpenMP library to its users without them needing to depend
on it explicitly.

Reviewers: lebedev.ri

Subscribers: mgorny, yaxunl, bollu, guansong, martong, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D77574
2020-04-06 13:04:26 -05:00
Reid Kleckner 76221c734e Remove llvm::Error include form Diagnostic.h
Saves ~400 related LLVM ADT. llvm/ADT/Error.h takes 90ms to parse.

$ diff -u <(sort thedeps-before.txt) <(sort thedeps-after.txt) \
    | grep '^[-+] ' | sort | uniq -c | sort -nr
    403 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/Error.h
    403 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm-c/Error.h
    397 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/Format.h
    397 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/Debug.h
    377 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/ADT/StringExtras.h
    158 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm-c/ExternC.h
    138 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/ErrorOr.h
     13 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/raw_ostream.h
     13 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/ADT/SmallString.h
      5 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/ADT/Twine.h
2020-04-06 10:42:17 -07:00
Reid Kleckner b36c19bc4f [AST] Remove DeclCXX.h dep on ASTContext.h
Saves only 36 includes of ASTContext.h and related headers.

There are two deps on ASTContext.h:
- C++ method overrides iterator types (TinyPtrVector)
- getting LangOptions

For #1, duplicate the iterator type, which is
TinyPtrVector<>::const_iterator.

For #2, add an out-of-line accessor to get the language options. Getting
the ASTContext from a Decl is already an out of line method that loops
over the parent DeclContexts, so if it is ever performance critical, the
proper fix is to pass the context (or LangOpts) into the predicate in
question.

Other changes are just header fixups.
2020-04-06 10:09:01 -07:00
Johannes Doerfert 931c0cd713 [OpenMP][NFC] Move and simplify directive -> allowed clause mapping
Move the listing of allowed clauses per OpenMP directive to the new
macro file in `llvm/Frontend/OpenMP`. Also, use a single generic macro
that specifies the directive and one allowed clause explicitly instead
of a dedicated macro per directive.

We save 800 loc and boilerplate for all new directives/clauses with no
functional change. We also need to include the macro file only once and
not once per directive.

Depends on D77112.

Reviewed By: JonChesterfield

Differential Revision: https://reviews.llvm.org/D77113
2020-04-06 00:04:08 -05:00
Reid Kleckner ba1ffd25c1 [OpenMP][NFC] Remove the need to include `OpenMPClause.h`
See rational here: https://reviews.llvm.org/D76173#1922916
Time to compile Attr.h in isolation goes from 2.6s to 1.8s.

Original patch by Johannes, plus some additions from Reid to fix some
clang tooling targets.

Effect on transitive includes is marginal, though:

$ diff -u <(sort thedeps-before.txt) <(sort thedeps-after.txt) \
   | grep '^[-+] ' | sort | uniq -c | sort -nr
    104 -    /usr/local/google/home/rnk/llvm-project/clang/include/clang/AST/OpenMPClause.h
     87 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Frontend/OpenMP/OMPContext.h
     19 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/ADT/SmallSet.h
     19 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/ADT/SetVector.h
     14 -    /usr/include/c++/9/set
...

Differential Revision: https://reviews.llvm.org/D76184
2020-04-03 13:27:52 -07:00
Marcel Hlopko cdce2fe561 [Syntax] Remove delayed folding from tree building.
Summary:
This patch removes delayed folding and replaces it with forward peeking.

Delayed folding was previously used as a solution to the problem that
declaration doesn't have a representation in the AST. For example following
code:

```
int a,b;
```

is expressed in the AST as:

```
TranslationUnitDecl
|-...
|-VarDecl `int a`
`-VarDecl `int b`
```

And in the syntax tree we need:

```
*: TranslationUnit
`-SimpleDeclaration
  |-int
  |-SimpleDeclarator
  | `-a
  |-,
  |-SimpleDeclarator
  | `-b
  |-;
```

So in words, we need to create SimpleDeclaration to be a parent of
SimpleDeclarator nodes. Previously we used delayed folding to make sure SimpleDeclarations will be
eventually created. And in case multiple declarators requested declaration
creation, declaration range was extended to cover all declarators.

This design started to be hard to reason about, so we decided to replace it with
forward peeking. The last declarator node in the chain is responsible for creating
SimpleDeclaration for the whole chain. Range of the declaration corresponds to
the source range of the declarator node. Declarator decides whether its the last
one by peeking to the next AST node (see `isResponsibleForCreatingDeclaration`).

This patch does following:
* Removed delayed folding logic
* Tweaks Token.dumpForTests
* Moves getQualifiedNameStart inside BuildTreeVisitor
* Extracts BuildTreeVisitor.ProcessDeclaratorAndDeclaration
* Renames Builder.getDeclRange to Builder.getDeclarationRange and uses the
  method in all places.
* Adds a bunch of tests

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76922
2020-03-31 09:47:50 +02:00
Kadir Cetinkaya 9619c2cc9a
[clang][Syntax] Handle macro arguments in spelledForExpanded
Reviewers: sammccall

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D75446
2020-03-28 16:35:46 +01:00
Simon Pilgrim 896fa30fc0 Fix unused variable warning 2020-03-24 11:51:49 +00:00
Marcel Hlopko a711a3a460 [Syntax] Build mapping from AST to syntax tree nodes
Summary:
Copy of https://reviews.llvm.org/D72446, submitting with Ilya's permission.

Only used to assign roles to child nodes for now. This is more efficient
than doing range-based queries.

In the future, will be exposed in the public API of syntax trees.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76355
2020-03-23 16:22:14 +01:00
Marcel Hlopko 88bf9b3d26 [Syntax] Build template declaration nodes
Summary:
Rollforward of
https://reviews.llvm.org/rGdd12826808f9079e164b82e64b0697a077379241 after
temporarily adding -fno-delayed-template-parsing to the TreeTest.

Original summary:

> Copy of https://reviews.llvm.org/D72334, submitting with Ilya's permission.
>
> Handles template declaration of all kinds.
>
> Also builds template declaration nodes for specializations and explicit
> instantiations of classes.
>
> Some missing things will be addressed in the follow-up patches:
>
> * specializations of functions and variables,
> * template parameters.

Reviewers: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76418
2020-03-19 17:43:07 +01:00
Nico Weber 881f5b5a7b Revert "[Syntax] Build template declaration nodes"
This reverts commit dd12826808.
Breaks tests on Windows, see https://reviews.llvm.org/D76346#1929208
2020-03-18 12:57:55 -04:00
Marcel Hlopko dd12826808 [Syntax] Build template declaration nodes
Summary:
Copy of https://reviews.llvm.org/D72334, submitting with Ilya's permission.

Handles template declaration of all kinds.

Also builds template declaration nodes for specializations and explicit
instantiations of classes.

Some missing things will be addressed in the follow-up patches:

specializations of functions and variables,
template parameters.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76346
2020-03-18 16:16:59 +01:00
Marcel Hlopko 7d382dcd46 [Syntax] Build declarator nodes
Summary:
Copy of https://reviews.llvm.org/D72089 with Ilya's permission. See
https://reviews.llvm.org/D72089 for the first batch of comments.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D76220
2020-03-16 19:13:59 +01:00
Reid Kleckner d7c5037e6b Prune TargetInfo.h include from ParsedAttr.h, NFC
Saves ~400 includes of related headers:

$ diff -u <(sort thedeps-before.txt) <(sort thedeps-after.txt) \
    | grep '^[-+] ' | sort | uniq -c | sort -nr
    468 -    llvm-project/clang/include/clang/Basic/TargetInfo.h
    468 -    llvm-project/clang/include/clang/Basic/TargetCXXABI.h
    368 -    llvm-project/llvm/include/llvm/Support/CodeGen.h
    368 -    llvm-project/clang/include/clang/Basic/XRayInstr.h
    368 -    llvm-project/clang/include/clang/Basic/CodeGenOptions.h
    368 -    llvm-project/clang/include/clang/Basic/CodeGenOptions.def
    367 -    llvm-project/llvm/include/llvm/ADT/FloatingPointMode.h
    367 -    llvm-project/clang/include/clang/Basic/DebugInfoOptions.h
2020-03-11 20:47:11 -07:00
Reid Kleckner 213aea4c58 Remove unused Endian.h includes, NFC
Mainly avoids including Host.h everywhere:

$ diff -u <(sort thedeps-before.txt) <(sort thedeps-after.txt) \
    | grep '^[-+] ' | sort | uniq -c | sort -nr
   3141 - /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/Host.h
2020-03-11 15:45:34 -07:00
Reid Kleckner e08464fb45 Avoid including FileManager.h from SourceManager.h
Most clients of SourceManager.h need to do things like turning source
locations into file & line number pairs, but this doesn't require
bringing in FileManager.h and LLVM's FS headers.

The main code change here is to sink SM::createFileID into the cpp file.
I reason that this is not performance critical because it doesn't happen
on the diagnostic path, it happens along the paths of macro expansion
(could be hot) and new includes (less hot).

Saves some includes:
    309 -    /usr/local/google/home/rnk/llvm-project/clang/include/clang/Basic/FileManager.h
    272 -    /usr/local/google/home/rnk/llvm-project/clang/include/clang/Basic/FileSystemOptions.h
    271 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/VirtualFileSystem.h
    267 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/FileSystem.h
    266 -    /usr/local/google/home/rnk/llvm-project/llvm/include/llvm/Support/Chrono.h

Differential Revision: https://reviews.llvm.org/D75406
2020-03-11 13:53:12 -07:00
Kadir Cetinkaya cd9b2e18bd
[clang][Syntax] Add spelledTokenAt helper to TokenBuffer
Reviewers: sammccall

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D75503
2020-03-03 14:30:41 +01:00
Reid Kleckner 86565c1309 Avoid SourceManager.h include in RawCommentList.h, add missing incs
SourceManager.h includes FileManager.h, which is expensive due to
dependencies on LLVM FS headers.

Remove dead BeforeThanCompare specialization.

Sink ASTContext::addComment to cpp file.

This reduces the time to compile a file that does nothing but include
ASTContext.h from ~3.4s to ~2.8s for me.

Saves these includes:
    219 -    ../clang/include/clang/Basic/SourceManager.h
    204 -    ../clang/include/clang/Basic/FileSystemOptions.h
    204 -    ../clang/include/clang/Basic/FileManager.h
    165 -    ../llvm/include/llvm/Support/VirtualFileSystem.h
    164 -    ../llvm/include/llvm/Support/SourceMgr.h
    164 -    ../llvm/include/llvm/Support/SMLoc.h
    161 -    ../llvm/include/llvm/Support/Path.h
    141 -    ../llvm/include/llvm/ADT/BitVector.h
    128 -    ../llvm/include/llvm/Support/MemoryBuffer.h
    124 -    ../llvm/include/llvm/Support/FileSystem.h
    124 -    ../llvm/include/llvm/Support/Chrono.h
    124 -    .../MSVCSTL/include/stack
    122 -    ../llvm/include/llvm-c/Types.h
    122 -    ../llvm/include/llvm/Support/NativeFormatting.h
    122 -    ../llvm/include/llvm/Support/FormatProviders.h
    122 -    ../llvm/include/llvm/Support/CBindingWrapping.h
    122 -    .../MSVCSTL/include/xtimec.h
    122 -    .../MSVCSTL/include/ratio
    122 -    .../MSVCSTL/include/chrono
    121 -    ../llvm/include/llvm/Support/FormatVariadicDetails.h
    118 -    ../llvm/include/llvm/Support/MD5.h
    109 -    .../MSVCSTL/include/deque
    105 -    ../llvm/include/llvm/Support/Host.h
    105 -    ../llvm/include/llvm/Support/Endian.h

Reviewed By: aaron.ballman, hans

Differential Revision: https://reviews.llvm.org/D75279
2020-02-27 13:49:40 -08:00
Joe Turner b26c88e3c6 [clang-tidy] Store all ranges in clang::tooling::Diagnostic
Summary: Instead of dropping all the ranges associated with a Diagnostic when
converting them to a ClangTidy error, instead attach them to the ClangTidyError,
so they can be consumed by other APIs.

Patch by Joe Turner <joturner@google.com>.
Differential Revision: https://reviews.llvm.org/D69782
2020-02-27 19:39:42 +01:00
Yitzhak Mandelbaum 38b4516de8 [libTooling] Add function to determine associated text of a declaration.
Summary:
Second attempt -- the first was reverted in commit 0e480b39c6, because of test breakages. This revision fixes the cause of the test breakages.

Original description follows:
This patch adds `getAssociatedRange` which, for a given decl, computes preceding
and trailing text that would conceptually be associated with the decl by the
reader. This includes comments, whitespace, and separators like ';'.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D72153
2020-02-26 09:56:48 -05:00
Kadir Cetinkaya 8c2cf499e6
[clang][Tooling] Add a way to tokenize a FileRange
Reviewers: sammccall

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D74962
2020-02-26 13:50:41 +01:00
Kadir Cetinkaya da236f2350
Strip preceeding -Xclang when stripping -fcolor-diagnostics or -fdiagnostics-color
Summary: Fixes https://github.com/clangd/clangd/issues/279. We were removing the color options but not the preceeding -Xclang which causes errors since the -Xclang would now apply to the next option in the list of options. Now, when removing a color option, we check if there was a preceeding -Xclang and remove it as well.

Patch By @DaanDeMeyer !

Reviewers: sammccall, kadircet

Reviewed By: sammccall

Subscribers: ilya-biryukov, usaxena95

Differential Revision: https://reviews.llvm.org/D75019
2020-02-26 09:05:05 +01:00
Nico Weber 0e480b39c6 Revert "[libTooling] Add function to determine associated text of a declaration."
This reverts commit 9c54f6154f.
Breaks two tests on Windows, see e.g.
http://lab.llvm.org:8011/builders/clang-x64-windows-msvc/builds/14505/steps/stage%201%20check/logs/stdio
2020-02-25 19:21:52 -05:00
Yitzhak Mandelbaum 9c54f6154f [libTooling] Add function to determine associated text of a declaration.
Summary:
This patch adds `getAssociatedRange` which, for a given decl, computes preceding
and trailing text that would conceptually be associated with the decl by the
reader. This includes comments, whitespace, and separators like ';'.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D72153
2020-02-25 16:37:20 -05:00
Haojian Wu e6d0bad843 [clang-rename] Add the USR of incomplete decl to the USRSet.
Summary:
This fixes a clangd rename issue, which is missing the reference of
an incomplete specialization.

Unfortunately, I didn't reproduce this issue in clang-rename, I guess
the input `FoundDecl` of AdditionalUSRFinder is different in clangd vs
clang-rename, clang-rename uses the underlying CXXRecordDecl of the
ClassTemplateDecl, which is fixed in 5d862c042b;
while clangd-rename uses the ClassTemplateDecl.

Reviewers: kbobyrev

Reviewed By: kbobyrev

Subscribers: ilya-biryukov, jkorous, arphaman, kadircet, usaxena95, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D74829
2020-02-25 16:56:35 +01:00
Yitzhak Mandelbaum 523cae324d [libTooling] Add option for `buildAST` to report diagnostics.
Summary:
Currently, `buildAST[WithArgs]` either succeeds or fails.  This patch adds
support for the caller to pass a `DiagnosticConsumer` to receive all relevant
diagnostics.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D74763
2020-02-18 09:11:51 -05:00
Benjamin Kramer 5fc5c7db38 Strength reduce vectors into arrays. NFCI. 2020-02-17 15:37:35 +01:00
Alexandre Ganea 8404aeb56a [Support] On Windows, ensure hardware_concurrency() extends to all CPU sockets and all NUMA groups
The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, so that parallel computations such as LLD/ThinLTO can use all hardware threads in the system. Before this patch, on Windows, a maximum of 64 hardware threads could be used at most, in some cases dispatched only on one CPU socket.

== Background ==
Windows doesn't have a flat cpu_set_t like Linux. Instead, it projects hardware CPUs (or NUMA nodes) to applications through a concept of "processor groups". A "processor" is the smallest unit of execution on a CPU, that is, an hyper-thread if SMT is active; a core otherwise. There's a limit of 32-bit processors on older 32-bit versions of Windows, which later was raised to 64-processors with 64-bit versions of Windows. This limit comes from the affinity mask, which historically is represented by the sizeof(void*). Consequently, the concept of "processor groups" was introduced for dealing with systems with more than 64 hyper-threads.

By default, the Windows OS assigns only one "processor group" to each starting application, in a round-robin manner. If the application wants to use more processors, it needs to programmatically enable it, by assigning threads to other "processor groups". This also means that affinity cannot cross "processor group" boundaries; one can only specify a "preferred" group on start-up, but the application is free to allocate more groups if it wants to.

This creates a peculiar situation, where newer CPUs like the AMD EPYC 7702P (64-cores, 128-hyperthreads) are projected by the OS as two (2) "processor groups". This means that by default, an application can only use half of the cores. This situation could only get worse in the years to come, as dies with more cores will appear on the market.

== The problem ==
The heavyweight_hardware_concurrency() API was introduced so that only *one hardware thread per core* was used. Once that API returns, that original intention is lost, only the number of threads is retained. Consider a situation, on Windows, where the system has 2 CPU sockets, 18 cores each, each core having 2 hyper-threads, for a total of 72 hyper-threads. Both heavyweight_hardware_concurrency() and hardware_concurrency() currently return 36, because on Windows they are simply wrappers over std:🧵:hardware_concurrency() -- which can only return processors from the current "processor group".

== The changes in this patch ==
To solve this situation, we capture (and retain) the initial intention until the point of usage, through a new ThreadPoolStrategy class. The number of threads to use is deferred as late as possible, until the moment where the std::threads are created (ThreadPool in the case of ThinLTO).

When using hardware_concurrency(), setting ThreadCount to 0 now means to use all the possible hardware CPU (SMT) threads. Providing a ThreadCount above to the maximum number of threads will have no effect, the maximum will be used instead.
The heavyweight_hardware_concurrency() is similar to hardware_concurrency(), except that only one thread per hardware *core* will be used.

When LLVM_ENABLE_THREADS is OFF, the threading APIs will always return 1, to ensure any caller loops will be exercised at least once.

Differential Revision: https://reviews.llvm.org/D71775
2020-02-14 10:24:22 -05:00
Reid Kleckner cd62511496 Remove clang::ast_type_traits namespace in favor of clang
DynTypedNode and ASTNodeKind are implemented as part of the clang AST
library, which uses the main clang namespace. There doesn't seem to be a
need for this extra level of namespacing.

I left behind aliases in the ast_type_traits namespace for out of tree
clients of these APIs. To provide aliases for the enumerators, I used
this pattern:
  namespace ast_type_traits {
  constexpr TraversalKind TK_AsIs = ::clang::TK_AsIs;
  }
I think the typedefs will be useful for migration, but we might be able
to drop these enumerator aliases.

Reviewed By: aaron.ballman

Differential Revision: https://reviews.llvm.org/D74499
2020-02-13 10:46:47 -08:00
Justin Lebar 027eb71696 Use std::foo_t rather than std::foo in clang.
Summary: No functional change.

Reviewers: bkramer, MaskRay, martong, shafik

Subscribers: martong, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D74414
2020-02-11 10:37:08 -08:00
Haojian Wu a7fd548a4f [clang-rename] Fix the missing template constructors.
Summary:
When renaming a class with template constructors, we are missing the
occurrences of the template constructors, because getUSRsForDeclaration doesn't
give USRs of the templated constructors (they are not in the normal `ctors()`
method).

Reviewers: kbobyrev

Subscribers: jkorous, arphaman, kadircet, usaxena95, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D74216
2020-02-11 09:41:39 +01:00
Kirill Bobyrev 6bfc45cf60
Fix build after D72746
D72746 was missing a part of the patch before landing.
2020-02-06 08:55:19 +01:00
Benjamin Kramer adcd026838 Make llvm::StringRef to std::string conversions explicit.
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.

This doesn't actually modify StringRef yet, I'll do that in a follow-up.
2020-01-28 23:25:25 +01:00
Reid Kleckner 8a81daaa8b [AST] Split parent map traversal logic into ParentMapContext.h
The only part of ASTContext.h that requires most AST types to be
complete is the parent map. Nothing in Clang proper uses the ParentMap,
so split it out into its own class. Make ASTContext own the
ParentMapContext so there is still a one-to-one relationship.

After this change, 562 fewer files depend on ASTTypeTraits.h, and 66
fewer depend on TypeLoc.h:
  $ diff -u deps-before.txt deps-after.txt | \
    grep '^[-+] ' | sort | uniq -c | sort -nr | less
      562 -    ../clang/include/clang/AST/ASTTypeTraits.h
      340 +    ../clang/include/clang/AST/ParentMapContext.h
       66 -    ../clang/include/clang/AST/TypeLocNodes.def
       66 -    ../clang/include/clang/AST/TypeLoc.h
       15 -    ../clang/include/clang/AST/TemplateBase.h
  ...
I computed deps-before.txt and deps-after.txt with `ninja -t deps`.

This removes a common and key dependency on TemplateBase.h and
TypeLoc.h.

This also has the effect of breaking the ParentMap RecursiveASTVisitor
instantiation into its own file, which roughly halves the compilation
time of ASTContext.cpp (29.75s -> 17.66s). The new file takes 13.8s to
compile.

I left behind forwarding methods for getParents(), but clients will need
to include a new header to make them work:
  #include "clang/AST/ParentMapContext.h"

I noticed that this parent map functionality is unfortunately duplicated
in ParentMap.h, which only works for Stmt nodes.

Reviewed By: rsmith

Differential Revision: https://reviews.llvm.org/D71313
2020-01-24 13:42:28 -08:00
Michael Spencer 356a4b433b [clang][clang-scan-deps] Aggregate the full dependency information.
Differential Revision: https://reviews.llvm.org/D70268

This is a recommit of f978ea4983 with a fix for the PowerPC failure.

The issue was that:
* `CompilerInstance::ExecuteAction` calls
  `getTarget().adjust(getLangOpts());`.
* `PPCTargetInfo::adjust` changes `LangOptions::HasAltivec`.
* This happens after the first few calls to `getModuleHash`.

There’s even a FIXME saying:
```
  // FIXME: We shouldn't need to do this, the target should be immutable once
  // created. This complexity should be lifted elsewhere.
```

This only showed up on PowerPC because it's one of the few targets that
almost always changes a hashed langopt.

I looked into addressing the fixme, but that would be a much larger
change, and it's not the only thing that happens in `ExecuteAction` that
can change the module context hash. Instead I changed the code to not
call `getModuleHash` until after it has been modified in `ExecuteAction`.
2020-01-23 16:58:50 -08:00
Yitzhak Mandelbaum b9d2bf38e8 [libTooling] Fix bug in Stencil handling of macro ranges
Summary: Currently, an attempt to rewrite source code inside a macro expansion succeeds, but results in empty text, rather than failing with an error.  This patch restructures to the code to explicitly validate ranges before attempting to edit them.

Reviewers: gribozavr

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D72274
2020-01-17 12:11:25 -05:00
Michael Liao a3490e3e3d Remove trailing `;`. NFC. 2020-01-14 16:52:20 -05:00
Ilya Biryukov 013c07f697 [Syntax] Unset IsOriginal flag on nodes removed from the tree
And add a corresponding test.
Only nodes inside the TranslationUnit subtree can be marked as original,
computeReplacements() relies on this.
2020-01-14 17:00:33 +01:00
Ilya Biryukov 07a41018e9 [Syntax] Mark synthesized nodes as modifiable
This was an oversight in the original patch.
Also add corresponding tests.
2020-01-14 16:41:09 +01:00
Ilya Biryukov 3b929fe776 [Syntax] Assert invariants on tree structure and fix a bug in mutations
Add checks for some structural invariants when building and mutating
the syntax trees.

Fix a bug failing the invariants after mutations: the parent of nodes
added into the tree was null.
2020-01-14 16:31:08 +01:00
Ilya Biryukov 04f627f6b9 [Syntax] Build spanning SimpleDecalration for classes, structs, etc
When they are free-standing, e.g. `struct X;` or `struct X {};`.
Although this complicates the common case (of free-standing class
declarations), this ensures the less common case (e.g. `struct X {} a;`)
are handled uniformly and produce similar syntax trees.
2020-01-03 12:33:11 +01:00
Mark de Wever 8dc7b982b4 [NFC] Fixes -Wrange-loop-analysis warnings
This avoids new warnings due to D68912 adds -Wrange-loop-analysis to -Wall.

Differential Revision: https://reviews.llvm.org/D71857
2020-01-01 20:01:37 +01:00
Dmitri Gribenko 73f423e739 Parenthesized a logical expression.
This change fixes a warning on the ppc64be buildbot.
2019-12-20 14:36:21 +01:00
Dmitri Gribenko 9caf12e849 Use isa instead of dyn_cast 2019-12-20 14:36:21 +01:00
Kadir Cetinkaya c2377eae28
[clang][Tooling] Prefer -x over -std when interpolating
Summary:
Currently interpolation logic prefers -std over -x. But the latter is a
more strong signal, so this patch inverts the order and only makes use of -std
if -x didn't exist.

Fixes https://github.com/clangd/clangd/issues/185

Thanks @sammccall for tracking this down!

Reviewers: sammccall

Subscribers: ilya-biryukov, usaxena95, cfe-commits, sammccall

Tags: #clang

Differential Revision: https://reviews.llvm.org/D71727
2019-12-20 11:10:36 +01:00
Ilya Biryukov c1bbefef9d [Syntax] Use a hash table to search for tokens by their location
This is both more efficient and avoids corner cases in
`SourceManager::isBeforeInTranslationUnit`.

The change is trivial and clearly a performance improvement on the hot
path of building the syntax tree, so sending without review.
2019-12-18 12:24:00 +01:00
Ilya Biryukov 1ad15046dc [Syntax] Allow to mutate syntax trees
Summary:
This patch adds facilities to mutate the syntax trees and produce
corresponding text replacements.

The public interface of the syntax library now includes facilities to:
    1. perform type-safe modifications of syntax trees,
    2. compute textual replacements to apply the modifications,
    3. create syntax trees not backed by the source code.

For each of the three, we only add a few example transformations in this
patch to illustrate the idea, support for more kinds of nodes and
transformations will be done in follow-up patches.

The high-level mutation operations are implemented on top of operations
that allow to arbitrarily change the trees. They are considered to be
implementation details and are not available to the users of the
library.

Reviewers: sammccall, gribozavr2

Reviewed By: gribozavr2

Subscribers: merge_guards_bot, mgorny, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64573
2019-12-18 12:19:03 +01:00
Sam McCall 22f8125088 [Tooling/Syntax] Helpers to find spelled tokens touching a location.
Summary: Useful when positions are used to target nodes, with before/after ambiguity.

Reviewers: ilya-biryukov, kbobyrev

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D71356
2019-12-13 16:57:03 +01:00
Sam McCall 3f8da5d091 [Tooling/Syntax] Helpers to find spelled tokens touching a location.
Summary: Useful when positions are used to target nodes, with before/after ambiguity.

Reviewers: ilya-biryukov, kbobyrev

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D71356
2019-12-12 12:59:50 +01:00
Ilya Biryukov be14a22b47 [Syntax] Build nodes for simple cases of top level declarations
Summary:
More complicated nodes (e.g. template declarations) will be implemented
in the follow-up patches.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: merge_guards_bot, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D70856
2019-12-12 08:04:22 +01:00
Michael Spencer 5bcd34a03f Revert "[clang][clang-scan-deps] Aggregate the full dependency information."
This reverts commit f978ea4983.

It broke clang-ppc64be-linux, but not sure why yet.
2019-12-11 16:35:55 -08:00
Michael Spencer f978ea4983 [clang][clang-scan-deps] Aggregate the full dependency information.
Differential Revision: https://reviews.llvm.org/D70268
2019-12-11 14:40:51 -08:00
Sam McCall 966fac1941 [clang][Tooling] Fix potential UB in ExpandResponseFilesCompilationDatabase
Summary:
`vector::assign` will cause UB at here.

fixes: https://github.com/clangd/clangd/issues/223

Reviewers: kadircet, sammccall, hokein

Reviewed By: sammccall

Subscribers: merge_guards_bot, ilya-biryukov, usaxena95, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D71172
2019-12-09 12:24:23 +01:00
Reid Kleckner 60573ae6fe Remove Expr.h include from ASTContext.h, NFC
ASTContext.h is popular, prune its includes. Expr.h brings in Attr.h,
which is also expensive.

Move BlockVarCopyInit to Expr.h to accomplish this.
2019-12-06 15:30:49 -08:00
Kadir Cetinkaya b3b3778303
Reapply "[llvm][Support] Take in CurrentDirectory as a parameter in ExpandResponseFiles"
Attemps to fix windows buildbots.
2019-12-04 17:00:47 +01:00
Kadir Cetinkaya 0f12f9096e
Revert "[llvm][Support] Take in CurrentDirectory as a parameter in ExpandResponseFiles"
This reverts commit 75656005db.
2019-12-04 15:58:01 +01:00
Kadir Cetinkaya 75656005db
[llvm][Support] Take in CurrentDirectory as a parameter in ExpandResponseFiles
Summary:
This is a follow-up to D70769 and D70222, which allows propagation of
current directory down to ExpandResponseFiles for handling of relative paths.

Previously clients had to mutate FS to achieve that, which is not thread-safe
and can even be thread-hostile in the case of real file system.

Reviewers: sammccall

Subscribers: hiraditya, cfe-commits, llvm-commits

Tags: #clang, #llvm

Differential Revision: https://reviews.llvm.org/D70857
2019-12-04 15:13:12 +01:00
Kadir Cetinkaya 45ef055d4f
[clang][Tooling] Add support for .rsp files in compile_commands.json
Summary:
Add support for .rsp files.

Fixes https://github.com/clangd/clangd/issues/81

Patch By: liu hui(@lh123)

Reviewers: sammccall, ilya-biryukov, hokein, kadircet

Reviewed By: kadircet

Subscribers: merge_guards_bot, mgorny, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits

Tags: #clang-tools-extra, #clang

Differential Revision: https://reviews.llvm.org/D70222
2019-12-04 15:13:12 +01:00
Sam McCall c9c714c705 Reland [clangd] Rethink how SelectionTree deals with macros and #includes.
This reverts commit 905b002c13.

Avoid tricky (and invalid) comparator for std::set.
2019-12-03 17:53:43 +01:00
Sam McCall 905b002c13 Revert "[clangd] Rethink how SelectionTree deals with macros and #includes."
This reverts commit 19daa21f84.

It causes a bunch of failures on a bot that I've been unable to
reproduce so far:
http://45.33.8.238/mac/3308/step_7.txt
2019-11-29 19:59:02 +01:00
Sam McCall 19daa21f84 [clangd] Rethink how SelectionTree deals with macros and #includes.
Summary:
The exclusive-claim model is successful at resolving conflicts over tokens
between parent/child or siblings. However claims at the spelled-token
level do the wrong thing for macro expansions, where siblings can be
equally associated with the macro invocation.
Moreover, any model that only uses the endpoints in a range can fail when
a macro invocation occurs inside the node.

To address this, we use the existing TokenBuffer in more depth.
Claims are expressed in terms of expanded tokens, so there is no need to worry
about macros, includes etc.

Once we know which expanded tokens were claimed, they are mapped onto
spelled tokens for hit-testing.
This mapping is fairly flexible, currently the handling of macros is
pretty simple (map macro args onto spellings, other macro expansions onto the
macro name token).
This mapping is in principle token-by-token for correctness (though
there's some batching for performance).

The aggregation of the selection enum is now more principled as we need to be
able to aggregate several hit-test results together.

For simplicity i removed the ability to determine selectedness of TUDecl.
(That was originally implemented in 90a5bf92ff97b1, but doesn't seem to be very
important or worth the complexity any longer).

The expandedTokens(SourceLocation) helper could be added locally, but seems to
make sense on TokenBuffer.

Fixes https://github.com/clangd/clangd/issues/202
Fixes https://github.com/clangd/clangd/issues/126

Reviewers: hokein

Subscribers: MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits, ilya-biryukov

Tags: #clang

Differential Revision: https://reviews.llvm.org/D70512
2019-11-29 15:21:13 +01:00
Ilya Biryukov e702bdb859 [Syntax] Build SimpleDeclaration node that groups multiple declarators
Summary:
Also remove the temporary TopLevelDeclaration node and add
UnknownDeclaration to represent other unknown nodes.

See the follow-up change for building more top-level declarations.
Adding declarators is also pretty involved and will be done in another
follow-up patch.

Reviewers: gribozavr2

Reviewed By: gribozavr2

Subscribers: merge_guards_bot, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D70787
2019-11-29 11:39:45 +01:00
Ilya Biryukov 302cb3bc3d [Syntax] Add a comment explaining the pointer keys in std::map<Token*, ...>. NFC 2019-11-29 09:49:24 +01:00