For an user define `<`, `x < y` would yield the syntax tree:
```
BinaryOperatorExpression
|-IdExpression
| `-UnqualifiedId
| `-x
|-IdExpression
| `-UnqualifiedId
| `-<
`-IdExpression
`-UnqualifiedId
`-y
```
But there is no syntatic difference at call site between call site or
built-in `<`. As such they should generate the same syntax tree, namely:
```
BinaryOperatorExpression
|-IdExpression
| `-UnqualifiedId
| `-x
|-<
`-IdExpression
`-UnqualifiedId
`-y
```
Differential Revision: https://reviews.llvm.org/D85750
Summary:
We want NestedNameSpecifier syntax nodes to be generally supported, not
only for `DeclRefExpr` and `DependentScopedDeclRefExpr`.
To achieve this we:
* Use the `RecursiveASTVisitor`'s API to traverse
`NestedNameSpecifierLoc`s and automatically create its syntax nodes
* Add links from the `NestedNameSpecifierLoc`s to their syntax nodes.
In this way, from any semantic construct that has a `NestedNameSpecifier`,
we implicitly generate its syntax node via RAV and we can easily access
this syntax node via the links we added.
This is our grammar rule for nested-name-specifiers:
globalbal-specifier:
/*empty*/
simple-template-specifier:
template_opt simple-template-id
name-specifier:
global-specifier
decltype-specifier
identifier
simple-template-specifier
nested-name-specifier:
list(name-specifier, ::, non-empty, terminated)
It is a relaxed version of C++ [expr.prim.id] and quite simpler to map to our API.
TODO: refine name specifiers, `simple-template-name-specifier` and
decltype-name-specifier` are token soup for now.
This was the last binary operator that we supported but didn't have any
test coverage. The recent fix in a crash in member pointers allowed us
to add this test.
Differential Revision: https://reviews.llvm.org/D85185
Currently an Arena can only be built while consuming a TokenBuffer,
some users (like clangd) might want to share a TokenBuffer with multiple
compenents. This patch changes Arena's TokenBuffer member to be a reference so
that it can be created with read-only token buffers.
Differential Revision: https://reviews.llvm.org/D84973
Summary:
This allows efficiently accessing all expansions (without iterating over each
token and searching), and also identifying tokens within a range that are
affected by the preprocessor (which is how clangd will use it).
Subscribers: ilya-biryukov, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D84009
Summary:
Given an UserDefinedLiteral `1.2_w`:
Problem: Lexer generates one Token for the literal, but ClangAST
references two source locations
Fix: Ignore the operator and interpret it as the underlying literal.
e.g.: `1.2_w` token generates syntax node IntegerLiteral(1.2_w)
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D82157
Summary:
Previously, AST Matchers tests were using a custom way to run a test
with a specific C++ standard version. I'm migrating them to a shared
infrastructure to specify a Clang target from libClangTesting. I'm also
changing tests for AST Matchers to run in multiple language standards
versions, and under multiple triples that have different behavior with
regards to templates.
To keep the size of the patch manageable, in this patch I'm only
migrating one file to get the process started and get feedback on this
approach.
One caveat is that increasing the number of test configuration does
significantly increase the runtime of AST Matchers tests. On my machine,
the test runtime increases from 2.0 to 6.0s. I think it is worth the
improved test coverage.
Reviewers: jdoerfert, ymandel
Reviewed By: ymandel
Subscribers: gribozavr2, jfb, sstefan1, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D82179
Summary:
MSVC does not handle raw string literals with embedded double quotes
correctly. I switched the affected test case to use regular string
literals insetad.
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D82636
Summary:
The macro `INSTANTIATE_TEST_CASE_P` is defined as
```
\# define INSTANTIATE_TEST_CASE_P(prefix, test_case_name, generator, ...) \
...
```
If we build the test case with -werror, we will get an error like
```
error: ISO C++11 requires at least one argument for the "..." in a
variadic macro
testing::ValuesIn(TestClangConfig::allConfigs()));
^
```
This patch fixes that.
Reviewers: gribozavr, hlopko, eduucaldas, gribozavr2
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D81388
Summary:
The syntax tree test uses a helper function that executes all testing
assertions. When an assertion fails, the only line number that gets
printed to the log refers to the helper function. After this change, we
would also get the line number of the EXPECT_TRUE macro invocation
(unfortunately, the line number of the last token of it, not the first
one, but there's not much I can do about it).
Reviewers: hlopko, eduucaldas
Reviewed By: hlopko, eduucaldas
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D81107
Summary:
Doing so allows us to increase test coverage by removing unnecessary
language restrictions.
Reviewers: hlopko, eduucaldas
Reviewed By: hlopko, eduucaldas
Subscribers: gribozavr2, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D81040
Summary:
I changed `markStmtChild` to ignore implicit expressions the same way as
`markExprChild` does it already. The test that I modified crashes
without this change.
Reviewers: hlopko, eduucaldas
Reviewed By: hlopko, eduucaldas
Subscribers: gribozavr2, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D81019
Summary:
I think we would be better off with tests explicitly specifying the
language mode. Right now Lang_C means C99, but reads as "any C version",
or as "unspecified C version".
I also changed '-std=c++98' to '-std=c++03' because they are aliases (so
there is no difference in practice), because Clang implements C++03
rules in practice, and because 03 makes a nice sortable progression
between 03, 11, 14, 17, 20.
Reviewers: shafik, hlopko
Reviewed By: hlopko
Subscribers: jfb, martong, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D81000
Summary:
* Test if the code sourcing the SyntaxTree compiles
* Output compiler errors and warnings to err
* Fix tests with code that did not compile
Reviewers: gribozavr2
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D80731
Summary:
Same restrictions apply as in the other direction: macro arguments are
not supported yet, only full macro expansions can be mapped.
Taking over from https://reviews.llvm.org/D72581.
Reviewers: gribozavr2, sammccall
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D77209
Summary:
Our previous definition of "top-level" was too informal, and didn't
allow for overlapping macros that each directly produce expanded tokens.
See D77507 for previous discussion.
Fixes http://bugs.llvm.org/show_bug.cgi?id=45428
Reviewers: kadircet, vabridgers
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D77615
Summary:
This patch removes delayed folding and replaces it with forward peeking.
Delayed folding was previously used as a solution to the problem that
declaration doesn't have a representation in the AST. For example following
code:
```
int a,b;
```
is expressed in the AST as:
```
TranslationUnitDecl
|-...
|-VarDecl `int a`
`-VarDecl `int b`
```
And in the syntax tree we need:
```
*: TranslationUnit
`-SimpleDeclaration
|-int
|-SimpleDeclarator
| `-a
|-,
|-SimpleDeclarator
| `-b
|-;
```
So in words, we need to create SimpleDeclaration to be a parent of
SimpleDeclarator nodes. Previously we used delayed folding to make sure SimpleDeclarations will be
eventually created. And in case multiple declarators requested declaration
creation, declaration range was extended to cover all declarators.
This design started to be hard to reason about, so we decided to replace it with
forward peeking. The last declarator node in the chain is responsible for creating
SimpleDeclaration for the whole chain. Range of the declaration corresponds to
the source range of the declarator node. Declarator decides whether its the last
one by peeking to the next AST node (see `isResponsibleForCreatingDeclaration`).
This patch does following:
* Removed delayed folding logic
* Tweaks Token.dumpForTests
* Moves getQualifiedNameStart inside BuildTreeVisitor
* Extracts BuildTreeVisitor.ProcessDeclaratorAndDeclaration
* Renames Builder.getDeclRange to Builder.getDeclarationRange and uses the
method in all places.
* Adds a bunch of tests
Reviewers: gribozavr2
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76922
Summary:
This increases the coverage for things that differ between Linux and Windows, such as -fdelayed-template-parsing. This would have prevented the rollback of https://reviews.llvm.org/D76346.
While at it, update -std=c++11 to c++17 for the test.
Reviewers: gribozavr2
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76497
Summary:
This increases the coverage for things that differ between Linux and Windows, such as `-fdelayed-template-parsing`. This would have prevented the rollback of https://reviews.llvm.org/D76346.
While at it, update -std=c++11 to c++17 for the test.
Reviewers: gribozavr2
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76433
Summary:
This patch split Basic test into multple individual tests to allow simpler
filtering and clearer signal into what's broken when it's broken.
Reviewers: gribozavr2
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76366
Summary:
Rollforward of
https://reviews.llvm.org/rGdd12826808f9079e164b82e64b0697a077379241 after
temporarily adding -fno-delayed-template-parsing to the TreeTest.
Original summary:
> Copy of https://reviews.llvm.org/D72334, submitting with Ilya's permission.
>
> Handles template declaration of all kinds.
>
> Also builds template declaration nodes for specializations and explicit
> instantiations of classes.
>
> Some missing things will be addressed in the follow-up patches:
>
> * specializations of functions and variables,
> * template parameters.
Reviewers: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76418
Summary:
Copy of https://reviews.llvm.org/D72334, submitting with Ilya's permission.
Handles template declaration of all kinds.
Also builds template declaration nodes for specializations and explicit
instantiations of classes.
Some missing things will be addressed in the follow-up patches:
specializations of functions and variables,
template parameters.
Reviewers: gribozavr2
Reviewed By: gribozavr2
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D76346
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
When they are free-standing, e.g. `struct X;` or `struct X {};`.
Although this complicates the common case (of free-standing class
declarations), this ensures the less common case (e.g. `struct X {} a;`)
are handled uniformly and produce similar syntax trees.
This is both more efficient and avoids corner cases in
`SourceManager::isBeforeInTranslationUnit`.
The change is trivial and clearly a performance improvement on the hot
path of building the syntax tree, so sending without review.
Summary:
This patch adds facilities to mutate the syntax trees and produce
corresponding text replacements.
The public interface of the syntax library now includes facilities to:
1. perform type-safe modifications of syntax trees,
2. compute textual replacements to apply the modifications,
3. create syntax trees not backed by the source code.
For each of the three, we only add a few example transformations in this
patch to illustrate the idea, support for more kinds of nodes and
transformations will be done in follow-up patches.
The high-level mutation operations are implemented on top of operations
that allow to arbitrarily change the trees. They are considered to be
implementation details and are not available to the users of the
library.
Reviewers: sammccall, gribozavr2
Reviewed By: gribozavr2
Subscribers: merge_guards_bot, mgorny, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64573
Summary: Useful when positions are used to target nodes, with before/after ambiguity.
Reviewers: ilya-biryukov, kbobyrev
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D71356
Summary:
The exclusive-claim model is successful at resolving conflicts over tokens
between parent/child or siblings. However claims at the spelled-token
level do the wrong thing for macro expansions, where siblings can be
equally associated with the macro invocation.
Moreover, any model that only uses the endpoints in a range can fail when
a macro invocation occurs inside the node.
To address this, we use the existing TokenBuffer in more depth.
Claims are expressed in terms of expanded tokens, so there is no need to worry
about macros, includes etc.
Once we know which expanded tokens were claimed, they are mapped onto
spelled tokens for hit-testing.
This mapping is fairly flexible, currently the handling of macros is
pretty simple (map macro args onto spellings, other macro expansions onto the
macro name token).
This mapping is in principle token-by-token for correctness (though
there's some batching for performance).
The aggregation of the selection enum is now more principled as we need to be
able to aggregate several hit-test results together.
For simplicity i removed the ability to determine selectedness of TUDecl.
(That was originally implemented in 90a5bf92ff97b1, but doesn't seem to be very
important or worth the complexity any longer).
The expandedTokens(SourceLocation) helper could be added locally, but seems to
make sense on TokenBuffer.
Fixes https://github.com/clangd/clangd/issues/202
Fixes https://github.com/clangd/clangd/issues/126
Reviewers: hokein
Subscribers: MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits, ilya-biryukov
Tags: #clang
Differential Revision: https://reviews.llvm.org/D70512
Summary:
Also remove the temporary TopLevelDeclaration node and add
UnknownDeclaration to represent other unknown nodes.
See the follow-up change for building more top-level declarations.
Adding declarators is also pretty involved and will be done in another
follow-up patch.
Reviewers: gribozavr2
Reviewed By: gribozavr2
Subscribers: merge_guards_bot, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D70787
Summary:
Most of the statements mirror the ones provided by clang AST.
Major differences are:
- expressions are wrapped into 'ExpressionStatement' instead of being
a subclass of statement,
- semicolons are always consumed by the leaf expressions (return,
expression satement, etc),
- some clang statements are not handled yet, we wrap those into an
UnknownStatement class, which is not present in clang.
We also define an 'Expression' and 'UnknownExpression' classes in order
to produce 'ExpressionStatement' where needed. The actual implementation
of expressions is not yet ready, it will follow later.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D63835
Summary:
Returns the first token in every mapping where the token is an identifier.
This API is required to be able to highlight macro expansions in clangd.
Reviewers: hokein, ilya-biryukov
Subscribers: kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66470
llvm-svn: 369385
Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.
Differential revision: https://reviews.llvm.org/D66259
llvm-svn: 368942
Summary:
While useful as a sentinel value when iterating over tokens, having
'eof' in the tree, seems to do more harm than good.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: javed.absar, kristof.beyls, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64576
llvm-svn: 368062
Summary:
Preivously we would only discard it if we failed to parse parameter lists.
If we do not consume the body, parser sees tokens inside directive. In
turn, this leads to spurious diagnostics and a crash in TokenBuffer, see
the added tests.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D65517
llvm-svn: 367530
Summary:
A tooling-focused alternative to the AST. This commit focuses on the
memory-management strategy and the structure of the AST.
More to follow later:
- Operations to mutate the syntax trees and corresponding textual
replacements.
- Mapping between clang AST nodes and syntax tree nodes.
- More node types corresponding to the language constructs.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: llvm-commits, mgorny, cfe-commits
Tags: #clang, #llvm
Differential Revision: https://reviews.llvm.org/D61637
........
Fixes buildbots which were crashing on SyntaxTests.exe
llvm-svn: 365465
Summary:
A tooling-focused alternative to the AST. This commit focuses on the
memory-management strategy and the structure of the AST.
More to follow later:
- Operations to mutate the syntax trees and corresponding textual
replacements.
- Mapping between clang AST nodes and syntax tree nodes.
- More node types corresponding to the language constructs.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: llvm-commits, mgorny, cfe-commits
Tags: #clang, #llvm
Differential Revision: https://reviews.llvm.org/D61637
llvm-svn: 365355
Summary:
This change makes sure we have a single mapping for each macro expansion,
even if the result of expansion was empty.
To achieve that, we take information from PPCallbacks::MacroExpands into
account. Previously we relied only on source locations of expanded tokens.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D62953
llvm-svn: 364236
Summary: Used in clangd for a code tweak that expands a macro.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D62954
llvm-svn: 363698
To be more consistent with conventions used in the codebase. The new
name will be a better fit when more bits of the syntax library land.
llvm-svn: 361264
Summary:
TokenBuffer stores the list of tokens for a file obtained after
preprocessing. This is a base building block for syntax trees,
see [1] for the full proposal on syntax trees.
This commits also starts a new sub-library of ClangTooling, which
would be the home for the syntax trees and syntax-tree-based refactoring
utilities.
[1]: https://lists.llvm.org/pipermail/cfe-dev/2019-February/061414.html
Reviewers: gribozavr, sammccall
Reviewed By: sammccall
Subscribers: mgrang, riccibruno, Eugene.Zelenko, mgorny, jdoerfert, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D59887
llvm-svn: 361148
Eduardo Caldas