Summary:
This returns incremental highlights as a set of edits against the
previous highlights.
Server-side, we compute the full set of highlights, this just saves
wire-format size.
For now, the diff used is trivial: everything from the first change to
the last change is sent as a single edit.
The wire format is grungy - the replacement offset/length refer to
positions in the encoded array instead of the logical list of tokens.
We use token-oriented structs and translating to LSP forms when serializing.
This departs from LSP (but is consistent with semanticTokens today).
Tested in VSCode insiders (with a patched client to enable experimental
features).
Reviewers: hokein
Subscribers: ilya-biryukov, MaskRay, jkorous, mgrang, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D77225
Summary:
This ties to an LSP feature (diagnostic versioning) but really a lot
of the value is in being able to log what's happening with file versions
and queues more descriptively and clearly.
As such it's fairly invasive, for a logging patch :-\
Key decisions:
- at the LSP layer, we don't reqire the client to provide versions (LSP
makes it mandatory but we never enforced it). If not provided,
versions start at 0 and increment. DraftStore handles this.
- don't propagate magically using contexts, but rather manually:
addDocument -> ParseInputs -> (ParsedAST, Preamble, various callbacks)
Context-propagation would hide the versions from ClangdServer, which
would make producing good log messages hard
- within ClangdServer, treat versions as opaque and unordered.
std::string is a convenient type for this, and allows richer versions
for embedders. They're "mandatory" but "null" is a reasonable default.
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D75582
This has the same behavior as converting std::string_view to
std::string. This is an expensive conversion, so explicit conversions
are helpful for avoiding unneccessary string copies.
This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.
This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.
This doesn't actually modify StringRef yet, I'll do that in a follow-up.
Summary:
This reflects its current function better and avoids confusion with clang::DiagnosticConsumer.
The old name/constructor is left around temporarily for compatibility.
(Metagame: merging with out-of-tree changes is harder than usual this month)
Reviewers: hokein
Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D73346
Summary:
The historic behavior of TestTU is to gather diagnostics and otherwise ignore
them. So if a test has a syntax error, and doesn't assert diagnostics, it
silently misbehaves.
This can be annoying when developing tests, as evidenced by various tests
gaining "assert no diagnostics" where that's not really the point of the test.
This patch aims to make that default behavior. For the first error
(not warning), TestTU will call ADD_FAILURE().
This can be suppressed with a comment containing "error-ok". For now that will
suppress any errors in the TU. We can make this stricter later -verify style.
(-verify itself is hard to reuse because of DiagnosticConsumer interfaces...)
A magic-comment was chosen over a TestTU option because of table-driven tests.
In addition to the behavior change, this patch:
- adds //error-ok where we're knowingly testing invalid code
(e.g. for diagnostics, crash-resilience, or token-level tests)
- fixes a bunch of errors in the checked-in tests, mostly trivial (missing ;)
- removes a bunch of now-redundant instances of "assert no diagnostics"
Reviewers: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D73199
This reverts commit 7f93cb6228.
The assertion at RecursiveASTVisitor.h:1169 fails when passed a TypeLocNode.
Not sure if the correct fix is to use getTypeLocClass or something else.
Summary:
To keep the logic of finding locations of interesting AST nodes in one
place.
The advantage is better coverage of various AST nodes, both now and in
the future: as new nodes get added to `findExplicitReferences`, semantic
highlighting will automatically pick them up.
The drawback of this change is that we have to traverse declarations
inside our file twice in order to highlight dependent names, 'auto'
and 'decltype'. Hopefully, this should not affect the actual latency
too much, most time should be spent in building the AST and not
traversing it.
Reviewers: hokein
Reviewed By: hokein
Subscribers: nridge, merge_guards_bot, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D69673
Summary:
Editors are good at highlightings the keywords themselves.
Note that this only affects highlightings of builtin types spelled out
as keywords in the source code. Highlightings of typedefs to builtin
types are unchanged.
Reviewers: hokein
Reviewed By: hokein
Subscribers: merge_guards_bot, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D69431
The Windows triple currently turns on delayed template parsing, which
confuses several unit tests that use templates.
For now, just explicitly disable delayed template parsing. This isn't
ideal, but:
- the Windows triple will soon no longer use delayed template parsing
by default
- there's precedent for this in the clangd unit tests already
- let's get the clangd tests pass on Windows first before making
behavioral changes
Part of PR43592.
llvm-svn: 374718
Summary:
- store all macro references in the ParsedAST;
- unify the two variants of CollectMainFileMacros;
Reviewers: ilya-biryukov
Subscribers: MaskRay, jkorous, mgrang, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D67496
llvm-svn: 372725
Summary:
We still attempt to highlight them as underlying types, but fallback to
the generic 'typedef' highlighting kind if the underlying type is too
complicated.
Reviewers: hokein
Reviewed By: hokein
Subscribers: nridge, MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D67290
llvm-svn: 371402
Summary:
Template parameters were handled outside `addType`, this led to lack of highlightings for typedefs
to template types.
This was never desirable, we want to highlight our typedefs as their underlying type.
Note that typedefs to more complicated types, like pointers and references are still not highlighted.
Original patch by Johan Vikström.
Reviewers: hokein, jvikstrom
Reviewed By: hokein
Subscribers: nridge, javed.absar, kristof.beyls, MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66516
llvm-svn: 371379
Summary:
Instead of matching lists of highlightings, we annotate input code with
resulting highlightings and diff it against the expected annotated input.
In case of failures, this produces much nicer output in form of text-based
diffs.
Reviewers: hokein
Reviewed By: hokein
Subscribers: nridge, MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D67274
llvm-svn: 371373
Summary: Structured bindings are in a BindingDecl. The decl the declRefExpr points to are the BindingDecls. So this adds an additional if statement in the addToken function to highlight them.
Reviewers: hokein, ilya-biryukov
Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66738
llvm-svn: 370473
Summary: This would make the client life (tracking the changes) easier.
Reviewers: jvikstrom
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66541
llvm-svn: 369884
Summary:
Adds semantic highlighting for tokens that are a macro argument.
Example:
```
D_V(SomeVar);
```
The "SomeVar" inside the macro is highlighted as a variable now.
Tokens that are in a macro body expansion are ignored in this patch for three reasons.
* The spelling loc is inside the macro "definition" meaning it would highlight inside the macro definition (could probably easily be fixed by using getExpansionLoc instead of getSpellingLoc?)
* If wanting to highlight the macro definition this could create duplicate tokens. And if the tokens are of different types there would be conflicts (tokens in the same range but with different types). Say a macro defines some name and both a variable declaration and a function use this, there would be two tokens in the macro definition but one with Kind "Variable" and the other with Kind "Function".
* Thirdly, macro body expansions could come from a file that is not the main file (easily fixed, just check that the Loc is in the main file and not even a problem if we wanted to highlight the actual macro "invocation")
Reviewers: hokein, sammccall, ilya-biryukov
Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64741
llvm-svn: 369275
Summary: This means that function parameters are no longer highlighted as variable.other.cpp but instead as variable.parameter.cpp which is the more "correct" TextMate scope for them.
Reviewers: hokein, ilya-biryukov
Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66335
llvm-svn: 369238
Summary: Non type templates were not being highlighted. This highlights
them as TemplateParameters.
Reviewers: hokein, ilya-biryukov
Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66221
llvm-svn: 369090
Summary: It is possible to write include code from other files so that the decls from there do not become topLevelDecls (For example by including methods for a class). These Decls are not filtered by topLevelDecls and therefore SemanticHighlighting must manually check that every SLoc belongs in the main file. Otherwise there can be highlightings appearing in places where they should not.
Reviewers: hokein, ilya-biryukov
Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66083
llvm-svn: 368563
Summary:
Done in VisitDeclaratorDecl as the AutoTypeLoc is not deduced.
Scoped to only work for variables.
auto function return values need to be handled in a special way (separate patch for that).
auto's that are in lambdas ar enot highlighted as we don't highlight their underlying type (it's a RecordDecl, but the name is not an identifier so it returns in addToken).
Reviewers: hokein, ilya-biryukov
Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D65996
llvm-svn: 368546
Summary: Constructor initializers were not being highlighted. This adds highlighting for them by using TraverseConstructorInitializer. Uses the Traverse* because there is no visit for CXXCtorInitializer.
Reviewers: hokein, ilya-biryukov
Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66001
llvm-svn: 368434
Summary:
Clang generates function bodies and puts them in the AST for default methods if it is defaulted outside the class definition.
`
struct A {
A &operator=(A &&O);
};
A &A::operator=(A &&O) = default;
`
This will generate a function body for the `A &A::operator=(A &&O)` and put it in the AST. This body should not be visited if implicit code is not visited as it is implicit.
This was causing SemanticHighlighting in clangd to generate duplicate tokens and putting them in weird places.
Reviewers: hokein, ilya-biryukov, gribozavr
Subscribers: mgorny, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D65938
llvm-svn: 368402
Summary:
Adds a new HighlightingKind "Primitive". Adds a special case for TypeLocs that have an underlying TypePtr that is are builtin types, adding them as primitives.
The primary reason for this change is because otherwise typedefs that typedef primitives `typedef int A` would not get highlighted (so in the example `A` would not get any highlightings.)
Reviewers: hokein, ilya-biryukov
Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D65943
llvm-svn: 368291
Summary:
Conversion operators contain invalid MemberLocs which caused SemanticHighlighting
to emit a lot of error logs in large files as they can occur fairly
often (for example converting StringRef to std string).
As the only thing happening was a lot of error logs being
emited there doesn't really seem to be any way to test this
(no erroneous tokens are added). But emiting as many logs as
were being emited is not wanted.
This also adds a test to guard against regressions for highlightings
disapearing from places where the conversion operators are used as their
behaviour differ from the other CXXMethodDecls.
Reviewers: hokein, ilya-biryukov
Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D65928
llvm-svn: 368287
Summary: The parser gives implicit template instantiations to the action's HandleTopLevelDecls callback. This makes semantic highlighting highlight these templated functions multiple times. Fixed by filtering on if a Decl is an implicit function/variable/class instantiation. Also added a testcase to semantic highlighting on this.
Reviewers: hokein, ilya-biryukov
Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D65510
llvm-svn: 368261
Summary: Added a class for diffing highlightings and removing duplicate lines. Integrated into the highlighting generation flow. Only works correctly if all tokens are on a single line. Also returns empty lines if the IDE should remove previous highlightings on a line.
Reviewers: hokein, sammccall, ilya-biryukov
Subscribers: MaskRay, jkorous, mgrang, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64475
llvm-svn: 367521
Summary: Template parameters and specializations were not being highlighted before. This adds highlightings to those types of tokens by adding two Visit* methods.
Reviewers: hokein, sammccall, ilya-biryukov
Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64855
llvm-svn: 366420
Summary:
In `typedef int A` the `A` was not highlighted previously.
This patch gives `A` the same kind of highlighting that the underlying type has (class/enum) (which in this example is no special highlighting because builtins are not handled yet)
Will add highlightings for built ins in another patch.
Reviewers: hokein, sammccall, ilya-biryukov
Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64754
llvm-svn: 366207
Summary: The RecursiveASTVisitor sometimes visits exprs in initializer lists twice. Added deduplication to prevent duplicate highlighting tokens from appearing. Done by sorting and a linear search.
Reviewers: hokein, sammccall, ilya-biryukov
Subscribers: MaskRay, jkorous, mgrang, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64634
llvm-svn: 366070
Summary: VSCode does not have a scope for enum constants. So they were placed under "constant.other.enum" as that seems to be the most correct scope for enum constants. However, this makes theia color them blue (the same color it uses for keywords).
Reviewers: hokein, sammccall, ilya-biryukov
Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D64624
llvm-svn: 366045
Nathan Ridge