If the enum is a dependent type, we would crash somewhere in
getIntWidth(). -Wswitch diagnostic doesn't work on dependent enums
either.
Differential Revision: https://reviews.llvm.org/D92051
The plan is to use this to use this for .clang-format, .clang-tidy, and
compile_commands.json. (Currently the former two are reparsed every
time, and the latter is cached forever and changes are never seen).
Differential Revision: https://reviews.llvm.org/D88172
When type/function is defined in the middle of the file, previuosly we
would sometimes insert a "using" line before that definition, leading to
a compilation error. With this fix, we pick a point after such
definition in translation unit.
This is not a perfect solution. For example, it still doesn't handle
"using namespace" directives. It is, however, a significant improvement.
Differential Revision: https://reviews.llvm.org/D92053
Seeing an implicit this in the AST is pretty confusing I think.
While here, also mention when `this` is const.
Differential Revision: https://reviews.llvm.org/D91868
This was confusing, as testRoot on windows results in C:\\clangd-test
and testPath generated with posix explicitly still contained backslashes.
This patch ensures not only the relative part, but the whole final result
respects passed in Style.
Differential Revision: https://reviews.llvm.org/D91947
D71880 makes this dependency redundant and we can safely remove it. Tested for
both shared lib build and static lib build.
Reviewed By: hokein
Differential Revision: https://reviews.llvm.org/D91951
This patch introduces new canonicalization rules which are used for AST-based
rename in Clangd. By comparing two canonical declarations of inspected nodes,
Clangd determines whether both of them belong to the same entity user would
like to rename. Such functionality is relatively concise compared to the
Clang-Rename API that is used right now. It also helps to overcome the
limitations that Clang-Rename originally had and helps to eliminate several
classes of bugs.
Clangd AST-based rename currently relies on Clang-Rename which has design
limitations and also lacks some features. This patch breaks this dependency and
significantly reduces the amount of code to maintain (Clang-Rename is ~2000 LOC,
this patch is just <30 LOC of replacement code).
We eliminate technical debt by simultaneously
* Maintaining feature parity and ensuring no regressions
* Opening a straightforward path to improving existing rename bugs
* Making it possible to add more capabilities to rename feature which would not
be possible with Clang-Rename
Reviewed By: hokein
Differential Revision: https://reviews.llvm.org/D71880
Put project-aware-index between command-line specified static index and
ClangdServer indexes.
This also moves remote-index dependency from clangDaemon to ClangdMain
in an attempt to prevent cyclic dependency between clangDaemon and
remote-index-marshalling.
Differential Revision: https://reviews.llvm.org/D91860
An index implementation that can dispatch to a variety of indexes
depending on the file path. Enables clangd to work with multiple indexes in the
same instance, configured via config files.
Depends on D90749, D90746
Differential Revision: https://reviews.llvm.org/D90750
Compilation logic for External blocks. A few of the high level points:
- Requires exactly one-of File/Server at a time:
- Server is ignored in case of both, with a warning.
- Having none is an error, would render ExternalBlock void.
- Ensures mountpoint is an absolute path:
- Interprets it as relative to FragmentDirectory.
- Defaults to FragmentDirectory when empty.
- Marks Background as Skip.
Depends on D90748.
Differential Revision: https://reviews.llvm.org/D90749
Enable configuration of remote and static indexes through config files
in addition to command line arguments.
Differential Revision: https://reviews.llvm.org/D90748
Some of the buildbots were failing due to what seems to be them using a non c++14 compilant std::string implementation.
Since c++14 std::basic_string::append(const basic_string, size_t, size_t) has a defaulted 3rd paramater, but some of the build bots were reporting that it wasn't defaulted in their implementation.
First step of implementing clang-tidy configuration into clangd config.
This is just adding support for reading and verifying the clang tidy options from the config fragments.
No support is added for actually using the options within clang-tidy yet.
That will be added in a follow up as its a little more involved.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D90531
This is a mass-market version of the "dump AST" tweak we have behind
-hidden-features.
I think in this friendlier form it'll be useful for people outside clang
developers, which would justify making it a real feature.
It could be useful as a step towards lightweight clang-AST tooling in clangd
itself (like matcher-based search).
Advantages over the tweak:
- simplified information makes it more accessible, likely somewhat useful
without learning too much clang internals
- can be shown in a tree view
- structured information gives some options for presentation (e.g.
icon + two text colors + tooltip in vscode)
- clickable nodes jump to the corresponding code
Disadvantages:
- a bunch of code to handle different node types
- likely missing some important info vs dump-ast due to brevity/oversight
- may end up chasing/maintaining support for the long tail of nodes
Demo with VSCode support: https://imgur.com/a/6gKfyIV
Differential Revision: https://reviews.llvm.org/D89571
If there is a "-verify" flag in the compile command, clangd will crash
(hit the assertion) inside the `~VerifyDiagnosticConsumer` (Looks like our
compiler invocation doesn't setup correctly?).
This patch disables the verify mode as it is rarely useful in clangd.
Differential Revision: https://reviews.llvm.org/D91777
Address Sanitizer crashes on large allocations:
```c++
// Try to crash rather than hang on large allocation.
ScopedMemoryLimit MemLimit(1000 * 1024 * 1024); // 1GB
```
This adds a couple of missed tests from existing clang-rename ones and
introduces several new ones (e.g. static class member). This patch is required
to ensure feature parity for migration off Clang-Rename API D71880.
Reviewed By: hokein
Differential Revision: https://reviews.llvm.org/D91337
- Add verbose logging of payloads
- Add public logging of request summaries
- fix non-logging of messages in request scopes (oops!)
- add test for public/non-public logging, extending pipeline_helper a bit.
We've accumulated quite a lot of duplication in the request handlers by now.
I should factor that out, but not in this patch...
Differential Revision: https://reviews.llvm.org/D90654
Avoid requiring an actual MemoryBuffer in ComputePreambleBounds, when
a MemoryBufferRef will do just fine.
Differential Revision: https://reviews.llvm.org/D90890
Previously a corrupted index shard could cause us to resize arrays to an
arbitrary int32. This tends to be a huge number, and can render the
system unresponsive.
Instead, cap this at the amount of data that might reasonably be read
(e.g. the #bytes in the file). If the specified length is more than that,
assume the data is corrupt.
Differential Revision: https://reviews.llvm.org/D91258
This is a try to improve clangd-indexer tool performance:
- avoid processing already processed files.
- use different mutexes for different entities (e.g. do not block insertion of references while symbols are inserted)
Results for LLVM project indexing:
- before: ~30 minutes
- after: ~10 minutes
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D91051
https://reviews.llvm.org/D89670 changed the Ref structure, we need to
bump the version to invalidate all stored stale data, otherwise we will
get ` Error while reading shard: malformed or truncated refs` when
building the background index.
Differential Revision: https://reviews.llvm.org/D91131
* Even though remote index is still somewhat experimental, it can now be
used withing clangd itself: this should be the primary way of trying
it out
* Remove `protobuf-compiler` from list of needed Debian packages as it
`protobuf-compiler-grpc` already depends on it
* Bump recommended gRPC version to 1.32.3
We plan to eliminate error-prone and obsolete Clang-Rename API from Clangd. To
do that, we will introduce Decl canonicalization rules that will make renaming
code simpler and easier to maintain (D71880).
To ensure smooth transition to the new implementation, many Clang-Rename tests
will be adopted in Clangd to prevent any possible regressions. This patch is
the first in the chain of test migration patches. It improves existing tests
and adopts tests from Clang-Rename's alias, class and enum testing files.
Reviewed By: hokein
Differential Revision: https://reviews.llvm.org/D91102
With this patch, we reject the rename if the new name would conflict with
any other decls in the decl context of the renamed decl.
Differential Revision: https://reviews.llvm.org/D89790
There is not reason to check `std::make_unique<...>(..)` return value,
but `clangd::clang::loadIndex()` returns `nullptr` if an index file could not be loaded (e.g. incorrect version).
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D91049
If an enum has different names for the same constant, make sure only the first one declared gets added into the switch. Failing to do so results in a compiler error as 2 case labels can't represent the same value.
```
lang=c
enum Numbers{
One,
Un = One,
Two,
Deux = Two,
Three,
Trois = Three
};
// Old behaviour
switch (<Number>) {
case One:
case Un:
case Two:
case Duex:
case Three:
case Trois: break;
}
// New behaviour
switch (<Number>) {
case One:
case Two:
case Three: break;
}
```
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D90555
While here, clean up ParsedAST::build a bit:
- remove FIXMEs that were fixed long ago by ReplayPreamble
- remove redundant if, ClangTidyContext is not actually optional
Differential Revision: https://reviews.llvm.org/D90975
Probably not essential as afaik only one check uses this field. but still good to have consistent behaviour.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D90552
RemoteIndexClient implementations only depends on clangdSupport for
logging functionality and has no dependence on clangDeamon itself. This clears
out that link time dependency and enables depending on it in clangDeamon itself,
so that we can have other index implementations that makes use of the
RemoteIndex.
Differential Revision: https://reviews.llvm.org/D90746
This enables using the arrow operator to access members of the contained item.
```lang=c++
Located<std::string> X;
const char* CStr = X->c_str();
```
This is inline with how classes like `Optional` handle the arrow operator.
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D90682
This introduces a mechanism for providers to interpret paths specified
in a fragment either as absolute or relative to fragment location.
This information should be used during compile stage to handle blocks correctly.
Differential Revision: https://reviews.llvm.org/D90270
The index server has access to potentially-sensitive information, e.g. a
sequence of fuzzyFind requests reveals details about code completions in the
editor which in turn reveals details about the code being edited.
This information is necessary to provide the service, and our intention[1] is it
should never be retained beyond the scope of the request (e.g. not logged).
At the same time, some log messages should be exposed:
- server startup/index reloads etc that don't pertain to a user request
- basic request logs (method, latency, #results, error code) for monitoring
- errors while handling requests, without request-specific data
The -log=public design accommodates these by allowing three types of logs:
- those not associated with any user RPC request (via context-propagation)
- those explicitly tagged as [public] in the log line
- logging of format strings only, with no interpolated data (error level only)
[1] Specifically: Google is likely to run public instances of this server
for LLVM and potentially other projects, they will run in this configuration.
The details agreed in a Google-internal privacy review.
As clangd developers, we'd encourage others to use this configuration for public
instances too.
Differential Revision: https://reviews.llvm.org/D90526
- pass required=False to use_clang(), as we don't need it
- fix required=False (which was unused and rotted):
- make derived substitutions conditional on it
- add a feature so we can disable tests that need it
- conditionally disable our one test that depends on %resource_dir.
This doesn't seem right from first principles, but isn't a big deal.
Differential Revision: https://reviews.llvm.org/D90528
The vendor will be prefixed to the "clangd" and can be an arbitrary
string, so account for it in the test.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D90517
We were default initializing SymbolIDs before, which would leave
indeterminate values in underlying std::array.
This patch updates the underlying data initalization to be value-init and adds a
way to check for validness of a SymbolID.
Differential Revision: https://reviews.llvm.org/D90397
Introduce a separate thread that will kill `clangd-index-server` after 10 seconds regardless. This helps shut down the test if the server hangs and `stderr.readline()` does not contain inititalizatiton message. It prevents "necessary" waiting delay for the server warm-up and only introduces additional delay if the test fails.
It also makes use of `subprocess.Popen.kill()` which is a portable way of handling process shutdown and avoids using signals.
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D90590
Google test matcher `DeclKind` uses `NamedDecl::getDeclKindName()` to compare its result with expected declaration name.
Both, returned value of this function and the expected kind name argument have type `const char *`, so this matcher effectively
compares two pointers instead of the respective strings.
The test was passing on most platforms because compilers mostly were able to coalesce these string literals.
Patch By: Ilya Golovenko
Reviewed By: hokein
Differential Revision: https://reviews.llvm.org/D90384
With every incremental change, one needs to check-in new model upstream.
This also significantly increases the size of the git repo with every
new model.
Testing and comparing the old and previous model is also not possible as
we run only a single model at any point.
One solution is to have a "staging" decision forest which can be
injected into clangd without pushing it to upstream. Compare the
performance of the staging model with the live model. After a couple of
enhancements have been done to staging model, we can then replace the
live model upstream with the staging model. This reduces upstream churn
and also allows us to compare models with current baseline model.
This is done by having a callback in CodeCompleteOptions which is called
only when we want to use a decision forest ranking model. This allows us
to inject different completion model internally.
Differential Revision: https://reviews.llvm.org/D90014
This is an initial attempt to start using Syntax Trees in clangd while improving state of folding ranges feature and experimenting with Syntax Tree capabilities.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D88553
Both `SymbolKind` and `indexSymbolKindToSymbolKind` support constructors and
separate them into a different category from regular methods.
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D89935
Previous attempt (15f6bad6d7) introduced
add_dependencies but unfortunately it does not actually add a dependency
between RemoteIndexProto and RemoteIndexServiceProto. This is likely due
to some requirements of it that clang_add_library violates.
As a workaround, we will link RemoteIndexProto library to
RemoteIndexServiceProto which is logical because the library can not be
without linking to RemoteIndexProto anyway.
TestWorkspace allows easily writing tests involving multiple
files that can have inclusion relationships between them.
BackgroundIndexTest.RelationsMultiFile is refactored to use
TestWorkspace, and moved to FileIndexTest as it no longer
depends on BackgroundIndex.
Differential Revision: https://reviews.llvm.org/D89297
In `ReplayPreamble::replay`, use `getFileRef` instead of `getFile`, and
then use that `FileEntryRef` later to avoid needing
`FileEntryRef::FileEntryRef`. The latter is going to become private to
`FileManager` in a later commit.
We only need to version these messages if they actually diverge.
Unlike the service, the namespace name isn't part of the wire format.
clangd-index-server was broken by 81e5f298c4
as the namespace names weren't updated there, this fixes it (by adding
them for the service, and not requiring them elsewhere).
This allows it to have a separate namespace (grpc versioned service) without
putting versioning info on all of the other protos (before we need it).
clang-index-server is still broken (from 81e5f298c4).
Differential Revision: https://reviews.llvm.org/D90031
`llvm::sys::path` is used a lot in the remote index marshalling code. We can save space by avoiding spelling it out explicitly for most functions and times.
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D90016
This allows us to check whether enum field is actually sent over the wire or missing.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D89882
And also introduce Protobuf package versioning, it will help to deal
with breaking changes. Inroducing package version itself is a breaking
change, clients and servers need to be updated.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D89862
Nullability annotations are implmented using attributes; previusly
clangd would skip over AttributedTypeLoc since their location
points to the attribute instead of the modified type.
Also add some test cases for this.
Differential Revision: https://reviews.llvm.org/D89579
The patch adjusts the existing `llvm::DenseMap<unsigned, T>` and
`llvm::DenseSet<unsigned>` objects that store source locations, so
that they use `SourceLocation` directly instead of `unsigned`.
This patch relies on the `DenseMapInfo` trait added in D89719.
It also replaces the construction of `SourceLocation` objects from
the constants -1 and -2 with calls to the trait's methods `getEmptyKey`
and `getTombstoneKey` where appropriate.
Reviewed By: dexonsmith
Differential Revision: https://reviews.llvm.org/D69840
Without this patch 6 marshalling tests fail on Windows.
This patch contains the following changes:
- Allow paths with Windows slashes (convert to the POSIX style instead of assertion)
- Add support for URI with Windows path.
- Change the value of the second parameter of several `llvm::sys::path::convert_to_slash()` calls: we should use `windows` instead of `posix` to ensure UNIX slashes in the path.
- Port `RemoteMarshallingTest::IncludeHeaderURI` test to Windows.
Reviewed By: kbobyrev
Differential Revision: https://reviews.llvm.org/D89529
Update `Lexer` / `Lexer::Lexer` to use `MemoryBufferRef` instead of
`MemoryBuffer*`. Callers that were acquiring a `MemoryBuffer*` via
`SourceManager::getBuffer` were updated, such that if they checked
`Invalid` they use `getBufferOrNone` and otherwise `getBufferOrFake`.
Differential Revision: https://reviews.llvm.org/D89398
This was causing duplicate `symbols` components on the path as both the
edge from an index to filesymbols and filesymbols to symbolslabs were named
symbols.
Differential Revision: https://reviews.llvm.org/D89685
Performs a detailed profiling of clangd lsp server and conveys the
result to the client as a json object. It is of the form:
{
"_self": 0,
"_total": 8,
"child1": {
"_self": 4,
"_total": 4,
}
"child2": {
"_self": 2,
"_total": 4,
"child_deep": {
"_self": 2,
"_total": 2,
}
}
}
Differential Revision: https://reviews.llvm.org/D89277
Update clang-tools-extra, clang/tools, clang/unittests to migrate from
`SourceManager::getBuffer`, which returns an always dereferenceable
`MemoryBuffer*`, to `getBufferOrNone` or `getBufferOrFake`, both of
which return a `MemoryBufferRef`, depending on whether the call site was
checking for validity of the buffer. No functionality change intended.
Differential Revision: https://reviews.llvm.org/D89416
so that we could start experiment for C.
Previously, these flags in clangd were only meaningful for C++. We need
to flip them for C, this patch repurpose these flags.
- if true, just set it.
- if false, just respect the value in clang.
this would allow us to keep flags on for C++, and optionally flip them on for C.
Differential Revision: https://reviews.llvm.org/D89233
Given the following VarTemplateDecl AST,
```
VarTemplateDecl col:26 X
|-TemplateTypeParmDecl typename depth 0 index 0
`-VarDecl X 'bool' cinit
`-CXXBoolLiteralExpr 'bool' true
```
previously, we returned the VarDecl as the top-level decl, which was not
correct, the top-level decl should be VarTemplateDecl.
Differential Revision: https://reviews.llvm.org/D89098
This will enable queries like "clangd::" to find symbols under clangd
namespace, without requiring full "clang::clangd::" qualification.
Since Fuzzyfind performs the search under all scopes and only boosts the symbols
from relevant namespaces, we might get symbols from non-matching namespaces.
This patch chooses to drop those as they clearly do not match the query.
Fixes https://github.com/clangd/clangd/issues/550.
Differential Revision: https://reviews.llvm.org/D88814
This patch introduces hoisting detection logic into prepare state with
a partial AST traversal of the enclosing function.
We had some complaints from the users about this code action being almost always
available but failing most of the time. Hopefully this should reduce that noise.
The latency/correctness tradeoff is a bunch of hand-waving, but at least today
we don't have any other actions that are available on selection of statements,
so when we get to do the traversal it is quite likely that all the other checks
will bail out early. But this is still up for discussion, I am happy to abandon
the patch if you believe this is not practical.
Differential Revision: https://reviews.llvm.org/D85354
Unreachable file distances are represented as
`std::numeric_limits<unsigned>::max()`.
The previous dataset recorded the signals as `signed int` capturing this default
value as `-1`.
A new dataset was regenerated and a new model is trained that
interprets this unreachable as the intended value.
Distribution of `SymbolScopeDistance`:
Value Normalised Frequency
0 46.6184
4294967295 29.5342
6 14.5666
4 6.4433
2 1.4534
8 0.5760
10 0.3581
....
Distribution of `FileProximityDistance`:
Value Normalised Frequency
4294967295 39.9378
12 5.1997
14 4.9828
15 4.4221
16 4.3820
13 4.2765
17 3.8957
11 3.6387
19 3.4799
18 3.4076
....
Differential Revision: https://reviews.llvm.org/D89035
If the NewName is provided, prepareRename would perform a name
validation.
The motivation is to allow our internal embeder implement the customized
"canRenameInto" functionality on top of prepareRename.
Differential Revision: https://reviews.llvm.org/D88881
Up until now, we relied on matching the filename.
This depends on unstable details of libstdc++ and doesn't work well on other
stdlibs. Also we'd like to remove it (see D88204).
Differential Revision: https://reviews.llvm.org/D88885
The default value is 1.3f, but it was cast to true, which is not a good
base for code completion score.
Differential Revision: https://reviews.llvm.org/D88970
With this patch, we don't treat `using ns::X` as a first-class declaration like `using Z = ns::Y`, reference to X that goes through this using-decl is considered a direct reference (without the Underlying bit).
Fix the workaround in https://reviews.llvm.org/D87225 and https://reviews.llvm.org/D74054.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D88472
The protocol doesn't really incorporate ranking.
As with code completion, most clients respect what the server sends, but
VSCode re-ranks items, with predictable results.
See https://github.com/clangd/vscode-clangd/issues/81
There's no filterText field so we may be unable to construct a good workaround.
But expose the score so we may be able to do this on the client in future.
Differential Revision: https://reviews.llvm.org/D88844
several changes:
- return a structure result in rename API;
- prepareRename now returns more information (main-file occurrences);
- remove the duplicated detecting-touch-identifier code in prepareRename (which is implemented in rename API);
Differential Revision: https://reviews.llvm.org/D88634
It fixes the -Wswitch warning, though we mark it as a fix even if that is off.
This makes it the "recommended" action on an incomplete switch, which seems OK.
Differential Revision: https://reviews.llvm.org/D88726
Intent was a nice idea but it ends up being a bit awkward/heavyweight
without adding much.
In particular, it makes it hard to implement `CodeActionParams.only` properly
(there's an inheritance hierarchy for kinds).
Differential Revision: https://reviews.llvm.org/D88427
This allows us MSAN to instrument this function. Previous version is not
instrumentable due to it shear volume.
Differential Revision: https://reviews.llvm.org/D88536
This is a tool to simply parse a file as clangd would, and run some
common features (code actions, go-to-definition, hover) in an attempt to
trigger or reproduce crashes, error diagnostics, etc.
This is easier and more predictable than loading the file in clangd, because:
- there's no editor/plugin variation to worry about
- there's no accidental variation of user behavior or other extraneous requests
- we trigger features at every token, rather than guessing
- everything is synchronoous, logs are easier to reason about
- it's easier to (get users to) capture logs when running on the command-line
This is a fairly lightweight variant of this idea.
We could do a lot more with it, and maybe we should.
But I can't in the near future, and experience will tell us if we made
the right tradeoffs and if it's worth investing further.
Differential Revision: https://reviews.llvm.org/D88338
Relative paths received from the server are always in posix style. So
we need to ensure they are relative using that style, and not the native one.
Differential Revision: https://reviews.llvm.org/D88507
In preparation for making moving TweakFilter from ClangdServer::Options to
a ClangdLSPServer option, and letting it vary per-request.
(In order to implement CodeActionParams.only)
Also a general overdue cleanup.
Differential Revision: https://reviews.llvm.org/D88470
Extend the Trivial setter documentation to support cases where the value is moved into a field using `std::move`.
Reviewed By: sammccall, kadircet
Differential Revision: https://reviews.llvm.org/D88297
As @kadircet mentions in D84912#2184144, `findNearbyIdentifier()` traverses the whole file if there is no identifier for the word.
This patch ensures give up after 2^N lines in any case.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D87891
MSAN build times out for generated DecisionForest inference runtime.
A solution worth trying is splitting the function into 300 smaller
functions and then re-enable msan.
For now we are disabling instrumentation for the generated function.
Differential Revision: https://reviews.llvm.org/D88495
Improve the recently-added PopulateSwitch tweak to work on non-empty switches.
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D88434
Since we have 2 scoring functions (heuristics and decision forest),
renaming the existing evaluate() function to be more descriptive of the
Heuristics being evaluated in it.
Differential Revision: https://reviews.llvm.org/D88431
By default clangd will score a code completion item using heuristics model.
Scoring can be done by Decision Forest model by passing `--ranking_model=decision_forest` to
clangd.
Features omitted from the model:
- `NameMatch` is excluded because the final score must be multiplicative in `NameMatch` to allow rescoring by the editor.
- `NeedsFixIts` is excluded because the generating dataset that needs 'fixits' is non-trivial.
There are multiple ways (heuristics) to combine the above two features with the prediction of the DF:
- `NeedsFixIts` is used as is with a penalty of `0.5`.
Various alternatives of combining NameMatch `N` and Decision forest Prediction `P`
- N * scale(P, 0, 1): Linearly scale the output of model to range [0, 1]
- N * a^P:
- More natural: Prediction of each Decision Tree can be considered as a multiplicative boost (like NameMatch)
- Ordering is independent of the absolute value of P. Order of two items is proportional to `a^{difference in model prediction score}`. Higher `a` gives higher weightage to model output as compared to NameMatch score.
Baseline MRR = 0.619
MRR for various combinations:
N * P = 0.6346, advantage%=2.5768
N * 1.1^P = 0.6600, advantage%=6.6853
N * **1.2**^P = 0.6669, advantage%=**7.8005**
N * **1.3**^P = 0.6668, advantage%=**7.7795**
N * **1.4**^P = 0.6659, advantage%=**7.6270**
N * 1.5^P = 0.6646, advantage%=7.4200
N * 1.6^P = 0.6636, advantage%=7.2671
N * 1.7^P = 0.6629, advantage%=7.1450
N * 2^P = 0.6612, advantage%=6.8673
N * 2.5^P = 0.6598, advantage%=6.6491
N * 3^P = 0.6590, advantage%=6.5242
N * scaled[0, 1] = 0.6465, advantage%=4.5054
Differential Revision: https://reviews.llvm.org/D88281
Replaces the dummy CodeCompletion model with a trained DecisionForest
model.
The features.json needs to be manually curated specifying the features
to be used. This is a one-time cost and does not change if the model
changes until we decide to add/remove features.
Differential Revision: https://reviews.llvm.org/D88071
Add a tweak that populates an empty switch statement of an enumeration type with all of the enumerators of that type.
Before:
```
enum Color { RED, GREEN, BLUE };
void f(Color color) {
switch (color) {}
}
```
After:
```
enum Color { RED, GREEN, BLUE };
void f(Color color) {
switch (color) {
case RED:
case GREEN:
case BLUE:
break;
}
}
```
Reviewed By: sammccall
Differential Revision: https://reviews.llvm.org/D88383
Translating between JSON objects and C++ strutctures is common.
From experience in clangd, fromJSON/ObjectMapper work well and save a lot of
code, but aren't adopted elsewhere at least partly due to total lack of error
reporting beyond "ok"/"bad".
The recently-added error model should be rich enough for most applications.
It requires tracking the path within the root object and reporting local
errors at appropriate places.
To do this, we exploit the fact that the call graph of recursive
parse functions mirror the structure of the JSON itself.
The current path is represented as a linked list of segments, each of which is
on the stack as a parameter. Concretely, fromJSON now looks like:
bool fromJSON(const Value&, T&, Path);
Beyond the signature change, this is reasonably unobtrusive: building
the path segments is mostly handled by ObjectMapper and the vector<T> fromJSON.
However the root caller of fromJSON must now create a Root object to
store the errors, which is a little clunky.
I've added high-level parse<T>(StringRef) -> Expected<T>, but it's not
general enough to be the primary interface I think (at least, not usable in
clangd).
All existing users (mostly just clangd) are updated in this patch,
making this change backwards-compatible is a bit hairy.
Differential Revision: https://reviews.llvm.org/D88103
Current implementation of heuristic-based scoring function also contains
computation of derived signals (e.g. whether name contains a word from
context, computing file distances, scope distances.)
This is an attempt to separate out the logic for computation of derived
signals from the scoring function.
This will allow us to have a clean API for scoring functions that will
take only concrete code completion signals as input.
Differential Revision: https://reviews.llvm.org/D88146
We intend to replace heuristics based code completion ranking with a Decision Forest Model.
This patch introduces a format for representing the model and an inference runtime that is code-generated at build time.
- Forest.json contains all the trees as an array of trees.
- Features.json describes the features to be used.
- Codegen file takes the above two files and generates CompletionModel containing Feature struct and corresponding Evaluate function.
The Evaluate function maps a feature to a real number describing the relevance of this candidate.
- The codegen is part of build system and these files are generated at build time.
- Proposes a way to test the generated runtime using a test model.
- Replicates the model structure in unittests.
- unittest tests both the test model (for correct tree traversal) and the real model (for sanity).
This reverts commit 549e55b3d5.
Summary:
[WIP]
- Proposes a json format for representing Random Forest model.
- Proposes a way to test the generated runtime using a test model.
TODO:
- Add generated source code snippet for easier review.
- Fix unused label warning.
- Figure out required using declarations for CATEGORICAL columns from Features.json.
- Necessary Google3 internal modifications for blaze before landing.
- Add documentation for format of the model.
- Document more.
Subscribers: mgorny, ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D83814
For style guides forbid "using" declarations for namespaces like "std".
With this new config option, AddUsing can be selectively disabled on
those.
Differential Revision: https://reviews.llvm.org/D87775
This fixes a bug in dbf486c0de, which
introduced the Index section of the config, but did not register the
parse method, so it didn't work in a YAML file (but did in a test).
Differential Revision: https://reviews.llvm.org/D87710
This patch adds a mechanism to load new versions of index into
clangd-index-server using SwapIndex and FileStatus information about last
modification time without downtime.
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D87450
Without this patch `clangd` crashes at try to load compressed string table when `zlib` is not available.
Example:
- Build `clangd` with MinGW (`zlib` found)
- Build index
- Build `clangd` with Visual Studio compiler (`zlib` not found)
- Try to load index
Reviewed By: sammccall, adamcz
Differential Revision: https://reviews.llvm.org/D87673
Summary:
This is considerably terser than the makeStringError and friends, and
avoids verbosity cliffs that discourage adding log information.
It follows the syntax used in log/elog/vlog/dlog that have been successful.
The main caveats are:
- it's strictly out-of-place in logger.h, though kind of fits thematically and
in implementation
- it claims the "error" identifier, which seems a bit too opinionated
to put higher up in llvm
I've updated some users of StringError mostly at random - there are lots
more mechanical changes but I'd like to get this reviewed before making
them all.
Reviewers: kbobyrev, hokein
Subscribers: mgorny, ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D83419
We guess the style based on the existing using declarations. If there
are any and they all start with ::, we add it to the newly added one
too.
Differential Revision: https://reviews.llvm.org/D86473
This can happen when building implicit modules, as demonstrated in test.
The CompilerInstance uses the same StoredDiags, but different
SourceManager. This used to crash clangd when it tried to relocate the
diagnostic to the main file, which, according to SourceManager from the
diagnostic, is a fake <module-includes> file.
Differential Revision: https://reviews.llvm.org/D85753
Now that Clang is able to constant-evaluate void-typed expressions,
disable showing hover-card values for them. It's not useful to say that
an expression cast to void has value '<no value>', even if we can
constant-evaluate it to that result!
Currently, clangd crashes when opening a file with `#pragma clang __debug parser_crash` (e.g. clang/test/Modules/Inputs/crash.h).
This patch disables these crashes.
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D86279
This addresses a FIXME in ASTReader.
Modules were already re-exported for Preprocessor, but not for Sema.
The result was that, with -fmodules-local-submodule-visibility, all AST
nodes belonging to a module that was loaded in a premable where not
accesible from the main part of the file and a diagnostic recommending
importing those modules would be generated.
Differential Revision: https://reviews.llvm.org/D86069
When preamble contains #undef, indexing code finds the matching #define
and uses that during indexing. However, it would only look for local
definitions. If the macro was defined in a module, MacroInfo
would be nullptr and clangd would crash.
This change makes clangd ignore any #undef without a matching #define
inside the same TU.
The indexing of macros happens for preamble only, so then #undef must be
in the preamble, which is why we need two .h files in a test.
Note that clangd is currently not ready for module support, but this
brings us one step closer.
This was previously attempted in
4061d9e42c, but had to be reverted due to
broken test. This version fixes that test-only bug by setting a custom module
cache path to avoid re-use of modules across test invocations.
Differential Revision: https://reviews.llvm.org/D85923
Summary: This will be needed to support call hierarchy
Reviewers: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D83536
Summary:
This enables sharing the logic between standalone clangd and embedders
of it. The new approach should be same performance-wise, as it is only called
once per addDocument call.
This patch also introduces a blacklisting code path for disabling crashy or
high-noise tests, until we figure out a way to make them work with clangd-setup.
The biggest difference is the way we make use of preambles, hence those checks
can't see directives coming from the preamble section of the file. The second
thing is the fact that code might-not be compiling while clangd is trying to
build an AST, hence some checks might choke on those incomplete ASTs.
Reviewers: sammccall
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, aaron.ballman, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D83224
Summary:
Some clang-tidy checkers, e.g. llvm-include-order can emit diagnostics
at this callback (as mentioned in the comments).
Clangd was resetting diag consumer to IgnoreDiags before sending EOF, hence we
were unable to emit diagnostics for such checkers.
This patch changes the order of that reset and preprocosser event to make sure
we emit that diag.
Fixes https://github.com/clangd/clangd/issues/314.
Reviewers: sammccall
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D83178
I've dropped the background context parameter, since we in practice just pass the
current context there, and we now have a different way to specify context too.
While here, clean up a couple of comments.
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D83157
This reverts commit 4061d9e42c.
Tests are failing in some configuration, likely due to not cleaning up
module cache path before running the test.
Differential Revision: https://reviews.llvm.org/D85907
This lets basic AST-consuming actions be defined outside clangdserver.
(it essentially exposes TUScheduler::runWithAST).
Two use cases for now:
- replace ClangdServer::dumpAST, which doesn't really belong in the public
interface (a followup patch will do this)
- allow embedders to add/experiment with extra features
(e.g. I know some who want to try crazy AST+ML code completion...)
Maybe in the future we should use this sort of mechanism to make ClangdServer
less of a monolith, but that's not in scope for now.
This would probably require a richer and more carefully-thought-out API.
Differential Revision: https://reviews.llvm.org/D85883
When running dexp in remote mode without --project-root it shuts down
with an assertion. This is not the desired behaviour: instruct user on
how to run it properly when the configuration is incorrect.
Summary:
When preamble contains #undef, indexing code finds the matching #define
and uses that during indexing. However, it would only look for local
definitions. If the macro was defined in a module, MacroInfo
would be nullptr and clangd would crash.
This change makes clangd ignore any #undef without a matching #define
inside the same TU.
The indexing of macros happens for preamble only, so then #undef must be
in the preamble, which is why we need two .h files in a test.
Note that clangd is currently not ready for module support, but this
brings us one step closer.
Reviewers: sammccall
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D80525
This fixes a crash bug in clangd when used with modules. ASTWriter would
end up writing references to submodules into the PCH file, but upon
reading the submodules would not exists and
HeaderFileInfoTrait::ReadData would crash.
Differential Revision: https://reviews.llvm.org/D85532
Inside clangd, clang-tidy checks don't see preprocessor events in the preamble.
This leads to `Token::PtrData == nullptr` for tokens that the macro is defined to.
E.g. `#define SIGTERM 15`:
- Token::Kind == tok::numeric_constant (Token::isLiteral() == true)
- Token::UintData == 2
- Token::PtrData == nullptr
As the result of this, bugprone-bad-signal-to-kill-thread check crashes at null-dereference inside clangd.
Reviewed By: hokein
Differential Revision: https://reviews.llvm.org/D85417
It is necessary to traverse children of unnamed declaration contexts
to get symbols which are currently missing in document outline, e.g.:
extern "C" {
void foo();
}
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D84839
This is a refactoring: errors should be logged only on the highest level.
Switch from Optional to Expected in the serialization code.
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D84939
This is the last missing bit in the core remote index implementation. The only
remaining bits are some API refactorings (replacing Optional with Expected and
being better at reporting errors).
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D84894
Without this patch the word occurrence search always returns the first token of the file.
Despite of that, `findNeardyIdentifier()` returns the correct result (but inefficently) until there are several matched tokens with the same value `floor(log2(<token line> - <word line>))` (e.g. several matched tokens on the same line).
Reviewed By: kadircet
Differential Revision: https://reviews.llvm.org/D84912
The assertion is not true anymore after D82739, this patch just removes
it, and rename related functions.
And also fixes a missing cases.
Differential Revision: https://reviews.llvm.org/D84837
Function void run() on line 286 overrides a virtual function on line 92 of
clang-tools-extra/clangd/index/dex/dexp/Dexp.cpp. Not marking it override will
cause a build failure when we use -Werror (every warning is treated as an error).
Reviewed By: kbobyrev (Kirill Bobyrev)
Differential Revision: https://reviews.llvm.org/D84794
This cleans up several CMakeLists.txt's where -Wno-suggest-override was manually specified. These test targets now inherit this flag from the gtest target.
Some unittests CMakeLists.txt's, in particular Flang and LLDB, are not touched by this patch. Flang manually adds the gtest sources itself in some configurations, rather than linking to LLVM's gtest target, so this fix would be insufficient to cover those cases. Similarly, LLDB has subdirectories that manually add the gtest headers to their include path without linking to the gtest target, so those subdirectories still need -Wno-suggest-override to be manually specified to compile without warnings.
Differential Revision: https://reviews.llvm.org/D84554
Summary:
Without this patch clangd does not collect references for main-file symbols if there is no public declaration in preamble.
Example:
`test1.c`
```
void f1() {}
```
`test2.c`
```
extern void f1();
void f2() {
f^1();
}
```
`Find all references` does not show definition of f1() in the result, but GTD works OK.
Reviewers: sammccall, kadircet
Reviewed By: kadircet
Subscribers: ilya-golovenko, ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D84513
Summary:
Only FuzzyFindRequest is implemented via Marshaller even though other requests
also follow a similar pattern. Unify them under the marshalling umbrella and
make the server requests even more uniform to complement D84499.
Reviewers: kadircet
Reviewed By: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits, sammccall
Tags: #clang
Differential Revision: https://reviews.llvm.org/D84525
Summary:
When dereferencing Optional's it makes sense to use ASSERT_TRUE for better
test failures readability. Switch from EXPECT_TRUE to ASSERT_TRUE where
it is appropriate.
Reviewers: kadircet
Reviewed By: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D84535
Signed-off-by: Kirill Bobyrev <kbobyrev@google.com>
add_compile_options is more sensitive to its location in the file than add_definitions--it only takes effect for sources that are added after it. This updated patch ensures that the add_compile_options is done before adding any source files that depend on it.
Using add_definitions caused the flag to be passed to rc.exe on Windows and thus broke Windows builds.
After lots of follow-up fixes, there are still problems, such as
-Wno-suggest-override getting passed to the Windows Resource Compiler
because it was added with add_definitions in the CMake file.
Rather than piling on another fix, let's revert so this can be re-landed
when there's a proper fix.
This reverts commit 21c0b4c1e8.
This reverts commit 81d68ad27b.
This reverts commit a361aa5249.
This reverts commit fa42b7cf29.
This reverts commit 955f87f947.
This reverts commit 8b16e45f66.
This reverts commit 308a127a38.
This reverts commit 274b6b0c7a.
This reverts commit 1c7037a2a5.
Summary:
The fix improves handling of Windows UNC paths to align with Appendix E. Nonstandard Syntax Variations of RFC 8089.
Before this fix it was difficult to use Windows UNC paths in compile_commands.json database as such paths were converted to file URIs using 'file:////auth/share/file.cpp' notation instead of recommended 'file://auth/share/file.cpp'.
As an example, VS.Code cannot understand file URIs with 4 starting slashes, thus such features as go-to-definition, jump-to-file, hover tooltip, etc. stop working. This also applicable to files which reside on Windows network-mapped drives because clangd internally resolves file paths to real paths in some cases and such paths get resolved to UNC paths.
Reviewers: sammccall, kadircet
Reviewed By: sammccall
Subscribers: ormris, ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, usaxena95, kbobyrev, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D84172
Summary:
This was resulting in macros coming from preambles vanishing when user
have opened the source header. For example:
```
// test.h:
#define X
```
and
```
// test.cc
#include "test.h
^
```
If user only opens test.cc, we'll get `X` as a completion candidate,
since it is indexed as part of the preamble. But if the user opens
test.h afterwards we would index it as part of the main file and lose
the symbol (as new index shard for test.h will override the existing one
in dynamic index).
Also we were not setting origins for macros correctly, this patch also
fixes it.
Fixes https://github.com/clangd/clangd/issues/461
Reviewers: hokein
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D84297
Summary:
Remote server should not send messages that are invalid and will cause problems
on the client side. The client should not be affected by server's failures
whenever possible.
Also add more error messages and logs.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D83826
When a semantic checking fails on a syntactic InitListExpr, we will
get an ill-formed semantic InitListExpr (e.g. some inits are nullptr),
using this semantic InitListExpr in clang (without setting the err-bits) is crashy.
Differential Revision: https://reviews.llvm.org/D84140
* Try to apply heuristic resolution recursively to the base
expression of a CXXDependentScopeMemberExpr.
* Try to apply heuristic resolution recursively to the callee
expression in a call expression.
Fixes https://github.com/clangd/clangd/issues/441
Subscribers: ilya-biryukov, jkorous, arphaman, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D82739
Summary:
This patch mostly reverts D74850.
We could not use `AST.getTokens()` here, because it does not have tokens from the preamble.
Reviewers: sammccall, kadircet
Reviewed By: kadircet
Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, usaxena95, kbobyrev, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D84144
Adam Czachorowski