Commit Graph

588 Commits

Author SHA1 Message Date
Eric Liu e3395b9016 [clangd] Sort includes when formatting code or inserting new includes.
Reviewers: hokein, ilya-biryukov

Subscribers: klimek, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D44138

llvm-svn: 326773
2018-03-06 10:42:50 +00:00
Sam McCall 77a719cb9e [clangd] Fix unintentionally loose fuzzy matching, and the tests masking it.
Summary:
The intent was that [ar] doesn't match "FooBar"; the first character must match
a Head character (hard requirement, not just a low score).
This matches VSCode, and was "tested" but the tests were defective.

The tests expected matches("FooBar") to fail for lack of a match. But instead
it fails because the string should be annotated - matches("FooB[ar]").
This patch makes matches("FooBar") ignore annotations, as was intended.

Fixing the code to reject weak matches for the first char causes problems:
-  [bre] no longer matches "HTMLBRElement".
   We allow matching against an uppercase char even if we don't think it's head.
   Only do this if there's at least one lowercase, to avoid triggering on MACROS
-  [print] no longer matches "sprintf".
   This is hard to fix without false positives (e.g. [int] vs "sprintf"])
   This patch leaves this case broken. A future patch will add a dictionary
   providing custom segmentation to common names from the standard library.

Fixed a couple of index tests that indirectly relied on broken fuzzy matching.
Added const in a couple of missing places for consistency with new code.

Subscribers: klimek, ilya-biryukov, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D44003

llvm-svn: 326721
2018-03-05 17:34:33 +00:00
Sam McCall 7363a2f270 [clangd] Extract ClangdServer::Options struct.
Summary:
This subsumes most of the params to ClangdServer and ClangdLSPServer.
Adjacent changes:
 - tests use a consistent set of options, except when testing specific options
 - tests that previously used synchronous mode for convenience no longer do
 - added a runAddDocument helper to SyncAPIs to mitigate the extra code
 - rearranged main a bit to follow the structure of the options

Reviewers: ilya-biryukov

Subscribers: klimek, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D44088

llvm-svn: 326719
2018-03-05 17:28:54 +00:00
Ilya Biryukov ddedde9aa0 [clangd] Use higher timoout values in TUSchedulerTest::Debounce
Should unbreak windows buildbots.

llvm-svn: 326598
2018-03-02 18:23:41 +00:00
Sam McCall d1e0deb271 [clangd] Debounce streams of updates.
Summary:
Don't actually start building ASTs for new revisions until either:
- 500ms have passed since the last revision, or
- we actually need the revision for something (or to unblock the queue)

In practice, this avoids the "first keystroke results in diagnostics" problem.
This is kind of awkward to test, and the test is pretty bad.
It can be observed nicely by capturing a trace, though.

Reviewers: hokein, ilya-biryukov

Subscribers: klimek, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D43648

llvm-svn: 326546
2018-03-02 08:56:37 +00:00
Eric Liu b96363da76 [clangd] Support include canonicalization in symbol leve.
Summary:
Symbols with different canonical includes might be defined in the same header
(e.g. symbols defined in STL <iosfwd>). This patch adds support for mapping from
qualified symbol names to canonical headers and special mapping for symbols in <iosfwd>

Reviewers: sammccall, hokein

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D43869

llvm-svn: 326456
2018-03-01 18:06:40 +00:00
Eric Liu cf8601b009 [clangd] Prefer the definition of a TagDecl (e.g. class) as CanonicalDeclaration.
Summary:
Currently, we pick the first declaration of a symbol in a TU, which is considered
canonical in the clangIndex, as the canonical declaration in clangd. This causes
forward declarations that might appear in a random header to be used as a
canonical declaration, which is not desirable for features like go-to-declaration
or include insertion.

For example, for class X, we would consider the forward declaration in fwd.h to
be the canonical declaration, while the preferred canonical declaration should
be the actual definition in x.h.
```
// fwd.h
class X;  // forward decl

// x.h
class X {};
```

This patch fixes the issue by making symbol collector favor the actual definition of
a TagDecl (i.e. class/struct/enum/union) found in a header file over the first seen
declarations in a TU. Other symbol types like functions are not handled because
using the first seen declarations as canonical declarations is usually a good
heuristic for them.

Reviewers: sammccall

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D43823

llvm-svn: 326313
2018-02-28 09:33:15 +00:00
Sam McCall db3ea4c0d9 [clangd] Remove codecomplete override content API. Long live addDocument!
llvm-svn: 326211
2018-02-27 17:15:50 +00:00
Roman Lebedev 12b40745ab Revert "[Tooling] [1/1] Refactor FrontendActionFactory::create() to return std::unique_ptr<>"
This reverts commit rL326202

This broke gcc4.8 builds, compiler just segfaults:
http://lab.llvm.org:8011/builders/clang-atom-d525-fedora-rel/builds/14909
http://lab.llvm.org:8011/builders/clang-x86_64-linux-abi-test/builds/22673

llvm-svn: 326203
2018-02-27 15:54:41 +00:00
Roman Lebedev 6b56a11961 [Tooling] [1/1] Refactor FrontendActionFactory::create() to return std::unique_ptr<>
Summary:
I'm not sure whether there are any principal reasons why it returns raw owning pointer,
or it is just a old code that was not updated post-C++11.

I'm not too sure what testing i should do, because `check-all` is not error clean here for some reason,
but it does not //appear// asif those failures are related to these changes.

This is Clang-tools-extra part.
Clang part is D43779.

Reviewers: klimek, bkramer, alexfh, pcc

Reviewed By: alexfh

Subscribers: ioeric, jkorous-apple, cfe-commits

Tags: #clang, #clang-tools-extra

Differential Revision: https://reviews.llvm.org/D43780

llvm-svn: 326202
2018-02-27 15:19:28 +00:00
Eric Liu 6c8e858551 [clangd] don't insert new includes if either original header or canonical header is already included.
Summary:
Changes:
o Store both the original header and the canonical header in LSP command.
o Also check that both original and canonical headers are not already included
by comparing both resolved header path and written literal includes.

This addresses the use case where private IWYU pragma is defined in a private
header while it would still be preferrable to include the private header, in the
internal implementation file. If we have seen that the priviate header is already
included, we don't try to insert the canonical include.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D43510

llvm-svn: 326070
2018-02-26 08:32:13 +00:00
Sam McCall 091557d2a8 [clangd] BindWithForward -> Bind. NFC
llvm-svn: 325868
2018-02-23 07:54:17 +00:00
Sam McCall b1ac64927f [clangd] fix test use-after-free from r325774
llvm-svn: 325801
2018-02-22 15:33:33 +00:00
Simon Marchi 5178f924c4 [clangd] DidChangeConfiguration Notification
Summary:

Implementation of DidChangeConfiguration notification handling in
clangd.  This currently only supports changing one setting: the path of
the compilation database to be used for the current project.   In other
words, it is no longer necessary to restart clangd with a different
command line argument in order to change the compilation database.

Reviewers: malaperle, krasimir, bkramer, ilya-biryukov

Subscribers: jkorous-apple, ioeric, simark, klimek, ilya-biryukov, arphaman, rwols, cfe-commits

Differential Revision: https://reviews.llvm.org/D39571

Signed-off-by: Simon Marchi <simon.marchi@ericsson.com>
Signed-off-by: William Enright <william.enright@polymtl.ca>
llvm-svn: 325784
2018-02-22 14:00:39 +00:00
Sam McCall 568e17f1e6 [clangd] Allow embedders some control over when diagnostics are generated.
Summary:
Through the C++ API, we support for a given snapshot version:
 - Yes: make sure we generate diagnostics for exactly this version
 - Auto: generate eventually-consistent diagnostics for at least this version
 - No: don't generate diagnostics for this version
Eventually auto should be debounced for better UX.

Through LSP, we force diagnostics for initial load (bypassing future debouncing)
and all updates follow the "auto" policy.

This is complicated to implement under the CancellationFlag design, so
rewrote that part to just inspect the queue instead.

It turns out we never pass None to the diagnostics callback, so remove Optional
from the signature. The questionable behavior of not invoking the callback at
all if CppFile::rebuild fails is not changed.

Reviewers: ilya-biryukov

Subscribers: klimek, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D43518

llvm-svn: 325774
2018-02-22 13:11:12 +00:00
Eric Liu 02ce01f1e8 [clangd] Not collect include headers for dynamic index for now.
Summary:
The new behaviors introduced by this patch:
o When include collection is enabled, we always set IncludeHeader field in Symbol
even if it's the same as FileURI in decl.
o Disable include collection in FileIndex which is currently only used to build
dynamic index. We should revisit when we actually want to use FileIndex to global
index.
o Code-completion only uses IncludeHeader to insert headers but not FileURI in
CanonicalDeclaration. This ensures that inserted headers are always canonicalized.
Note that include insertion can still be triggered for symbols that are already
included if they are merged from dynamic index and static index, but we would
only use includes that are already canonicalized (e.g. from static index).

Reason for change:
Collecting header includes in dynamic index enables inserting includes for headers
that are not indexed but opened in the editor. Comparing to inserting includes for
symbols in global/static index, this is nice-to-have but would probably require
non-trivial amount of work to get right. For example:
o Currently it's not easy to fully support CanonicalIncludes in dynamic index, given the way
we run dynamic index.
o It's also harder to reason about the correctness of include canonicalization for dynamic index
(i.e. symbols in the current file/TU) than static index where symbols are collected
offline and sanity check is possible before shipping to production.
o We have less control/flexibility over symbol info in the dynamic index
(e.g. URIs, path normalization), which could be used to help make decision when inserting includes.

As header collection (especially canonicalization) is relatively new, and enabling
it for dynamic index would immediately affect current users with only dynamic
index support, I propose we disable it for dynamic index for now to avoid
compromising other hot features like code completion and only support it for
static index where include insertion would likely to bring more value.

Reviewers: ilya-biryukov, sammccall, hokein

Subscribers: klimek, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D43550

llvm-svn: 325764
2018-02-22 10:14:05 +00:00
Marc-Andre Laperle 63a1098d73 [clangd] #include statements support for Open definition
Summary: ctrl-clicking on #include statements now opens the file being pointed by that statement.

Reviewers: malaperle, krasimir, bkramer, ilya-biryukov

Reviewed By: ilya-biryukov

Subscribers: jkorous-apple, ioeric, mgrang, klimek, ilya-biryukov, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D38639

llvm-svn: 325662
2018-02-21 02:39:08 +00:00
Eric Liu 709bde886d [clangd] Fixes for #include insertion.
Summary:
o Avoid inserting a header include into the header itself.
o Avoid inserting non-header files (by not indexing symbols in main
files at all).
o Canonicalize include paths for symbols in dynamic index.

Reviewers: sammccall, ilya-biryukov

Reviewed By: ilya-biryukov

Subscribers: klimek, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D43462

llvm-svn: 325523
2018-02-19 18:48:44 +00:00
Ilya Biryukov 7fac6e92a7 [clangd] Do not reuse preamble if compile args changed
Reviewers: hokein, ioeric, sammccall, simark

Reviewed By: simark

Subscribers: klimek, jkorous-apple, cfe-commits, simark

Differential Revision: https://reviews.llvm.org/D43454

llvm-svn: 325522
2018-02-19 18:18:49 +00:00
Sam McCall ab8e393f62 [clangd] Invert return value of fuzzyFind() (fix MemIndex's return value)
Have had way too many bugs by converting between "isComplete" and
"isIncomplete". LSP is immovable, so use isIncomplete everywhere.

llvm-svn: 325493
2018-02-19 13:04:41 +00:00
Sam McCall c901c5db78 [clangd] Tracing: name worker threads, and enforce naming scheduled async tasks
Summary:
This has a bit of a blast radius, but I think there's enough value in "forcing"
us to give names to these async tasks for debugging. Guessing about what
multithreaded code is doing is so unfun...

The "file" param attached to the tasks may seem to be redundant with the thread
names, but note that thread names are truncated to 15 chars on linux!
We'll be lucky to get the whole basename...

Reviewers: ilya-biryukov

Subscribers: klimek, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D43388

llvm-svn: 325480
2018-02-19 09:56:28 +00:00
Heejin Ahn f4a29252fb [clangd] Fix link failures for Preprocessor::addCommentHandler
Summary:
D42640 adds calls to `Preprocessor::addCommentHandler` in
`unittests/clangd/SymbolCollectorTests.cpp` and
`clangd/global-symbol-builder/GlobalSymbolBuilderMain.cpp` but does not
link `clangLex` library. This causes undefined reference errors when
built with `-DBUILD_SHARED_LIBS=ON`.

Reviewers: ioeric

Subscribers: klimek, mgorny, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D43437

llvm-svn: 325458
2018-02-18 10:50:16 +00:00
Marc-Andre Laperle 373e30a264 [clangd] Rename some protocol field to lower case
Summary:
Also fixes a GCC compilation error.

Signed-off-by: Marc-Andre Laperle <marc-andre.laperle@ericsson.com>

Subscribers: klimek, ilya-biryukov, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D43411

llvm-svn: 325409
2018-02-16 23:12:26 +00:00
Marc-Andre Laperle 3e618ed8f0 [clangd] Implement textDocument/hover
Summary: Implemention of textDocument/hover as described in LSP definition.

This patch adds a basic Hover implementation.  When hovering a variable,
function, method or namespace, clangd will return a text containing the
declaration's scope, as well as the declaration of the hovered entity.
For example, for a variable:

  Declared in class Foo::Bar

  int hello = 2

For macros, the macro definition is returned.

This patch doesn't include:

- markdown support (the client I use doesn't support it yet)
- range support (optional in the Hover response)
- comments associated to variables/functions/classes

They are kept as future work to keep this patch simpler.

I added tests in XRefsTests.cpp.  hover.test contains one simple
smoketest to make sure the feature works from a black box perspective.

Reviewers: malaperle, krasimir, bkramer, ilya-biryukov

Subscribers: sammccall, mgrang, klimek, rwols, ilya-biryukov, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D35894

Signed-off-by: Simon Marchi <simon.marchi@ericsson.com>
Signed-off-by: William Enright <william.enright@polymtl.ca>
llvm-svn: 325395
2018-02-16 21:38:15 +00:00
Eric Liu c5105f9e3c [clangd] collect symbol #include & insert #include in global code completion.
Summary:
o Collect suitable #include paths for index symbols. This also does smart mapping
for STL symbols and IWYU pragma (code borrowed from include-fixer).
o For global code completion, add a command for inserting new #include in each code
completion item.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, mgorny, ilya-biryukov, jkorous-apple, hintonda, cfe-commits

Differential Revision: https://reviews.llvm.org/D42640

llvm-svn: 325343
2018-02-16 14:15:55 +00:00
Sam McCall c156806844 [clangd] TestFS cleanup: getVirtualBlahBlah -> testPath/testRoot. Remove SmallString micro-opt for more ergonomic tests. NFC
llvm-svn: 325326
2018-02-16 09:41:43 +00:00
Ilya Biryukov 40bf17c1ea [clangd] Fixed compilation with MVSC.
llvm-svn: 325254
2018-02-15 15:41:49 +00:00
Ilya Biryukov 2c5e8e820b [clangd] Make functions of ClangdServer callback-based
Summary:
As a consequence, all LSP operations are now handled asynchronously,
i.e. they never block the main processing thread. However, if
-run-synchronously flag is specified, clangd still runs everything on
the main thread.

Reviewers: sammccall, ioeric, hokein

Reviewed By: sammccall, ioeric

Subscribers: klimek, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D43227

llvm-svn: 325233
2018-02-15 13:15:47 +00:00
Sam McCall 7929d006d7 [clangd] Fix tracing now that spans lifetimes can overlap on a thread.
Summary:
The chrome trace viewer requires events within a thread to strictly nest.
So we need to record the lifetime of the Span objects, not the contexts.

But we still want to show the relationship between spans where a context crosses
threads, so do this with flow events (i.e. arrows).

Before: https://photos.app.goo.gl/q4Dd9u9xtelaXk1v1
After: https://photos.app.goo.gl/5RNLmAMLZR3unvY83

(This could stand some further improvement, in particular I think we want a
container span whenever we schedule work on a thread. But that's another patch)

Reviewers: ioeric

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D43272

llvm-svn: 325220
2018-02-15 08:40:54 +00:00
Ilya Biryukov 12aa52fe22 [clangd] Fix data race in ClangdThreadingTest.StressTest
Prior to this patch, same instance of VFS was shared for concurrent
processing of the files in ClangdThreadingTest.StressTest.

It caused a data race as the same instance of InMemoryFileSystem was
mutated from multiple threads using setCurrentWorkingDirectory().

llvm-svn: 325132
2018-02-14 15:19:20 +00:00
Ilya Biryukov 7beea3ab73 [clangd] Explicitly initialize all primitive fields in Protocol.h
Summary:
Some of the existing structs had primimtive fields that were
not explicitly initialized on construction.
After this commit every struct consistently sets a defined value for
every field when default-initialized.

Reviewers: hokein, ioeric, sammccall

Reviewed By: sammccall

Subscribers: klimek, cfe-commits, jkorous-apple

Differential Revision: https://reviews.llvm.org/D43230

llvm-svn: 325113
2018-02-14 10:52:04 +00:00
Ilya Biryukov 18c0247852 [clangd] Fixed findDefinitions to always return absolute paths.
Relative paths could be returned in some cases, e.g. when relative
path is used in compilation arguments. This led to crash when trying
to convert the path to URI.

llvm-svn: 325029
2018-02-13 17:47:16 +00:00
Ilya Biryukov 42fed626bc [clangd] Remove the RealFS layer from test VFS. NFC.
It was required before because preambles could only be created on
disk. All tests use in-memory preambles now.

llvm-svn: 325021
2018-02-13 17:08:13 +00:00
Haojian Wu dc02a3d943 [clangd] SymbolLocation only covers symbol name.
Summary:
* Change the offset range to half-open, [start, end).
* Fix a few fixmes.

Reviewers: sammccall

Subscribers: klimek, ilya-biryukov, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D43182

llvm-svn: 324992
2018-02-13 09:53:50 +00:00
Sam McCall 0bb24cd4fa [clangd] Stop exposing Futures from ClangdServer operations.
Summary:
LSP has asynchronous semantics, being able to block on an async operation
completing is unneccesary and leads to tighter coupling with the threading.

In practice only tests depend on this, so we add a general-purpose "block until
idle" function to the scheduler which will work for all operations.

To get this working, fix a latent condition-variable bug in ASTWorker, and make
AsyncTaskRunner const-correct.

Reviewers: ilya-biryukov

Subscribers: klimek, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D43127

llvm-svn: 324990
2018-02-13 08:59:23 +00:00
Haojian Wu cee2059f9b [clang-move] Fix the incorrect expansion end location.
Summary:
Before the fix, if clang-move decides to move the following macro statement, it only moves the first line `DEFINE(A,`.

```
DEFINE(A,
       B);
```

Reviewers: ioeric

Reviewed By: ioeric

Subscribers: klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D43174

llvm-svn: 324886
2018-02-12 12:26:12 +00:00
Ilya Biryukov cd5eb00e8b [clangd] Remove codeComplete that returns std::future<>
Summary:
It was deprecated and callback version and is used everywhere.
Only changes to the testing code were needed.

Reviewers: hokein, ioeric, sammccall

Reviewed By: sammccall

Subscribers: mgorny, klimek, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D43068

llvm-svn: 324883
2018-02-12 11:37:28 +00:00
Haojian Wu d478634777 [clang-move] Don't dump macro symbols.
Reviewers: ioeric

Subscribers: klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D43075

llvm-svn: 324742
2018-02-09 15:57:30 +00:00
Ilya Biryukov be58c9605d [clangd] Fix crash in tests in debug mode.
Caused by the lack of checking of an Expected value in the previous
commit.

llvm-svn: 324736
2018-02-09 15:11:07 +00:00
Sam McCall 6003951c66 [clangd] Collect definitions when indexing.
Within a TU:
 - as now, collect a declaration from the first occurrence of a symbol
   (taking clang's canonical declaration)
 - when we first see a definition occurrence, copy the symbol and add it
Across TUs/sources:
 - mergeSymbol in Merge.h is responsible for combining matching Symbols.
   This covers dynamic/static merges and cross-TU merges in the static index.
 - it prefers declarations from Symbols that have a definition.
 - GlobalSymbolBuilderMain is modified to use mergeSymbol as a reduce step.
Random cleanups (can be pulled out):
 - SymbolFromYAML -> SymbolsFromYAML, new singular SymbolFromYAML added
 - avoid uninit'd SymbolLocations. Add an idiomatic way to check "absent".
 - CanonicalDeclaration (as well as Definition) are mapped as optional in YAML.
 - added operator<< for Symbol & SymbolLocation, for debugging

Reviewers: ioeric, hokein

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D42942

llvm-svn: 324735
2018-02-09 14:42:01 +00:00
Ilya Biryukov b6ad25cd4c [clangd] Fix crash when CompilerInvocation can't be created.
Summary:
This can happen if the CompileCommand provided by compilation database
is malformed.

Reviewers: hokein, ioeric, sammccall

Reviewed By: sammccall

Subscribers: klimek, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D43122

llvm-svn: 324732
2018-02-09 13:51:57 +00:00
Ilya Biryukov 7aca05102c [clangd] Update include guard in Annotations.h. NFC
llvm-svn: 324599
2018-02-08 12:46:34 +00:00
Ilya Biryukov 7e5ee26d1a Resubmit "[clangd] The new threading implementation"
Initially submitted as r324356 and reverted in r324386.

This change additionally contains a fix to crashes of the buildbots.
The source of the crash was undefined behaviour caused by
std::future<> whose std::promise<> was destroyed without calling
set_value().

llvm-svn: 324575
2018-02-08 07:37:35 +00:00
Eric Liu 7deda4f5af [clangd] Do not precent-encode numbers in URI.
Reviewers: ilya-biryukov

Subscribers: klimek, jkorous-apple, cfe-commits, sammccall

Differential Revision: https://reviews.llvm.org/D43009

llvm-svn: 324475
2018-02-07 12:12:06 +00:00
Ilya Biryukov 3693f5941a Revert "[clangd] The new threading implementation" (r324356)
And the follow-up changes r324361 and r324363.
These changes seem to break two buildbots:
  - http://lab.llvm.org:8011/builders/clang-atom-d525-fedora-rel/builds/14091
  - http://lab.llvm.org:8011/builders/clang-x86_64-linux-selfhost-modules-2/builds/16001

We will need to investigate what went wrong and resubmit the changes
afterwards.

llvm-svn: 324386
2018-02-06 19:22:40 +00:00
Ilya Biryukov b9cf83b16f [clangd] Fixed compilation on Windows buildbot.
llvm-svn: 324361
2018-02-06 16:32:36 +00:00
Eric Liu 7f24765912 [clangd] Use URIs in index symbols.
Reviewers: hokein, sammccall

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D42915

llvm-svn: 324358
2018-02-06 16:10:35 +00:00
Ilya Biryukov cce8883094 [clangd] The new threading implementation
Summary:
In the new threading model clangd creates one thread per file to manage
the AST and one thread to process each of the incoming requests.
The number of actively running threads is bounded by the semaphore to
avoid overloading the system.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, mgorny, jkorous-apple, ioeric, hintonda, cfe-commits

Differential Revision: https://reviews.llvm.org/D42573

llvm-svn: 324356
2018-02-06 15:53:42 +00:00
Haojian Wu 3b8e00c5e0 [clangd] Fix incorrect file path for symbols defined by the compile command-line option.
Summary:

Reviewers: ioeric

Reviewed By: ioeric

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D42913

llvm-svn: 324328
2018-02-06 09:50:35 +00:00
Sam McCall c40fdcdd7e [clangd] Fix ExternC test broken by r324081
llvm-svn: 324105
2018-02-02 17:01:36 +00:00
Eric Liu cf1773826f [clangd] Skip inline namespace when collecting scopes for index symbols.
Summary:
Some STL symbols are defined in inline namespaces. For example,
```
namespace std {
inline namespace __cxx11 {
 typedef ... string;
}
}
```
Currently, this will be `std::__cxx11::string`; however, `std::string` is desired.

Inline namespaces are treated as transparent scopes. This
reflects the way they're most commonly used for lookup. Ideally we'd
include them, but at query time it's hard to find all the inline
namespaces to query: the preamble doesn't have a dedicated list.

Reviewers: sammccall, hokein

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D42796

llvm-svn: 324065
2018-02-02 10:31:42 +00:00
Haojian Wu a8d12bbc3f [clangd] remove the unused code NFC.
llvm-svn: 323960
2018-02-01 13:06:58 +00:00
Eric Liu cda2526d86 [clangd] Fix URI scheme conflict and an unused variable warning in tests. NFC
llvm-svn: 323957
2018-02-01 12:44:52 +00:00
Sam McCall d1a7a37c22 [clangd] Pass Context implicitly using TLS.
Summary:
Instead of passing Context explicitly around, we now have a thread-local
Context object `Context::current()` which is an implicit argument to
every function.
Most manipulation of this should use the WithContextValue helper, which
augments the current Context to add a single KV pair, and restores the
old context on destruction.

Advantages are:
- less boilerplate in functions that just propagate contexts
- reading most code doesn't require understanding context at all, and
  using context as values in fewer places still
- fewer options to pass the "wrong" context when it changes within a
  scope (e.g. when using Span)
- contexts pass through interfaces we can't modify, such as VFS
- propagating contexts across threads was slightly tricky (e.g.
  copy vs move, no move-init in lambdas), and is now encapsulated in
  the threadpool

Disadvantages are all the usual TLS stuff - hidden magic, and
potential for higher memory usage on threads that don't use the
context. (In practice, it's just one pointer)

Reviewers: ilya-biryukov

Subscribers: klimek, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D42517

llvm-svn: 323872
2018-01-31 13:40:48 +00:00
Haojian Wu b018906051 [clangd] Better handling symbols defined in macros.
Summary:
For symbols defined inside macros:
 * use expansion location, if the symbol is formed via macro concatenation.
 * use spelling location, otherwise.

This will fix some symbols that have ill-format location (especial invalid filepath).

Reviewers: ioeric

Reviewed By: ioeric

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D42575

llvm-svn: 323867
2018-01-31 12:56:51 +00:00
Haojian Wu fb68ca1825 [clang-move] Clever on handling header file which includes itself.
Summary:
Previously, we assume only old.cc includes "old.h", which would
introduce incorrect fixes for the cases where old.h also includes `#include "old.h"`

Although it should not be occurred in real projects, clang-move should handle this.

Old.h:

```
class Foo {};
```

after moving to a new old.h:

```
class Foo {};
```

Reviewers: ioeric

Reviewed By: ioeric

Subscribers: cfe-commits, klimek

Differential Revision: https://reviews.llvm.org/D42639

llvm-svn: 323865
2018-01-31 12:12:29 +00:00
Ilya Biryukov 75f1dd9b98 [clangd] Refactored threading in ClangdServer
Summary:
We now provide an abstraction of Scheduler that abstracts threading
and resource management in ClangdServer.
No changes to behavior are intended with an exception of changed error
messages.
This patch is preliminary work to allow a revamped threading
implementation that will move the threading code out of CppFile.

Reviewers: sammccall, bkramer, jkorous-apple

Reviewed By: sammccall

Subscribers: hokein, mgorny, hintonda, ioeric, jkorous-apple, cfe-commits, klimek

Differential Revision: https://reviews.llvm.org/D42174

llvm-svn: 323851
2018-01-31 08:51:16 +00:00
Eric Liu b57da098e9 [clangd] Fix r323658 test failure on windows.
llvm-svn: 323703
2018-01-29 22:28:08 +00:00
Eric Liu 78ed91a76c [clangd] Use new URI with scheme support in place of the existing LSP URI
Summary:
o Replace the existing clangd::URI with a wrapper of FileURI which also
carries a resolved file path.
o s/FileURI/URI/
o Get rid of the URI hack in vscode extension.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D42419

llvm-svn: 323660
2018-01-29 15:37:46 +00:00
Eric Liu 278e2d1a26 [clangd] Add a fallback directory for collected symbols with relative paths.
Reviewers: hokein, sammccall

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D42638

llvm-svn: 323658
2018-01-29 15:13:29 +00:00
Sam McCall 1b475a1ad0 [clangd] Modify the Span API so that Spans propagate with contexts.
Summary:
This is probably the right behavior for distributed tracers, and makes unpaired
begin-end events impossible without requiring Spans to be bound to a thread.

The API is conceptually clean but syntactically awkward. As discussed offline,
this is basically a naming problem and will go away if (when) we use TLS to
store the current context.

The apparently-unrelated change to onScopeExit are because its move semantics
broken if Func is POD-like since r322838. This is true of function pointers,
and the lambda I use here that captures two pointers only.
I've raised this issue on llvm-dev and will revert this part if we fix it in
some other way.

Reviewers: ilya-biryukov

Subscribers: klimek, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D42499

llvm-svn: 323511
2018-01-26 09:00:30 +00:00
Sam McCall 034e11aca5 [clangd] Add ClangdUnit diagnostics tests using annotated code.
Summary:
This adds checks that our diagnostics emit correct ranges in a bunch of cases,
as promised in D41118.

The diagnostics-preamble test is also converted and extended to be a little more
precise.

diagnostics.test stays around as the smoke test for this feature.

Reviewers: ilya-biryukov

Subscribers: klimek, mgorny, cfe-commits

Differential Revision: https://reviews.llvm.org/D41454

llvm-svn: 323448
2018-01-25 17:29:17 +00:00
Ilya Biryukov df84234364 [clangd] Provide a helper to report estimated memory usage per-file
Reviewers: sammccall, ioeric, hokein

Reviewed By: ioeric

Subscribers: klimek, cfe-commits, jkorous-apple

Differential Revision: https://reviews.llvm.org/D42480

llvm-svn: 323425
2018-01-25 14:32:21 +00:00
Haojian Wu 58d208d6e4 [clangd] Disable typo correction when doing code completion.
Reviewers: ilya-biryukov

Reviewed By: ilya-biryukov

Subscribers: klimek, jkorous-apple, cfe-commits, ioeric

Differential Revision: https://reviews.llvm.org/D42491

llvm-svn: 323410
2018-01-25 09:44:06 +00:00
Haojian Wu 48b4865683 [clangd] Limit completion results.
Summary:
* truncate symbols from static/dynamic index to the limited number
(which would save lots of cost in constructing the merged symbols).
* add an CLI option allowing to limit the number of returned completion results.
(default to 100)

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D42484

llvm-svn: 323408
2018-01-25 09:20:09 +00:00
Sam McCall d5ea3e32a6 [clangd] add test for r323347 CodeComplete behavior we rely on
llvm-svn: 323350
2018-01-24 17:53:32 +00:00
Haojian Wu 061c73eb28 [clangd] Use accessible scopes to query indexes for global code completion.
Summary:
 * For qualified completion (foo::a^)
       * unresolved qualifier - use global namespace ("::")
       * resolved qualifier - use all accessible namespaces inside the resolved qualifier.
 * For unqualified completion (vec^), use scopes that are accessible from the
      scope from which code completion occurs.

Reviewers: sammccall, ilya-biryukov

Reviewed By: sammccall

Subscribers: jkorous-apple, ioeric, klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D42073

llvm-svn: 323189
2018-01-23 11:37:26 +00:00
Sam McCall d2a9592526 [clangd] Drop ~destructor completions - rarely helpful and work inconsistently
llvm-svn: 323149
2018-01-22 21:05:00 +00:00
Eric Liu 43cee17cbf [clangd] Try to fix build bot failue in r323101
llvm-svn: 323111
2018-01-22 13:03:25 +00:00
Eric Liu f5b8c82198 [clangd] Add support for different file URI schemas.
Summary: I will replace the existing URI struct in Protocol.h with the new URI and rename FileURI to URI in a followup patch.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: jkorous-apple, klimek, mgorny, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41946

llvm-svn: 323101
2018-01-22 11:48:20 +00:00
Sam McCall 8b2faeed02 [clangd] Change index scope convention from "outer::inner" to "outer::inner::"
Global scope is "" (was "")
Top-level namespace scope is "ns::" (was "ns")
Nested namespace scope is "ns::ns::" (was "ns::ns")

This composes more naturally:
- qname = scope + name
- full scope = resolved scope + unresolved scope (D42073 was the trigger)
It removes a wart from the old way: "foo::" has one more separator than "".

Another alternative that has these properties is "::ns", but that lacks
the property that both the scope and the name are substrings of the
qname as produced by clang.

This is re-landing r322996 which didn't build.

llvm-svn: 323000
2018-01-19 22:18:21 +00:00
Sam McCall 50584ea197 Revert "[clangd] Change index scope convention from "outer::inner" to "outer::inner::""
This reverts commit r322996.

llvm-svn: 322998
2018-01-19 22:09:34 +00:00
Sam McCall 1a8c55ecc6 [clangd] Change index scope convention from "outer::inner" to "outer::inner::"
Global scope is "" (was "")
Top-level namespace scope is "ns::" (was "ns")
Nested namespace scope is "ns::ns::" (was "ns::ns")

This composes more naturally:
 - qname = scope + name
 - full scope = resolved scope + unresolved scope (D42073 was the trigger)
It removes a wart from the old way: "foo::" has one more separator than "".

Another alternative that has these properties is "::ns", but that lacks
the property that both the scope and the name are substrings of the
qname as produced by clang.

llvm-svn: 322996
2018-01-19 21:58:58 +00:00
Sam McCall 545a20d90b [clangd] Merge index-provided completions with those from Sema.
Summary:
 - we match on USR, and do a field-by-field merge if both have results
 - scoring is post-merge, with both sets of information available
   (for now, sema priority is used if available, static score for index results)
 - limit is applied to the complete result set (previously index ignored limit)
 - CompletionItem is only produces for the returned results
 - If the user doesn't type a scope, we send the global scope for completion
   (we can improve this after D42073)

Reviewers: ioeric

Subscribers: klimek, ilya-biryukov, mgrang, cfe-commits

Differential Revision: https://reviews.llvm.org/D42181

llvm-svn: 322945
2018-01-19 14:34:02 +00:00
Haojian Wu 9873fdd4f5 [clangd] Collect enum constants in SymbolCollector
Summary:
* ignore nameless symbols
* include enum constant declarataion

Reviewers: ilya-biryukov, jkorous-apple

Reviewed By: ilya-biryukov

Subscribers: ioeric, cfe-commits, klimek

Differential Revision: https://reviews.llvm.org/D42074

llvm-svn: 322929
2018-01-19 09:35:55 +00:00
Sam McCall d8169a85cc [clangd] Remove unused IncludeGlobals completion option, always pass true to sema
llvm-svn: 322856
2018-01-18 15:31:30 +00:00
Sam McCall a15c2d683e [clangd] CodeCompleteTests cleanup: naming, ordering, helpers. NFC
llvm-svn: 322824
2018-01-18 09:27:56 +00:00
Sam McCall a5ce243fa3 [clangd] Use fuzzy match to select top N index results.
Summary:
This makes performance slower but more predictable (it always processes
every symbol). We need to find ways to make this fast, possibly by precomputing
short queries or capping the number of scored results. But our current approach
is too naive.

It also no longer returns results in a "good" order. In fact it's pathological:
the top N results are ranked from worst to best. Indexes aren't responsible for
ranking and MergedIndex can't do a good job, so I'm pleased that this will make
any hidden assumptions we have more noticeable :-)

Reviewers: hokein

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D42060

llvm-svn: 322821
2018-01-18 08:35:04 +00:00
Sam McCall 2c3849af22 [clangd] Avoid combinatorial explosion in CodeCompleteTests.
Summary:
This test dominates our unit test runtime, and the change speeds it up by 10x.
We lose coverage of some combinations of flags, but I'm not sure that's finding
many bugs.

3300 -> 300ms on my machine (3800 -> 800ms for the whole of CompletionTest).

Reviewers: ilya-biryukov

Subscribers: klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D42063

llvm-svn: 322547
2018-01-16 12:21:24 +00:00
Sam McCall 0faecf0c33 [clangd] Merge results from static/dynamic index.
Summary:
We now hide the static/dynamic split from the code completion, behind a
new implementation of the SymbolIndex interface. This will reduce the
complexity of the sema/index merging that needs to be done by
CodeComplete, at a fairly small cost in flexibility.

Reviewers: hokein

Subscribers: klimek, mgorny, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D42049

llvm-svn: 322480
2018-01-15 12:33:00 +00:00
Sam McCall 09b4be51a8 [clangd] Fix uninitialized-read found by asan
llvm-svn: 322443
2018-01-13 16:46:26 +00:00
Sam McCall 3d139c527b [clangd] Code completion uses Sema for NS-level things in the current file.
Summary:
To stay fast, it avoids deserializing anything outside the current file, by
disabling the LoadExternal code completion option added in r322377, when the
index is enabled.

Reviewers: hokein

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41996

llvm-svn: 322387
2018-01-12 18:30:08 +00:00
Sam McCall 93483e7f39 [clangd] Include debugging tags for both static and dynamic index. NFC
llvm-svn: 322379
2018-01-12 17:09:49 +00:00
Sam McCall 84652cc63c [clangd] Incorporate fuzzy-match into result rankings.
Summary: The scoring function is fuzzy-match-quality * existing quality score.

Reviewers: ioeric

Subscribers: klimek, cfe-commits, ilya-biryukov

Differential Revision: https://reviews.llvm.org/D40780

llvm-svn: 322377
2018-01-12 16:16:09 +00:00
Haojian Wu f4374f087f [clangd] Don't navigate to forward class declaration when go to definition.
Summary:
For some cases, GoToDefinition will navigate to the forward class
declaration, we should always navigate to the class definition.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41661

llvm-svn: 322370
2018-01-12 14:21:10 +00:00
Ilya Biryukov 9555839228 [clangd] Pass Context to onDiagnosticsReady callback
Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D41907

llvm-svn: 322199
2018-01-10 17:59:27 +00:00
Eric Liu 9af958fbf8 [clangd] Add more filters for collected symbols.
Summary:
o We only collect symbols in namespace or translation unit scopes.
o Add an option to only collect symbols in included headers.

Reviewers: hokein, ilya-biryukov

Reviewed By: hokein, ilya-biryukov

Subscribers: klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D41823

llvm-svn: 322193
2018-01-10 14:57:58 +00:00
Haojian Wu ba28e9ac5d [clangd] Add static index for the global code completion.
Summary:
Use the YAML-format symbols (generated by the global-symbol-builder tool) to
do the global code completion.
It is **experimental** only , but it allows us to experience global code
completion on a relatively small project.

Tested with LLVM project.

Reviewers: sammccall, ioeric

Reviewed By: sammccall, ioeric

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41668

llvm-svn: 322191
2018-01-10 14:44:34 +00:00
Ilya Biryukov f60bf347c9 [clangd] Remove duplicates from code completion
Summary:
This patch removes hidden items from code completion.
Items can be hidden, e.g., by other items in the child scopes.
This patch addresses a particular problem of a duplicate completion
item for the class in the following example:

    struct Adapter { void method(); };
    void Adapter::method() {
       Adapter^
    }

We should probably investigate if there are other duplicates in
completion and remove them, possibly adding assertions that it never
happens.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: cfe-commits, klimek

Differential Revision: https://reviews.llvm.org/D41901

llvm-svn: 322185
2018-01-10 13:51:09 +00:00
Eric Liu 76f6b44443 [clangd] Add more symbol information for code completion.
Reviewers: hokein, sammccall

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41345

llvm-svn: 322097
2018-01-09 17:32:00 +00:00
Eric Liu 5732c0e6ea [clangd] Use ToolExecutor to write the global-symbol-builder tool.
Summary:
This enables more execution modes like standalone and Mapreduce-style execution.

See also D41729

Reviewers: hokein, sammccall

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41730

llvm-svn: 322084
2018-01-09 15:21:45 +00:00
Haojian Wu fe22b7474b [clangd] Catch more symbols in SymbolCollector.
Summary:
We currently only collect external-linkage symbols in the collector,
which results in missing some typical symbols (like no-linkage type alias symbols).

This patch relaxes the constraint a bit to allow collecting more symbols.

Reviewers: ilya-biryukov

Reviewed By: ilya-biryukov

Subscribers: klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D41759

llvm-svn: 322067
2018-01-09 10:44:09 +00:00
Ilya Biryukov 5a5e1caf25 [clangd] Properly set filterText for index-based completion items
It was previously set to an identifier that the user typed, leading to
surprising behavior in VSCode (probably in other editors too).

llvm-svn: 321554
2017-12-29 14:59:22 +00:00
Sam McCall 4b9bbb378b [clangd] Use Builder for symbol slabs, and use sorted-vector for storage
Summary:
This improves a few things:
 - the insert -> freeze -> read sequence is now enforced/communicated by the
   type system
 - SymbolSlab::const_iterator iterates over symbols, not over id-symbol pairs
 - we avoid permanently storing a second copy of the IDs, and the
   string map's hashtable

The slab size is now down to 21.8MB for the LLVM project.
Of this only 2.7MB is strings, the rest is #symbols * `sizeof(Symbol)`.
`sizeof(Symbol)` is currently 96, which seems too big - I think
SymbolInfo isn't efficiently packed. That's a topic for another patch!

Also added simple API to see the memory usage/#symbols of a slab, since
it seems likely we will continue to care about this.

Reviewers: ilya-biryukov

Subscribers: klimek, mgrang, cfe-commits

Differential Revision: https://reviews.llvm.org/D41506

llvm-svn: 321412
2017-12-23 19:38:03 +00:00
Paul Robinson 1db24196d1 Update test after r321312
llvm-svn: 321323
2017-12-21 23:19:25 +00:00
Sam McCall 6c0d0f5775 [clangd] Index symbols share storage within a slab.
Summary:
Symbols are not self-contained - it's only safe to hand them out if you
guarantee the lifetime of the underlying data.

Before this lands, I'm going to measure the before/after memory usage of the
LLVM index loaded into memory in a single slab.

Reviewers: hokein

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41483

llvm-svn: 321272
2017-12-21 14:58:44 +00:00
Eric Liu 63696e14e3 [clangd] Pull CodeCompletionString handling logic into its own file and add unit test.
Reviewers: sammccall

Subscribers: klimek, mgorny, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41450

llvm-svn: 321193
2017-12-20 17:24:31 +00:00
Eric Liu 188adaf46b [clangd] Remove an unused lambda capture.
llvm-svn: 321192
2017-12-20 17:22:56 +00:00
Sam McCall 328cbdb9e4 [clangd] Switch xrefs and documenthighlight to annotated-code unit tests. NFC
Summary:
The goal here is again to make it easier to read and write the tests.

I've extracted `parseTextMarker` from CodeCompleteTests into an `Annotations`
class, adding features to it:
  - as well as points `^s` it allows ranges `[[...]]`
  - multiple points and ranges are supported
  - points and ranges may be named: `$name^` and `$name[[...]]`

These features are used for the xrefs tests. This also paves the way for
replacing the lit diagnostics.test with more readable unit tests, using named
ranges.

Alternative considered: `TestSelectionRange` in clang-refactor/TestSupport
Main problems were:
 - delimiting the end of ranges is awkward, requiring counting
 - comment syntax is long and at least as cryptic for most cases
 - no separate syntax for point vs range, which keeps xrefs tests concise
 - Still need to convert to Position everywhere
 - Still need helpers for common case of expecting exactly one point/range

(I'll probably promote the extra `PrintTo`s from some of the core Protocol types
into `operator<<` in `Protocol.h` itself in a separate, prior patch...)

Reviewers: ioeric

Subscribers: klimek, mgorny, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41432

llvm-svn: 321184
2017-12-20 16:06:05 +00:00
Sam McCall fffa8229e3 [clangd] Add debug printers for basic protocol types. NFC
llvm-svn: 321161
2017-12-20 10:26:53 +00:00