Commit Graph

186 Commits

Author SHA1 Message Date
Sam McCall 0bb24cd4fa [clangd] Stop exposing Futures from ClangdServer operations.
Summary:
LSP has asynchronous semantics, being able to block on an async operation
completing is unneccesary and leads to tighter coupling with the threading.

In practice only tests depend on this, so we add a general-purpose "block until
idle" function to the scheduler which will work for all operations.

To get this working, fix a latent condition-variable bug in ASTWorker, and make
AsyncTaskRunner const-correct.

Reviewers: ilya-biryukov

Subscribers: klimek, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D43127

llvm-svn: 324990
2018-02-13 08:59:23 +00:00
Ilya Biryukov cd5eb00e8b [clangd] Remove codeComplete that returns std::future<>
Summary:
It was deprecated and callback version and is used everywhere.
Only changes to the testing code were needed.

Reviewers: hokein, ioeric, sammccall

Reviewed By: sammccall

Subscribers: mgorny, klimek, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D43068

llvm-svn: 324883
2018-02-12 11:37:28 +00:00
Ilya Biryukov be58c9605d [clangd] Fix crash in tests in debug mode.
Caused by the lack of checking of an Expected value in the previous
commit.

llvm-svn: 324736
2018-02-09 15:11:07 +00:00
Sam McCall 6003951c66 [clangd] Collect definitions when indexing.
Within a TU:
 - as now, collect a declaration from the first occurrence of a symbol
   (taking clang's canonical declaration)
 - when we first see a definition occurrence, copy the symbol and add it
Across TUs/sources:
 - mergeSymbol in Merge.h is responsible for combining matching Symbols.
   This covers dynamic/static merges and cross-TU merges in the static index.
 - it prefers declarations from Symbols that have a definition.
 - GlobalSymbolBuilderMain is modified to use mergeSymbol as a reduce step.
Random cleanups (can be pulled out):
 - SymbolFromYAML -> SymbolsFromYAML, new singular SymbolFromYAML added
 - avoid uninit'd SymbolLocations. Add an idiomatic way to check "absent".
 - CanonicalDeclaration (as well as Definition) are mapped as optional in YAML.
 - added operator<< for Symbol & SymbolLocation, for debugging

Reviewers: ioeric, hokein

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D42942

llvm-svn: 324735
2018-02-09 14:42:01 +00:00
Ilya Biryukov b6ad25cd4c [clangd] Fix crash when CompilerInvocation can't be created.
Summary:
This can happen if the CompileCommand provided by compilation database
is malformed.

Reviewers: hokein, ioeric, sammccall

Reviewed By: sammccall

Subscribers: klimek, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D43122

llvm-svn: 324732
2018-02-09 13:51:57 +00:00
Ilya Biryukov 7aca05102c [clangd] Update include guard in Annotations.h. NFC
llvm-svn: 324599
2018-02-08 12:46:34 +00:00
Ilya Biryukov 7e5ee26d1a Resubmit "[clangd] The new threading implementation"
Initially submitted as r324356 and reverted in r324386.

This change additionally contains a fix to crashes of the buildbots.
The source of the crash was undefined behaviour caused by
std::future<> whose std::promise<> was destroyed without calling
set_value().

llvm-svn: 324575
2018-02-08 07:37:35 +00:00
Eric Liu 7deda4f5af [clangd] Do not precent-encode numbers in URI.
Reviewers: ilya-biryukov

Subscribers: klimek, jkorous-apple, cfe-commits, sammccall

Differential Revision: https://reviews.llvm.org/D43009

llvm-svn: 324475
2018-02-07 12:12:06 +00:00
Ilya Biryukov 3693f5941a Revert "[clangd] The new threading implementation" (r324356)
And the follow-up changes r324361 and r324363.
These changes seem to break two buildbots:
  - http://lab.llvm.org:8011/builders/clang-atom-d525-fedora-rel/builds/14091
  - http://lab.llvm.org:8011/builders/clang-x86_64-linux-selfhost-modules-2/builds/16001

We will need to investigate what went wrong and resubmit the changes
afterwards.

llvm-svn: 324386
2018-02-06 19:22:40 +00:00
Ilya Biryukov b9cf83b16f [clangd] Fixed compilation on Windows buildbot.
llvm-svn: 324361
2018-02-06 16:32:36 +00:00
Eric Liu 7f24765912 [clangd] Use URIs in index symbols.
Reviewers: hokein, sammccall

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D42915

llvm-svn: 324358
2018-02-06 16:10:35 +00:00
Ilya Biryukov cce8883094 [clangd] The new threading implementation
Summary:
In the new threading model clangd creates one thread per file to manage
the AST and one thread to process each of the incoming requests.
The number of actively running threads is bounded by the semaphore to
avoid overloading the system.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, mgorny, jkorous-apple, ioeric, hintonda, cfe-commits

Differential Revision: https://reviews.llvm.org/D42573

llvm-svn: 324356
2018-02-06 15:53:42 +00:00
Haojian Wu 3b8e00c5e0 [clangd] Fix incorrect file path for symbols defined by the compile command-line option.
Summary:

Reviewers: ioeric

Reviewed By: ioeric

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D42913

llvm-svn: 324328
2018-02-06 09:50:35 +00:00
Sam McCall c40fdcdd7e [clangd] Fix ExternC test broken by r324081
llvm-svn: 324105
2018-02-02 17:01:36 +00:00
Eric Liu cf1773826f [clangd] Skip inline namespace when collecting scopes for index symbols.
Summary:
Some STL symbols are defined in inline namespaces. For example,
```
namespace std {
inline namespace __cxx11 {
 typedef ... string;
}
}
```
Currently, this will be `std::__cxx11::string`; however, `std::string` is desired.

Inline namespaces are treated as transparent scopes. This
reflects the way they're most commonly used for lookup. Ideally we'd
include them, but at query time it's hard to find all the inline
namespaces to query: the preamble doesn't have a dedicated list.

Reviewers: sammccall, hokein

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D42796

llvm-svn: 324065
2018-02-02 10:31:42 +00:00
Haojian Wu a8d12bbc3f [clangd] remove the unused code NFC.
llvm-svn: 323960
2018-02-01 13:06:58 +00:00
Eric Liu cda2526d86 [clangd] Fix URI scheme conflict and an unused variable warning in tests. NFC
llvm-svn: 323957
2018-02-01 12:44:52 +00:00
Sam McCall d1a7a37c22 [clangd] Pass Context implicitly using TLS.
Summary:
Instead of passing Context explicitly around, we now have a thread-local
Context object `Context::current()` which is an implicit argument to
every function.
Most manipulation of this should use the WithContextValue helper, which
augments the current Context to add a single KV pair, and restores the
old context on destruction.

Advantages are:
- less boilerplate in functions that just propagate contexts
- reading most code doesn't require understanding context at all, and
  using context as values in fewer places still
- fewer options to pass the "wrong" context when it changes within a
  scope (e.g. when using Span)
- contexts pass through interfaces we can't modify, such as VFS
- propagating contexts across threads was slightly tricky (e.g.
  copy vs move, no move-init in lambdas), and is now encapsulated in
  the threadpool

Disadvantages are all the usual TLS stuff - hidden magic, and
potential for higher memory usage on threads that don't use the
context. (In practice, it's just one pointer)

Reviewers: ilya-biryukov

Subscribers: klimek, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D42517

llvm-svn: 323872
2018-01-31 13:40:48 +00:00
Haojian Wu b018906051 [clangd] Better handling symbols defined in macros.
Summary:
For symbols defined inside macros:
 * use expansion location, if the symbol is formed via macro concatenation.
 * use spelling location, otherwise.

This will fix some symbols that have ill-format location (especial invalid filepath).

Reviewers: ioeric

Reviewed By: ioeric

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D42575

llvm-svn: 323867
2018-01-31 12:56:51 +00:00
Ilya Biryukov 75f1dd9b98 [clangd] Refactored threading in ClangdServer
Summary:
We now provide an abstraction of Scheduler that abstracts threading
and resource management in ClangdServer.
No changes to behavior are intended with an exception of changed error
messages.
This patch is preliminary work to allow a revamped threading
implementation that will move the threading code out of CppFile.

Reviewers: sammccall, bkramer, jkorous-apple

Reviewed By: sammccall

Subscribers: hokein, mgorny, hintonda, ioeric, jkorous-apple, cfe-commits, klimek

Differential Revision: https://reviews.llvm.org/D42174

llvm-svn: 323851
2018-01-31 08:51:16 +00:00
Eric Liu b57da098e9 [clangd] Fix r323658 test failure on windows.
llvm-svn: 323703
2018-01-29 22:28:08 +00:00
Eric Liu 78ed91a76c [clangd] Use new URI with scheme support in place of the existing LSP URI
Summary:
o Replace the existing clangd::URI with a wrapper of FileURI which also
carries a resolved file path.
o s/FileURI/URI/
o Get rid of the URI hack in vscode extension.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D42419

llvm-svn: 323660
2018-01-29 15:37:46 +00:00
Eric Liu 278e2d1a26 [clangd] Add a fallback directory for collected symbols with relative paths.
Reviewers: hokein, sammccall

Subscribers: klimek, ilya-biryukov, jkorous-apple, cfe-commits

Differential Revision: https://reviews.llvm.org/D42638

llvm-svn: 323658
2018-01-29 15:13:29 +00:00
Sam McCall 1b475a1ad0 [clangd] Modify the Span API so that Spans propagate with contexts.
Summary:
This is probably the right behavior for distributed tracers, and makes unpaired
begin-end events impossible without requiring Spans to be bound to a thread.

The API is conceptually clean but syntactically awkward. As discussed offline,
this is basically a naming problem and will go away if (when) we use TLS to
store the current context.

The apparently-unrelated change to onScopeExit are because its move semantics
broken if Func is POD-like since r322838. This is true of function pointers,
and the lambda I use here that captures two pointers only.
I've raised this issue on llvm-dev and will revert this part if we fix it in
some other way.

Reviewers: ilya-biryukov

Subscribers: klimek, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D42499

llvm-svn: 323511
2018-01-26 09:00:30 +00:00
Sam McCall 034e11aca5 [clangd] Add ClangdUnit diagnostics tests using annotated code.
Summary:
This adds checks that our diagnostics emit correct ranges in a bunch of cases,
as promised in D41118.

The diagnostics-preamble test is also converted and extended to be a little more
precise.

diagnostics.test stays around as the smoke test for this feature.

Reviewers: ilya-biryukov

Subscribers: klimek, mgorny, cfe-commits

Differential Revision: https://reviews.llvm.org/D41454

llvm-svn: 323448
2018-01-25 17:29:17 +00:00
Ilya Biryukov df84234364 [clangd] Provide a helper to report estimated memory usage per-file
Reviewers: sammccall, ioeric, hokein

Reviewed By: ioeric

Subscribers: klimek, cfe-commits, jkorous-apple

Differential Revision: https://reviews.llvm.org/D42480

llvm-svn: 323425
2018-01-25 14:32:21 +00:00
Haojian Wu 58d208d6e4 [clangd] Disable typo correction when doing code completion.
Reviewers: ilya-biryukov

Reviewed By: ilya-biryukov

Subscribers: klimek, jkorous-apple, cfe-commits, ioeric

Differential Revision: https://reviews.llvm.org/D42491

llvm-svn: 323410
2018-01-25 09:44:06 +00:00
Haojian Wu 48b4865683 [clangd] Limit completion results.
Summary:
* truncate symbols from static/dynamic index to the limited number
(which would save lots of cost in constructing the merged symbols).
* add an CLI option allowing to limit the number of returned completion results.
(default to 100)

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, jkorous-apple, ioeric, cfe-commits

Differential Revision: https://reviews.llvm.org/D42484

llvm-svn: 323408
2018-01-25 09:20:09 +00:00
Sam McCall d5ea3e32a6 [clangd] add test for r323347 CodeComplete behavior we rely on
llvm-svn: 323350
2018-01-24 17:53:32 +00:00
Haojian Wu 061c73eb28 [clangd] Use accessible scopes to query indexes for global code completion.
Summary:
 * For qualified completion (foo::a^)
       * unresolved qualifier - use global namespace ("::")
       * resolved qualifier - use all accessible namespaces inside the resolved qualifier.
 * For unqualified completion (vec^), use scopes that are accessible from the
      scope from which code completion occurs.

Reviewers: sammccall, ilya-biryukov

Reviewed By: sammccall

Subscribers: jkorous-apple, ioeric, klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D42073

llvm-svn: 323189
2018-01-23 11:37:26 +00:00
Sam McCall d2a9592526 [clangd] Drop ~destructor completions - rarely helpful and work inconsistently
llvm-svn: 323149
2018-01-22 21:05:00 +00:00
Eric Liu 43cee17cbf [clangd] Try to fix build bot failue in r323101
llvm-svn: 323111
2018-01-22 13:03:25 +00:00
Eric Liu f5b8c82198 [clangd] Add support for different file URI schemas.
Summary: I will replace the existing URI struct in Protocol.h with the new URI and rename FileURI to URI in a followup patch.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: jkorous-apple, klimek, mgorny, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41946

llvm-svn: 323101
2018-01-22 11:48:20 +00:00
Sam McCall 8b2faeed02 [clangd] Change index scope convention from "outer::inner" to "outer::inner::"
Global scope is "" (was "")
Top-level namespace scope is "ns::" (was "ns")
Nested namespace scope is "ns::ns::" (was "ns::ns")

This composes more naturally:
- qname = scope + name
- full scope = resolved scope + unresolved scope (D42073 was the trigger)
It removes a wart from the old way: "foo::" has one more separator than "".

Another alternative that has these properties is "::ns", but that lacks
the property that both the scope and the name are substrings of the
qname as produced by clang.

This is re-landing r322996 which didn't build.

llvm-svn: 323000
2018-01-19 22:18:21 +00:00
Sam McCall 50584ea197 Revert "[clangd] Change index scope convention from "outer::inner" to "outer::inner::""
This reverts commit r322996.

llvm-svn: 322998
2018-01-19 22:09:34 +00:00
Sam McCall 1a8c55ecc6 [clangd] Change index scope convention from "outer::inner" to "outer::inner::"
Global scope is "" (was "")
Top-level namespace scope is "ns::" (was "ns")
Nested namespace scope is "ns::ns::" (was "ns::ns")

This composes more naturally:
 - qname = scope + name
 - full scope = resolved scope + unresolved scope (D42073 was the trigger)
It removes a wart from the old way: "foo::" has one more separator than "".

Another alternative that has these properties is "::ns", but that lacks
the property that both the scope and the name are substrings of the
qname as produced by clang.

llvm-svn: 322996
2018-01-19 21:58:58 +00:00
Sam McCall 545a20d90b [clangd] Merge index-provided completions with those from Sema.
Summary:
 - we match on USR, and do a field-by-field merge if both have results
 - scoring is post-merge, with both sets of information available
   (for now, sema priority is used if available, static score for index results)
 - limit is applied to the complete result set (previously index ignored limit)
 - CompletionItem is only produces for the returned results
 - If the user doesn't type a scope, we send the global scope for completion
   (we can improve this after D42073)

Reviewers: ioeric

Subscribers: klimek, ilya-biryukov, mgrang, cfe-commits

Differential Revision: https://reviews.llvm.org/D42181

llvm-svn: 322945
2018-01-19 14:34:02 +00:00
Haojian Wu 9873fdd4f5 [clangd] Collect enum constants in SymbolCollector
Summary:
* ignore nameless symbols
* include enum constant declarataion

Reviewers: ilya-biryukov, jkorous-apple

Reviewed By: ilya-biryukov

Subscribers: ioeric, cfe-commits, klimek

Differential Revision: https://reviews.llvm.org/D42074

llvm-svn: 322929
2018-01-19 09:35:55 +00:00
Sam McCall d8169a85cc [clangd] Remove unused IncludeGlobals completion option, always pass true to sema
llvm-svn: 322856
2018-01-18 15:31:30 +00:00
Sam McCall a15c2d683e [clangd] CodeCompleteTests cleanup: naming, ordering, helpers. NFC
llvm-svn: 322824
2018-01-18 09:27:56 +00:00
Sam McCall a5ce243fa3 [clangd] Use fuzzy match to select top N index results.
Summary:
This makes performance slower but more predictable (it always processes
every symbol). We need to find ways to make this fast, possibly by precomputing
short queries or capping the number of scored results. But our current approach
is too naive.

It also no longer returns results in a "good" order. In fact it's pathological:
the top N results are ranked from worst to best. Indexes aren't responsible for
ranking and MergedIndex can't do a good job, so I'm pleased that this will make
any hidden assumptions we have more noticeable :-)

Reviewers: hokein

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D42060

llvm-svn: 322821
2018-01-18 08:35:04 +00:00
Sam McCall 2c3849af22 [clangd] Avoid combinatorial explosion in CodeCompleteTests.
Summary:
This test dominates our unit test runtime, and the change speeds it up by 10x.
We lose coverage of some combinations of flags, but I'm not sure that's finding
many bugs.

3300 -> 300ms on my machine (3800 -> 800ms for the whole of CompletionTest).

Reviewers: ilya-biryukov

Subscribers: klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D42063

llvm-svn: 322547
2018-01-16 12:21:24 +00:00
Sam McCall 0faecf0c33 [clangd] Merge results from static/dynamic index.
Summary:
We now hide the static/dynamic split from the code completion, behind a
new implementation of the SymbolIndex interface. This will reduce the
complexity of the sema/index merging that needs to be done by
CodeComplete, at a fairly small cost in flexibility.

Reviewers: hokein

Subscribers: klimek, mgorny, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D42049

llvm-svn: 322480
2018-01-15 12:33:00 +00:00
Sam McCall 09b4be51a8 [clangd] Fix uninitialized-read found by asan
llvm-svn: 322443
2018-01-13 16:46:26 +00:00
Sam McCall 3d139c527b [clangd] Code completion uses Sema for NS-level things in the current file.
Summary:
To stay fast, it avoids deserializing anything outside the current file, by
disabling the LoadExternal code completion option added in r322377, when the
index is enabled.

Reviewers: hokein

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41996

llvm-svn: 322387
2018-01-12 18:30:08 +00:00
Sam McCall 93483e7f39 [clangd] Include debugging tags for both static and dynamic index. NFC
llvm-svn: 322379
2018-01-12 17:09:49 +00:00
Sam McCall 84652cc63c [clangd] Incorporate fuzzy-match into result rankings.
Summary: The scoring function is fuzzy-match-quality * existing quality score.

Reviewers: ioeric

Subscribers: klimek, cfe-commits, ilya-biryukov

Differential Revision: https://reviews.llvm.org/D40780

llvm-svn: 322377
2018-01-12 16:16:09 +00:00
Haojian Wu f4374f087f [clangd] Don't navigate to forward class declaration when go to definition.
Summary:
For some cases, GoToDefinition will navigate to the forward class
declaration, we should always navigate to the class definition.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41661

llvm-svn: 322370
2018-01-12 14:21:10 +00:00
Ilya Biryukov 9555839228 [clangd] Pass Context to onDiagnosticsReady callback
Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D41907

llvm-svn: 322199
2018-01-10 17:59:27 +00:00
Eric Liu 9af958fbf8 [clangd] Add more filters for collected symbols.
Summary:
o We only collect symbols in namespace or translation unit scopes.
o Add an option to only collect symbols in included headers.

Reviewers: hokein, ilya-biryukov

Reviewed By: hokein, ilya-biryukov

Subscribers: klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D41823

llvm-svn: 322193
2018-01-10 14:57:58 +00:00
Haojian Wu ba28e9ac5d [clangd] Add static index for the global code completion.
Summary:
Use the YAML-format symbols (generated by the global-symbol-builder tool) to
do the global code completion.
It is **experimental** only , but it allows us to experience global code
completion on a relatively small project.

Tested with LLVM project.

Reviewers: sammccall, ioeric

Reviewed By: sammccall, ioeric

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41668

llvm-svn: 322191
2018-01-10 14:44:34 +00:00
Ilya Biryukov f60bf347c9 [clangd] Remove duplicates from code completion
Summary:
This patch removes hidden items from code completion.
Items can be hidden, e.g., by other items in the child scopes.
This patch addresses a particular problem of a duplicate completion
item for the class in the following example:

    struct Adapter { void method(); };
    void Adapter::method() {
       Adapter^
    }

We should probably investigate if there are other duplicates in
completion and remove them, possibly adding assertions that it never
happens.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: cfe-commits, klimek

Differential Revision: https://reviews.llvm.org/D41901

llvm-svn: 322185
2018-01-10 13:51:09 +00:00
Eric Liu 76f6b44443 [clangd] Add more symbol information for code completion.
Reviewers: hokein, sammccall

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41345

llvm-svn: 322097
2018-01-09 17:32:00 +00:00
Eric Liu 5732c0e6ea [clangd] Use ToolExecutor to write the global-symbol-builder tool.
Summary:
This enables more execution modes like standalone and Mapreduce-style execution.

See also D41729

Reviewers: hokein, sammccall

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41730

llvm-svn: 322084
2018-01-09 15:21:45 +00:00
Haojian Wu fe22b7474b [clangd] Catch more symbols in SymbolCollector.
Summary:
We currently only collect external-linkage symbols in the collector,
which results in missing some typical symbols (like no-linkage type alias symbols).

This patch relaxes the constraint a bit to allow collecting more symbols.

Reviewers: ilya-biryukov

Reviewed By: ilya-biryukov

Subscribers: klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D41759

llvm-svn: 322067
2018-01-09 10:44:09 +00:00
Ilya Biryukov 5a5e1caf25 [clangd] Properly set filterText for index-based completion items
It was previously set to an identifier that the user typed, leading to
surprising behavior in VSCode (probably in other editors too).

llvm-svn: 321554
2017-12-29 14:59:22 +00:00
Sam McCall 4b9bbb378b [clangd] Use Builder for symbol slabs, and use sorted-vector for storage
Summary:
This improves a few things:
 - the insert -> freeze -> read sequence is now enforced/communicated by the
   type system
 - SymbolSlab::const_iterator iterates over symbols, not over id-symbol pairs
 - we avoid permanently storing a second copy of the IDs, and the
   string map's hashtable

The slab size is now down to 21.8MB for the LLVM project.
Of this only 2.7MB is strings, the rest is #symbols * `sizeof(Symbol)`.
`sizeof(Symbol)` is currently 96, which seems too big - I think
SymbolInfo isn't efficiently packed. That's a topic for another patch!

Also added simple API to see the memory usage/#symbols of a slab, since
it seems likely we will continue to care about this.

Reviewers: ilya-biryukov

Subscribers: klimek, mgrang, cfe-commits

Differential Revision: https://reviews.llvm.org/D41506

llvm-svn: 321412
2017-12-23 19:38:03 +00:00
Sam McCall 6c0d0f5775 [clangd] Index symbols share storage within a slab.
Summary:
Symbols are not self-contained - it's only safe to hand them out if you
guarantee the lifetime of the underlying data.

Before this lands, I'm going to measure the before/after memory usage of the
LLVM index loaded into memory in a single slab.

Reviewers: hokein

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41483

llvm-svn: 321272
2017-12-21 14:58:44 +00:00
Eric Liu 63696e14e3 [clangd] Pull CodeCompletionString handling logic into its own file and add unit test.
Reviewers: sammccall

Subscribers: klimek, mgorny, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41450

llvm-svn: 321193
2017-12-20 17:24:31 +00:00
Eric Liu 188adaf46b [clangd] Remove an unused lambda capture.
llvm-svn: 321192
2017-12-20 17:22:56 +00:00
Sam McCall 328cbdb9e4 [clangd] Switch xrefs and documenthighlight to annotated-code unit tests. NFC
Summary:
The goal here is again to make it easier to read and write the tests.

I've extracted `parseTextMarker` from CodeCompleteTests into an `Annotations`
class, adding features to it:
  - as well as points `^s` it allows ranges `[[...]]`
  - multiple points and ranges are supported
  - points and ranges may be named: `$name^` and `$name[[...]]`

These features are used for the xrefs tests. This also paves the way for
replacing the lit diagnostics.test with more readable unit tests, using named
ranges.

Alternative considered: `TestSelectionRange` in clang-refactor/TestSupport
Main problems were:
 - delimiting the end of ranges is awkward, requiring counting
 - comment syntax is long and at least as cryptic for most cases
 - no separate syntax for point vs range, which keeps xrefs tests concise
 - Still need to convert to Position everywhere
 - Still need helpers for common case of expecting exactly one point/range

(I'll probably promote the extra `PrintTo`s from some of the core Protocol types
into `operator<<` in `Protocol.h` itself in a separate, prior patch...)

Reviewers: ioeric

Subscribers: klimek, mgorny, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41432

llvm-svn: 321184
2017-12-20 16:06:05 +00:00
Sam McCall fffa8229e3 [clangd] Add debug printers for basic protocol types. NFC
llvm-svn: 321161
2017-12-20 10:26:53 +00:00
Eric Liu 125cda7d3b [clangd] Igore cases in index fuzzy find.
llvm-svn: 321157
2017-12-20 09:29:54 +00:00
Eric Liu bfac8f782c [clangd] Build dynamic index and use it for code completion.
Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41289

llvm-svn: 321092
2017-12-19 18:00:37 +00:00
Sam McCall e3e1570695 [clangd] Fix warnings/compiler pickiness after r321083
llvm-svn: 321086
2017-12-19 17:05:00 +00:00
Eric Liu 6f648df1b9 [clangd] Index-based code completion.
Summary: Use symbol index to populate completion results for qualfified IDs e.g. "nx::A^".

Reviewers: ilya-biryukov, sammccall

Reviewed By: ilya-biryukov, sammccall

Subscribers: rwols, klimek, mgorny, cfe-commits, sammccall

Differential Revision: https://reviews.llvm.org/D41281

llvm-svn: 321083
2017-12-19 16:50:37 +00:00
Sam McCall b536a2a5ba [clangd] Expose offset <-> LSP position functions, and fix bugs
Summary:
- Moved these functions to SourceCode.h
- added unit tests
- fix off by one in positionToOffset: Offset - 1 in final calculation was wrong
- fixed formatOnType which had an equal and opposite off-by-one
- positionToOffset and offsetToPosition both consistently clamp to beginning/end
  of file when input is out of range
- gave variables more descriptive names
- removed windows line ending fixmes where there is nothing to fix
- elaborated on UTF-8 fixmes

This will conflict with Eric's D41281, but in a pretty easy-to-resolve way.

Reviewers: ioeric

Subscribers: klimek, mgorny, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41351

llvm-svn: 321073
2017-12-19 12:23:48 +00:00
Eric Liu 4feda80a43 [clangd] Support filtering by fixing scopes in fuzzyFind.
Summary: When scopes are specified, only match symbols from scopes.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41367

llvm-svn: 321067
2017-12-19 11:37:40 +00:00
Sam McCall 800d4371f3 [clangd] Add unit tests for signature help. SigHelp/CodeComplete lit tests are smoke only.
llvm-svn: 321065
2017-12-19 10:29:27 +00:00
Eric Liu eea1633878 [clangd] Build in-memory index on symbols in files.
Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, mgorny, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41276

llvm-svn: 320807
2017-12-15 12:25:02 +00:00
Ilya Biryukov ee27d2ebae [clangd] Implemented tracing using Context
Reviewers: sammccall, ioeric, hokein

Reviewed By: sammccall

Subscribers: klimek, luckygeck, cfe-commits

Differential Revision: https://reviews.llvm.org/D40488

llvm-svn: 320706
2017-12-14 15:04:59 +00:00
Eric Liu d293bf127a [clangd] Add a FileSymbols container that manages symbols from multiple files.
Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, mgorny, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D41232

llvm-svn: 320701
2017-12-14 14:50:58 +00:00
Eric Liu a2d4607a4b [clangd] Fix a potential use-after-move bug.
llvm-svn: 320695
2017-12-14 12:31:04 +00:00
Haojian Wu 56a5fca473 [clangd] Construct SymbolSlab from YAML format.
Summary: This will be used together with D40548 for the global index source (experimental).

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: klimek, mgorny, ilya-biryukov, cfe-commits, ioeric

Differential Revision: https://reviews.llvm.org/D41178

llvm-svn: 320694
2017-12-14 12:17:14 +00:00
Eric Liu 3732cadc73 [clangd] Symbol index interfaces and an in-memory index implementation.
Summary:
o Index interfaces to support using different index sources (e.g. AST index, global index) for code completion, cross-reference finding etc. This patch focuses on code completion.

The following changes in the original patch has been split out.
o Implement an AST-based index.
o Add an option to replace sema code completion for qualified-id with index-based completion.
o Implement an initial naive code completion index which matches symbols that have the query string as substring.

Reviewers: malaperle, sammccall

Reviewed By: sammccall

Subscribers: hokein, klimek, malaperle, mgorny, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D40548

llvm-svn: 320688
2017-12-14 11:25:49 +00:00
Haojian Wu 03e2bd76c8 [clangd] Fix the unitttest build error on buildbot.
llvm-svn: 320678
2017-12-14 09:20:21 +00:00
Ilya Biryukov 5a85b8e6dd [clangd] clang-format the source code. NFC
llvm-svn: 320577
2017-12-13 12:53:16 +00:00
Ilya Biryukov 940901e8b1 [clangd] Implemented logging using Context
Reviewers: sammccall, ioeric, hokein

Reviewed By: sammccall

Subscribers: klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D40486

llvm-svn: 320576
2017-12-13 12:51:22 +00:00
Haojian Wu 4c1394d67d [clangd] Introduce a "Symbol" class.
Summary:
* The "Symbol" class represents a C++ symbol in the codebase, containing all the
  information of a C++ symbol needed by clangd. clangd will use it in clangd's
  AST/dynamic index and global/static index (code completion and code
  navigation).
* The SymbolCollector (another IndexAction) will be used to recollect the
  symbols when the source file is changed (for ASTIndex), or to generate
  all C++ symbols for the whole project.

In the long term (when index-while-building is ready), clangd should share a
same "Symbol" structure and IndexAction with index-while-building, but
for now we want to have some stuff working in clangd.

Reviewers: ioeric, sammccall, ilya-biryukov, malaperle

Reviewed By: sammccall

Subscribers: malaperle, klimek, mgorny, cfe-commits

Differential Revision: https://reviews.llvm.org/D40897

llvm-svn: 320486
2017-12-12 15:42:10 +00:00
Ilya Biryukov 9b5ffc22f6 [clangd] clang-format the code. NFC
llvm-svn: 320476
2017-12-12 12:56:46 +00:00
Ilya Biryukov 657159c273 [clangd] Introduced a Context that stores implicit data
Summary:
It will be used to pass around things like Logger and Tracer throughout
clangd classes.

Reviewers: sammccall, ioeric, hokein, bkramer

Reviewed By: sammccall

Subscribers: klimek, bkramer, mgorny, cfe-commits

Differential Revision: https://reviews.llvm.org/D40485

llvm-svn: 320468
2017-12-12 11:16:45 +00:00
Sam McCall 44fdcec26f [clangd] Convert lit code completion tests to unit-tests. NFC
Summary: This improves readability of tests and error messages.

Reviewers: ioeric

Subscribers: klimek, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D40952

llvm-svn: 320148
2017-12-08 15:00:59 +00:00
Shoaib Meenai d806af3499 [CMake] Use PRIVATE in target_link_libraries for executables
We currently use target_link_libraries without an explicit scope
specifier (INTERFACE, PRIVATE or PUBLIC) when linking executables.
Dependencies added in this way apply to both the target and its
dependencies, i.e. they become part of the executable's link interface
and are transitive.

Transitive dependencies generally don't make sense for executables,
since you wouldn't normally be linking against an executable. This also
causes issues for generating install export files when using
LLVM_DISTRIBUTION_COMPONENTS. For example, clang has a lot of LLVM
library dependencies, which are currently added as interface
dependencies. If clang is in the distribution components but the LLVM
libraries it depends on aren't (which is a perfectly legitimate use case
if the LLVM libraries are being built static and there are therefore no
run-time dependencies on them), CMake will complain about the LLVM
libraries not being in export set when attempting to generate the
install export file for clang. This is reasonable behavior on CMake's
part, and the right thing is for LLVM's build system to explicitly use
PRIVATE dependencies for executables.

Unfortunately, CMake doesn't allow you to mix and match the keyword and
non-keyword target_link_libraries signatures for a single target; i.e.,
if a single call to target_link_libraries for a particular target uses
one of the INTERFACE, PRIVATE, or PUBLIC keywords, all other calls must
also be updated to use those keywords. This means we must do this change
in a single shot. I also fully expect to have missed some instances; I
tested by enabling all the projects in the monorepo (except dragonegg),
and configuring both with and without shared libraries, on both Darwin
and Linux, but I'm planning to rely on the buildbots for other
configurations (since it should be pretty easy to fix those).

Even after this change, we still have a lot of target_link_libraries
calls that don't specify a scope keyword, mostly for shared libraries.
I'm thinking about addressing those in a follow-up, but that's a
separate change IMO.

Differential Revision: https://reviews.llvm.org/D40823

llvm-svn: 319840
2017-12-05 21:49:56 +00:00
Sam McCall f6ae323ddf [clangd] Clean up code complete unit tests. NFC
llvm-svn: 319820
2017-12-05 20:11:29 +00:00
Ilya Biryukov d3b04e3517 [clangd] Set completion options per-request.
Summary:
Previously, completion options were set per ClangdServer instance.
It will allow to change completion preferences during the lifetime
of a single ClangdServer instance.

Also rewrote ClangdCompletionTest.CompletionOptions to reuse single
ClangdServer instance, the test now runs 2x faster on my machine.

Reviewers: sammccall, ioeric, hokein

Reviewed By: sammccall, ioeric

Subscribers: klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D40654

llvm-svn: 319753
2017-12-05 10:42:57 +00:00
Sam McCall 93cd99173f [clangd] Remove unused test param. NFC
llvm-svn: 319742
2017-12-05 07:34:35 +00:00
Sam McCall 9aad25f193 [clangd] Split code-completion tests out of ClangdTests. NFC.
Summary:
Common parts are mostly FS related, so pulled out TestFS.h for the common stuff.
Deliberately resisted cleaning up much here, so this is pretty mechanical.

Reviewers: hokein

Subscribers: klimek, mgorny, ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D40784

llvm-svn: 319741
2017-12-05 07:20:26 +00:00
Sam McCall ecbeab0ea1 [clangd] GlobalCompilationDatabase interface changes
Summary:
- GlobalCompilationDatabase now returns a single command (that's all we use)
- fallback flags are now part of the GlobalCompilationDatabase.
  There's a default implementation that they can optionally customize.
- this allows us to avoid invoking the fallback logic on two separate codepaths
- race on extra flags fixed by locking the mutex
- made GCD const-correct (DBGCD does have mutating methods)

Reviewers: hokein

Subscribers: klimek, cfe-commits, ilya-biryukov

Differential Revision: https://reviews.llvm.org/D40733

llvm-svn: 319647
2017-12-04 10:08:45 +00:00
Sam McCall 87496417ff [clangd] Fuzzy match scorer
Summary:
This will be used for rescoring code completion results based on partial
identifiers.
Short-term use:
  - we want to limit the number of code completion results returned to
  improve performance of global completion. The scorer will be used to
  rerank the results to return when the user has applied a filter.
Long-term use case:
  - ranking of completion results from in-memory index
  - merging of completion results from multiple sources (merging usually
  works best when done at the component-score level, rescoring the
  fuzzy-match quality avoids different backends needing to have
  comparable scores)

Reviewers: ilya-biryukov

Subscribers: cfe-commits, mgorny

Differential Revision: https://reviews.llvm.org/D40060

llvm-svn: 319557
2017-12-01 17:08:02 +00:00
Sam McCall 9fbecd5b0d [clangd] Filter completion results by fuzzy-matching identifiers.
Summary:
This allows us to limit the number of results we return and still allow them
to be surfaced by refining a query (D39852).

The initial algorithm is very conservative - it accepts a completion if the
filter is any case-insensitive sub-sequence. It does not attempt to rank items
based on match quality.

Reviewers: ilya-biryukov

Subscribers: cfe-commits

Differential Revision: https://reviews.llvm.org/D39882

llvm-svn: 319552
2017-12-01 16:35:50 +00:00
Sam McCall ff8b874548 [clangd] New conventions for JSON-marshalling functions, centralize machinery
Summary:
 - JSON<->Obj interface is now ADL functions, so they play nicely with enums
 - recursive vector/map parsing and ObjectMapper moved to JSONExpr and tested
 - renamed (un)parse to (de)serialize, since text -> JSON is called parse
 - Protocol.cpp gets a bit shorter

Sorry for the giant patch, it's prety mechanical though

Reviewers: ilya-biryukov

Subscribers: klimek, cfe-commits

Differential Revision: https://reviews.llvm.org/D40596

llvm-svn: 319478
2017-11-30 21:32:29 +00:00
Sam McCall ec109029b1 [clangd] Switch from YAMLParser to JSONExpr
Summary:
 - Converted Protocol.h parse() functions to take JSON::Expr.
   These no longer detect and log unknown fields, as this is not that
   useful and no longer free.
   I haven't changed the error handling too much: fields that were
   treated as optional before are still optional, even when it's wrong.
   Exception: object properties with the wrong type are now ignored.
 - Made JSONRPCDispatcher parse using json::parse
 - The bug where 'method' must come before 'params' in the stream is
 fixed as a side-effect. (And the same bug in executeCommand).
 - Some parser crashers fixed as a side effect.
   e.g. https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=3890
 - The debug stream now prettyprints the input messages with --pretty.
 - Request params are attached to traces when tracing is enabled.
 - Fixed some bugs in tests (errors tolerated by YAMLParser, and
 off-by-ones in Content-Length that our null-termination was masking)
 - Fixed a random double-escape bug in ClangdLSPServer (it was our last
 use of YAMLParser!)

Reviewers: ilya-biryukov

Subscribers: cfe-commits

Differential Revision: https://reviews.llvm.org/D40406

llvm-svn: 319159
2017-11-28 09:37:43 +00:00
Sam McCall 1bb272bc74 [clangd] Add missing (but documented!) JSONExpr typed accessors
Summary:
Noticed this when I tried to port the Protocol.h parsers.
And tests for the inspect API, which caught a small bug.

Reviewers: ioeric

Subscribers: ilya-biryukov

Differential Revision: https://reviews.llvm.org/D40399

llvm-svn: 319157
2017-11-28 09:25:09 +00:00
Sam McCall 9cfd9c9a9b [clangd] Tracing improvements
Summary:
[clangd] Tracing improvements

Compose JSON using JSONExpr
Allow attaching metadata to spans (and avoid it if tracing is off)
Attach IDs and responses of JSON RPCs to their spans

The downside is that large responses make the trace viewer sluggish.
We should make our responses less huge :-) Or fix trace viewer.

Reviewers: ilya-biryukov

Subscribers: cfe-commits

Differential Revision: https://reviews.llvm.org/D40132

llvm-svn: 318928
2017-11-23 17:12:04 +00:00
Sam McCall adccab64f2 [clangd] Drop impossible completions (unavailable or inaccessible)
Summary: (There must be some reason why D38077 didn't just do this, but I don't get it!)

Reviewers: ilya-biryukov

Subscribers: cfe-commits

Differential Revision: https://reviews.llvm.org/D39836

llvm-svn: 318925
2017-11-23 16:58:22 +00:00
Sam McCall cc713cf885 [clangd] avoid divide by literal zero to satisfy msvc
llvm-svn: 318798
2017-11-21 19:37:35 +00:00
Sam McCall b67ce00e4d [clangd] use u8 strings in tests to avoid problems on windows
llvm-svn: 318793
2017-11-21 19:10:22 +00:00
Sam McCall 40f191f724 [clangd] Add JSON tests with invalid unicode
llvm-svn: 318791
2017-11-21 18:40:43 +00:00
Sam McCall adbaebc242 [clangd] Add parsing and value inspection to JSONExpr.
Summary:
This will replace the places where we're using YAMLParser to parse JSON now:
  - the new marshalling code (T::parse()) should handle fewer cases and require
    fewer explicit casts
  - we'll early-reject invalid JSON that YAMLParser accepts
  - we'll be able to fix protocol-parsing bugs caused by the fact that YAML can
    only parse forward

I plan to do the conversion as soon as this lands, but I don't want it in one
patch as the protocol.cpp changes are conflict-prone.

Reviewers: ioeric

Subscribers: ilya-biryukov, cfe-commits

Differential Revision: https://reviews.llvm.org/D40182

llvm-svn: 318774
2017-11-21 16:00:53 +00:00
Ilya Biryukov e9eb7f0cb8 [clangd] Use in-memory preambles in clangd.
Reviewers: klimek, bkramer, sammccall

Reviewed By: sammccall

Subscribers: cfe-commits

Differential Revision: https://reviews.llvm.org/D39843

llvm-svn: 318412
2017-11-16 16:25:18 +00:00