Commit Graph

924 Commits

Author SHA1 Message Date
Sam McCall d7babe4e12 [clangd] When replying, log the method name and latency.
Summary:
This information is strictly available in the log (you can find the original
call) but it makes the log easier to follow in practice.

Reviewers: kadircet

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D53647

llvm-svn: 345150
2018-10-24 15:18:40 +00:00
Sam McCall e2f3a7348d [clangd] Ensure that we reply to each call exactly once. NFC (I think!)
Summary:
In debug builds, getting this wrong will trigger asserts.
In production builds, it will send an error reply if none was sent,
and drop redundant replies. (And log).

No tests because this is always a programming error.
(We did have some cases of this, but I fixed them with the new dispatcher).

Reviewers: ioeric

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, jfb, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53399

llvm-svn: 345144
2018-10-24 14:26:26 +00:00
Sam McCall e8437cbf5e [clangd] Don't show base class versions of members as completions.
Summary:
These are available via qualifiers, but signal to noise level is low.
Keep required quailifier machinery around though, for cross-ns completion.

Reviewers: ioeric

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53571

llvm-svn: 345141
2018-10-24 13:51:44 +00:00
Eric Liu 52a11b5662 [clangd] Downrank members from base class
Reviewers: sammccall, ilya-biryukov

Reviewed By: sammccall

Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53638

llvm-svn: 345140
2018-10-24 13:45:17 +00:00
Haojian Wu 40d5684d41 [clangd] Hide position line and column fields.
Reviewers: sammccall

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53577

llvm-svn: 345134
2018-10-24 12:56:41 +00:00
Ilya Biryukov 5a74a9cf5f [clangd] Simplify auto hover
Summary:
Use helper from clang. Also fixes some weird corner cases, e.g.
    auto (*foo)() = bar;

Reviewers: kadircet, hokein

Reviewed By: kadircet, hokein

Subscribers: ioeric, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D53347

llvm-svn: 345128
2018-10-24 10:09:34 +00:00
Ilya Biryukov ad430c6ac4 [clangd] Remove outdated comment-out code. NFC
llvm-svn: 345126
2018-10-24 09:47:24 +00:00
Sam McCall 16e7070e3e [clangd] Embed fixes as CodeAction, instead of clangd_fixes. Clean up serialization.
Summary:
CodeAction provides us with a standard way of representing fixes inline, so
use it, replacing our existing ad-hoc extension.

After this, it's easy to serialize diagnostics using the structured
toJSON/Protocol.h mechanism rather than assembling JSON ad-hoc.

Reviewers: hokein, arphaman

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53391

llvm-svn: 345119
2018-10-24 07:59:38 +00:00
Sam McCall 668ac94ba4 [clangd] Truncate SymbolID to 16 bytes.
Summary:
The goal is 8 bytes, which has a nonzero risk of collisions with huge indexes.
This patch should shake out any issues with truncation at all, we can lower
further later.

Reviewers: ioeric

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53587

llvm-svn: 345113
2018-10-24 06:58:42 +00:00
Sam McCall d1c9d1163b [clangd] Lazily create CDB, remove setCompileCommandsDir.
Summary:
The only way to actually set the directory is at initialize time,
so now CDB is lazy we can pass it to the constructor.

Reviewers: hokein

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53572

llvm-svn: 345031
2018-10-23 14:19:54 +00:00
Sam McCall 2172ee931c [clangd] Remove caching of compilation database commands.
Summary:
The CDB implementations used in open-source code are fast, and our private
slow CDB will soon do the relevant caching itself.

Simplifying the GlobalCDB layer in clangd is important to get auto-index
implemented at the right layer.

Reviewers: ioeric, ilya-biryukov

Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53439

llvm-svn: 345024
2018-10-23 13:14:02 +00:00
Sam McCall f01ad10a36 [clangd] Support passing a relative path to -compile-commands-dir
Summary: This is useful when using clangd with CMake based projects in Visual Studio Code since when using CMake the `compile_commands.json` file is usually located in a `build` subdirectory which isn't a parent directory of the source files. Allowing passing relative paths to -compile-commands-dir allows specifying `clangd.arguments = ["-compile-commands-dir=build"]` in VSCode's settings file and having it work for each CMake based project that uses the `build` subdirectory as the build directory (instead of having to specify the absolute path to the compile commands directory for each separate project in VSCode's settings).

Patch by Daan De Meyer!

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Tags: #clang-tools-extra

Differential Revision: https://reviews.llvm.org/D53481

llvm-svn: 345022
2018-10-23 11:54:36 +00:00
Sam McCall 7189112951 Fix range length comparison in DraftStore::UpdateDraft when Unicode characters are removed from the document
Summary:
See http://lists.llvm.org/pipermail/clangd-dev/2018-October/000171.html for context.

I kept the error (instead of downgrading to a log message) since the range lengths differing does indicate either a bug in the client or server range calculation or the buffers being out of sync (which both seems serious enough to me to be an error). If any existing clients aside from VSCode break they should only break when accidentally typing a Unicode character which should only be a minor nuisance for a little while until the bug is fixed in the respective client.

Patch by Daan De Meyer!

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: ilya-biryukov, ioeric, jkorous, arphaman, kadircet, cfe-commits

Tags: #clang-tools-extra

Differential Revision: https://reviews.llvm.org/D53527

llvm-svn: 345020
2018-10-23 11:51:53 +00:00
Eric Liu 0b70a87480 [clangd] Support URISchemes configuration in BackgroundIndex.
Reviewers: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53503

llvm-svn: 344912
2018-10-22 15:37:58 +00:00
Sam McCall 45b2754097 [clangd] Fix unqualified make_unique after r344850. NFC
llvm-svn: 344858
2018-10-20 17:40:12 +00:00
Sam McCall c008af6466 [clangd] Namespace style cleanup in cpp files. NFC.
Standardize on the most common namespace setup in our *.cpp files:
  using namespace llvm;
  namespace clang {
  namespace clangd {
  void foo(StringRef) { ... }
And remove redundant llvm:: qualifiers. (Except for cases like
make_unique where this causes problems with std:: and ADL).

This choice is pretty arbitrary, but some broad consistency is nice.
This is going to conflict with everything. Sorry :-/

Squash the other configurations:

A)
  using namespace llvm;
  using namespace clang;
  using namespace clangd;
  void clangd::foo(StringRef);
This is in some of the older files. (It prevents accidentally defining a
new function instead of one in the header file, for what that's worth).

B)
  namespace clang {
  namespace clangd {
  void foo(llvm::StringRef) { ... }
This is fine, but in practice the using directive often gets added over time.

C)
  namespace clang {
  namespace clangd {
  using namespace llvm; // inside the namespace
This was pretty common, but is a bit misleading: name lookup preferrs
clang::clangd::foo > clang::foo > llvm:: foo (no matter where the using
directive is).

llvm-svn: 344850
2018-10-20 15:30:37 +00:00
Simon Pilgrim 15ee23fc40 Fix MSVC "truncation from 'double' to 'float'" warning. NFCI.
llvm-svn: 344845
2018-10-20 13:20:26 +00:00
Simon Pilgrim ad28838111 Fix MSVC "not all control paths return a value" warning. NFCI.
llvm-svn: 344844
2018-10-20 13:18:49 +00:00
Sam McCall 0d9b40f583 [clangd] Set workspace root when initializing ClangdServer, disallow mutation.
Summary:
Rename instance variable to WorkspaceRoot to match what we call it internally.

Add fixme to set it automatically. Don't do it yet, clients have assumptions
that the constructor won't access the FS.

Don't second-guess the provided root.

Reviewers: ioeric

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53404

llvm-svn: 344787
2018-10-19 15:42:23 +00:00
Haojian Wu 812b6c51c3 [clangd] Remove the overflow log.
Summary:
LLVM codebase has generated files (all are build/Target/XXX/*.inc) that
exceed the MaxLine & MaxColumn. Printing these log would be noisy.

Reviewers: sammccall

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53400

llvm-svn: 344777
2018-10-19 08:35:24 +00:00
Krasimir Georgiev 9035420091 [clangd] Fix msan failure after r344735 by initializing bitfields
That revision changed integer members to bitfields; the integers were
default initialized before and the bitfields lost that default
initialization. This started causing msan use-of-uninitialized memory in
clangd tests.

llvm-svn: 344773
2018-10-19 06:05:32 +00:00
Haojian Wu 6ece6e7dad [clangd] Clear the semantic of RefSlab::size.
Summary:
The RefSlab::size can easily cause confusions, it returns the number of
different symbols, rahter than the number of all references.

- add numRefs() method and cache it, since calculating it everytime is nontrivial.
- clear misused places.

Reviewers: sammccall

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53389

llvm-svn: 344745
2018-10-18 15:33:20 +00:00
Sam McCall 3d0adbe636 [clangd] Enforce rules around "initialize" request, and create ClangdServer lazily.
Summary:
LSP is a slightly awkward map to C++ object lifetimes: the initialize request
is part of the protocol and provides information that doesn't change over the
lifetime of the server.

Until now, we handled this by initializing ClangdServer and ClangdLSPServer
right away, and making anything that can be set in the "initialize" request
mutable.
With this patch, we create ClangdLSPServer immediately, but defer creating
ClangdServer until "initialize". This opens the door to passing the relevant
initialize params in the constructor and storing them immutably.
(That change isn't actually done in this patch).

To make this safe, we have the MessageDispatcher enforce that the "initialize"
method is called before any other (as required by LSP). That way each method
handler can assume Server is initialized, as today.

As usual, while implementing this I found places where our test cases violated
the protocol.

Reviewers: ioeric

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53398

llvm-svn: 344741
2018-10-18 14:41:50 +00:00
Sam McCall 2c30fbcac5 [clangd] Lay JSONRPCDispatcher to rest.
Summary:
Most of its functionality is moved into ClangdLSPServer.
The decoupling between JSONRPCDispatcher, ProtocolCallbacks, ClangdLSPServer
was never real, and only served to obfuscate.

Some previous implicit/magic stuff is now explicit:
 - the return type of LSP method calls are now in the signature
 - no more reply() that gets the ID using global context magic
 - arg tracing no longer relies on RequestArgs::stash context magic either

This is mostly refactoring, but some deliberate fixes while here:
 - LSP method params are now by const reference
 - notifications and calls are now distinct namespaces.
   (some tests had protocol errors and needed updating)
 - we now reply to calls we failed to decode
 - outgoing calls use distinct IDs
A few error codes and message IDs changed in unimportant ways (see tests).

Reviewers: ioeric

Subscribers: mgorny, ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, jfb, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53387

llvm-svn: 344737
2018-10-18 12:32:04 +00:00
Eric Liu 4859738cfe [clangd] Names that are not spelled in source code are reserved.
Summary:
These are often not expected to be used directly e.g.
```
TEST_F(Fixture, X) {
  ^  // "Fixture_X_Test" expanded in the macro should be down ranked.
}
```

Only doing this for sema for now, as such symbols are mostly coming from sema
e.g. gtest macros expanded in the main file. We could also add a similar field
for the index symbol.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53374

llvm-svn: 344736
2018-10-18 12:23:05 +00:00
Haojian Wu b515fabb3b [clangd] Encode Line/Column as a 32-bits integer.
Summary:
This would buy us more memory. Using a 32-bits integer is enough for
most human-readable source code (up to 4M lines and 4K columns).

Previsouly, we used 8 bytes for a position, now 4 bytes, it would save
us 8 bytes for each Ref and each Symbol instance.

For LLVM-project binary index file, we save ~13% memory.

| Before | After |
| 412MB  | 355MB |

Reviewers: sammccall

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53363

llvm-svn: 344735
2018-10-18 10:43:50 +00:00
Mikael Holmen fa41add2e5 Fix warning about unused variable [NFC]
llvm-svn: 344724
2018-10-18 06:00:39 +00:00
Eric Liu 3fac4ef1fd [clangd] Support scope proximity in code completion.
Summary:
This should make all-scope completion more usable. Scope proximity for
indexes will be added in followup patch.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53131

llvm-svn: 344688
2018-10-17 11:19:02 +00:00
Haojian Wu c014d863be [clangd] Fix buildbot failure.
llvm-svn: 344680
2018-10-17 08:54:48 +00:00
Haojian Wu 0404855529 [clangd] Print numbers of symbols and refs as well when loading the
index.

llvm-svn: 344679
2018-10-17 08:48:04 +00:00
Haojian Wu 7dd4950ea5 [clangd] Collect refs from headers.
Summary:
Add a flag to SymbolCollector to collect refs fdrom headers.

Note that we collect refs from headers in static index, and we don't do it for
dynamic index because of the preamble (we skip function body in preamble,
collecting it will result incomplete results).

Reviewers: sammccall

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53322

llvm-svn: 344678
2018-10-17 08:38:36 +00:00
Sam McCall 4340cc7336 [clangd] Hide unused function. NFC
llvm-svn: 344676
2018-10-17 07:41:53 +00:00
Sam McCall 645203a8c5 [clangd] Rename and move trivial logger to Logger.cpp. NFC
llvm-svn: 344675
2018-10-17 07:39:32 +00:00
Sam McCall bf6a2fc329 [clangd] Simplify client capabilities parsing.
Summary:
Instead of parsing into structs that mirror LSP, simply parse into a flat struct
that contains the info we need.
This is an exception to our strategy with Protocol.h, which seems justified:
 - the structure here is very large and deeply nested
 - we care about almost none of it
 - we should never have to serialize client capabilities

Reviewers: kadircet

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D53266

llvm-svn: 344673
2018-10-17 07:33:42 +00:00
Sam McCall dc8f3cf8b0 [clangd] Refactor JSON-over-stdin/stdout code into Transport abstraction. (re-land r344620)
Summary:
This paves the way for alternative transports (mac XPC, maybe messagepack?),
and also generally improves layering: testing ClangdLSPServer becomes less of
a pipe dream, we split up the JSONOutput monolith, etc.

This isn't a final state, much of what remains in JSONRPCDispatcher can go away,
handlers can call reply() on the transport directly, JSONOutput can be renamed
to StreamLogger and removed, etc. But this patch is sprawling already.

The main observable change (see tests) is that hitting EOF on input is now an
error: the client should send the 'exit' notification.
This is defensible: the protocol doesn't spell this case out. Reproducing the
current behavior for all combinations of shutdown/exit/EOF clutters interfaces.
We can iterate on this if desired.

Reviewers: jkorous, ioeric, hokein

Subscribers: mgorny, ilya-biryukov, MaskRay, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53286

llvm-svn: 344672
2018-10-17 07:32:05 +00:00
Krasimir Georgiev ae4e75fd6e Revert "[clangd] Refactor JSON-over-stdin/stdout code into Transport abstraction."
This reverts commit r344620.
Breaks upstream bots.

llvm-svn: 344637
2018-10-16 18:44:41 +00:00
Sam McCall 818ec5e965 [clangd] Refactor JSON-over-stdin/stdout code into Transport abstraction.
Summary:
This paves the way for alternative transports (mac XPC, maybe messagepack?),
and also generally improves layering: testing ClangdLSPServer becomes less of
a pipe dream, we split up the JSONOutput monolith, etc.

This isn't a final state, much of what remains in JSONRPCDispatcher can go away,
handlers can call reply() on the transport directly, JSONOutput can be renamed
to StreamLogger and removed, etc. But this patch is sprawling already.

The main observable change (see tests) is that hitting EOF on input is now an
error: the client should send the 'exit' notification.
This is defensible: the protocol doesn't spell this case out. Reproducing the
current behavior for all combinations of shutdown/exit/EOF clutters interfaces.
We can iterate on this if desired.

Reviewers: jkorous, ioeric, hokein

Subscribers: mgorny, ilya-biryukov, MaskRay, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53286

llvm-svn: 344620
2018-10-16 16:48:06 +00:00
Sam McCall 20841d41e7 [clangd] Send CodeAction responses to textDocument/codeAction (LSP 3.8)
Summary:
I don't bother mirroring the full capabilities struct, just parse the
bits we care about. I'll send a new patch to use this approach elsewhere too.

Reviewers: kadircet

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D53213

llvm-svn: 344617
2018-10-16 16:29:41 +00:00
Simon Marchi abeed66056 Remove possibility to change compile database path at runtime
Summary:
This patch removes the possibility to change the compilation database
path at runtime using the didChangeConfiguration request.  Instead, it
is suggested to use the setting on the initialize request, and clangd
whenever the user wants to use a different build configuration.

Subscribers: ilya-biryukov, ioeric, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53220

llvm-svn: 344614
2018-10-16 15:55:03 +00:00
Eric Liu 0c29722c60 [clangd] Allow disble down traversals from root.
Summary:
This is useful for symbo scope proximity, where down traversals from
the global scope if not desired.

Reviewers: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53317

llvm-svn: 344604
2018-10-16 10:41:17 +00:00
Sam McCall bca624ab03 [clangd] Fix threading bugs in (not-yet-used) BackgroundIndex, re-enable test.
Summary:
One relatively boring bug: forgot to notify the CV after enqueue.

One much more fun bug: the thread member could access instance variables before
they were initialized. Although the thread was last in the init list, QueueCV
etc were listed after Thread in the class, so their default constructors raced
with the thread itself.
We have to get very unlucky to lose this race, I saw it 0.02% of the time.

Reviewers: ioeric

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, jfb, cfe-commits

Differential Revision: https://reviews.llvm.org/D53313

llvm-svn: 344595
2018-10-16 09:05:13 +00:00
Sam McCall 96f2489557 [clangd] Optionally use dex for the preamble parts of the dynamic index.
Summary:
Reuse the old -use-dex-index experiment flag for this.

To avoid breaking the tests, make Dex deduplicate symbols, addressing an old FIXME.

Reviewers: hokein

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53288

llvm-svn: 344594
2018-10-16 08:53:52 +00:00
Sam McCall bc8aee15a2 [clangd] Revert include path change in Dexp. NFC
llvm-svn: 344533
2018-10-15 16:47:45 +00:00
Haojian Wu 397704ca40 [clangd] Add createIndex in dexp
Summary:
This would allow easily injecting our internal customization.

Also updates the stale "symbol-collection-file" flag.

Reviewers: sammccall

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53292

llvm-svn: 344521
2018-10-15 15:12:40 +00:00
Sam McCall 2b24ce61a0 [clangd] Use SyncAPI in more places in tests. NFC
llvm-svn: 344520
2018-10-15 15:04:03 +00:00
Sam McCall 8dc9dbb61a [clangd] Minimal implementation of automatic static index (not enabled).
Summary:
See tinyurl.com/clangd-automatic-index for design and goals.

Lots of limitations to keep this patch smallish, TODOs everywhere:
 - no serialization to disk
 - no changes to dynamic index, which now has a much simpler job
 - no partitioning of symbols by file to avoid duplication of header symbols
 - no reindexing of edited files
 - only a single worker thread
 - compilation database is slurped synchronously (doesn't scale)
 - uses memindex, rebuilds after every file (should be dex, periodically)

It's not hooked up to ClangdServer/ClangdLSPServer yet: the layering
isn't clear (it should really be in ClangdServer, but ClangdLSPServer
has all the CDB interactions).

Reviewers: ioeric

Subscribers: mgorny, ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, jfb, cfe-commits

Differential Revision: https://reviews.llvm.org/D53032

llvm-svn: 344513
2018-10-15 13:34:10 +00:00
Haojian Wu 82ba7121e8 [clangd] Remove an unused include header, NFC.
llvm-svn: 344510
2018-10-15 12:39:45 +00:00
Haojian Wu ddec850ceb [clangd] dump xrefs information in dexp tool.
Reviewers: sammccall

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53019

llvm-svn: 344508
2018-10-15 12:32:49 +00:00
Haojian Wu e83caccb58 [clangd] Fix some references missing in dynamic index.
Summary:
Previously, SymbolCollector postfilters all references at the end to
find all references of interesting symbols.
It was incorrect when indxing main AST where we don't see locations
of symbol declarations and definitions in the main AST (as those are in
preamble AST).

The fix is to do earily check during collecting references.

Reviewers: sammccall

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53273

llvm-svn: 344507
2018-10-15 11:46:26 +00:00
Benjamin Kramer c55e997556 Move some helpers from the global namespace into anonymous ones.
llvm-svn: 344468
2018-10-13 22:18:22 +00:00
Jonas Toth f79f8eecce [clangd] NFC fix semicolon warning
llvm-svn: 344384
2018-10-12 17:47:43 +00:00
Sam McCall cbf497f452 [clangd] Return Command objects from onCodeAction, rather than ad-hoc JSON. NFC
llvm-svn: 344363
2018-10-12 16:51:48 +00:00
Haojian Wu 903f2a71a4 [clangd] Support hover on "aut^o *".
Reviewers: kadircet

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D53186

llvm-svn: 344330
2018-10-12 10:11:02 +00:00
Sam McCall 0f80d9f27d [clangd] Remove no-op crash handler, we never set a crash context.
Summary:
I think this was just copied from somewhere with the belief that it actually
did some crash handling.

Of course the question arises: *should* we set one? I don't think so:
 - clangd used to crash a lot, now it's pretty stable, because we found and
   fixed the crashes. I think the long-term effects of crashing hard are good.
 - the implementation can't do any magic, it just uses longjmp to return without
   running any destructors by default. This is unsafe in general (e.g. mutexes
   won't unlock) and will certainly end up leaking memory. Whatever UB caused
   the crash may still stomp all over global state, etc.

I think there's an argument for isolating the background indexer (autoindex)
because it's not directly under the user's control, the crash surface is larger,
and it doesn't particularly need to interact with the rest of clangd.
But there, fork() and communicate through the FS is safer.

Reviewers: ioeric, ilya-biryukov

Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D53034

llvm-svn: 344245
2018-10-11 13:06:10 +00:00
Chandler Carruth d966ce4c98 Fix the qualification of `IntrusiveRefCntPtr` to use `llvm::`.
Without this, the code only compiled if the header was included after
something introduced the alias from `clang::` to `llvm::` for this type.
Any modules build would fail here.

llvm-svn: 344225
2018-10-11 08:05:10 +00:00
Jonas Devlieghere fc51490baf Lift VFS from clang to llvm (NFC)
This patch moves the virtual file system form clang to llvm so it can be
used by more projects.

Concretely the patch:
 - Moves VirtualFileSystem.{h|cpp} from clang/Basic to llvm/Support.
 - Moves the corresponding unit test from clang to llvm.
 - Moves the vfs namespace from clang::vfs to llvm::vfs.
 - Formats the lines affected by this change, mostly this is the result of
   the added llvm namespace.

RFC on the mailing list:
http://lists.llvm.org/pipermail/llvm-dev/2018-October/126657.html

Differential revision: https://reviews.llvm.org/D52783

llvm-svn: 344140
2018-10-10 13:27:25 +00:00
Sam McCall 26fac469d2 [clangd] Make FSProvider const-correct. NFC
llvm-svn: 344118
2018-10-10 07:46:15 +00:00
Haojian Wu 292a36a0d5 [clangd] Fix an accident change in r342999.
llvm-svn: 344054
2018-10-09 15:16:14 +00:00
Jonas Toth 3acdd020b4 [clangd] fix miscompiling lower_bound call
llvm-svn: 344044
2018-10-09 13:24:50 +00:00
Kadir Cetinkaya e324027b72 [clangd] Mark colon as a safe character when percent-encoding.
Summary: Also change output of percent-encoding to use upper-case letters.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D53016

llvm-svn: 344033
2018-10-09 10:29:54 +00:00
Kadir Cetinkaya 84bf607eac [clangd] Revert back to previous heuristic for diagnostic range extraction.
Summary: Also add a few new test cases and a special case into handling of empty fixit ranges that  collides with location of a diag.

Reviewers: sammccall, ilya-biryukov

Reviewed By: sammccall

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D52889

llvm-svn: 344025
2018-10-09 08:41:12 +00:00
Eric Liu 7734e63ad8 [clangd] Avoid cache main file status in preamble.
Summary: Main file can certainly change when reusing preamble.

Reviewers: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52991

llvm-svn: 344024
2018-10-09 08:27:31 +00:00
Haojian Wu 162510f619 [clangd] Update the out-of-date yaml-symbol-file flag in clangd.
Summary:
The flag is stale due to the recent changes of clangd indexer, this
patch renames the flag to "index-file".

Reviewers: sammccall

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52976

llvm-svn: 343963
2018-10-08 10:44:54 +00:00
Fangrui Song 8380c9e918 [clangd] Migrate to LLVM STLExtras range API
llvm-svn: 343946
2018-10-07 17:21:08 +00:00
Kirill Bobyrev 4a5ff88fdb [clangd] NFC: Migrate to LLVM STLExtras API where possible
This patch improves readability by migrating `std::function(ForwardIt
start, ForwardIt end, ...)` to LLVM's STLExtras range-based equivalent
`llvm::function(RangeT &&Range, ...)`.

Similar change in Clang: D52576.

Reviewed By: sammccall

Differential Revision: https://reviews.llvm.org/D52650

llvm-svn: 343937
2018-10-07 14:49:41 +00:00
Fangrui Song 39f7b3967a [clangd] Remove unused headers from CodeComplete.cpp
queue is not used after index-provided completions' merge with those from Sema
USRGeneration.h is not used after introduction of getSymbolID

llvm-svn: 343912
2018-10-06 07:00:50 +00:00
Sam McCall 5fb9746c49 [clangd] Remove last usage of ast matchers from SymbolCollector. NFC
llvm-svn: 343849
2018-10-05 14:03:04 +00:00
Sam McCall b2d530e035 [clangd] Fix a subtle case for GetBeginningOfIdentifier.
Calling getMacroArgExpansionLocation too early was causing
Lexer::getRawToken to do the wrong thing - lexing the macro name instead
of the arg contents.

Differential Revision: https://reviews.llvm.org/D52928

llvm-svn: 343844
2018-10-05 12:08:06 +00:00
Sam McCall 722d6c6041 [clangd] Make binary index format the default, remove dead flag.
Reviewers: hokein

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52872

llvm-svn: 343841
2018-10-05 09:05:28 +00:00
Sam McCall 50b89f0a9b [clangd] Simplify Dex query tree logic and fix missing-posting-list bug
Summary:
The bug being fixed: when a posting list doesn't exist in the index, it
was previously just dropped from the query rather than being treated as
empty. Now that we have the FALSE iterator, we can use it instead.

The query tree logic previously had a bunch of special cases to detect whether
subtrees are empty. Now we just naively build the whole tree, and rely
on the query optimizations to drop the trivial parts.

Finally, there was a bug in trigram generation: the empty query would
generate a single trigram "$$$" instead of no trigrams.
This had no effect (there was no posting list, so the other bug
cancelled it out). But we now have to fix this bug too.

Reviewers: ilya-biryukov

Subscribers: ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52796

llvm-svn: 343802
2018-10-04 17:18:55 +00:00
Sam McCall aa728f1afa [clangd] Dex: FALSE iterator, peephole optimizations, fix AND bug
Summary:
The FALSE iterator will be used in a followup patch to fix a logic bug in Dex
(currently, tokens that don't have posting lists in the index are simply dropped
from the query, changing semantics).

It can usually be optimized away, so added the following opmitizations:
 - simplify booleans inside AND/OR
 - replace effectively-empty AND/OR with booleans
 - flatten nested AND/ORs

While working on this, found a bug in the AND iterator: its constructor sync()
assumes that ReachedEnd is set if applicable, but the constructor never sets it.
This crashes if a non-first iterator is nonempty.

Reviewers: ilya-biryukov

Subscribers: ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52789

llvm-svn: 343801
2018-10-04 17:18:49 +00:00
Sam McCall 422f724618 [clangd] expose MergedIndex class
Summary:
This allows inheriting from it, so index() can ga away and allowing
TestTU::index) to be fixed.

Reviewers: ioeric

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52250

llvm-svn: 343780
2018-10-04 14:20:22 +00:00
Sam McCall cc21779c3c [clangd] clangd-indexer gathers refs and stores them in index files.
Reviewers: ioeric

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52531

llvm-svn: 343778
2018-10-04 14:09:55 +00:00
Sam McCall 2ec5a10db3 [clangd] Remove one-segment-skipping from Dex trigrams.
Summary:
Currently queries like "ab" can match identifiers like a_yellow_bee.
The value of allowing this for exactly one segment but no more seems dubious.
It costs ~3% of overall ram (~9% of posting list ram) and some quality.

Reviewers: ilya-biryukov, ioeric

Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52885

llvm-svn: 343777
2018-10-04 14:08:11 +00:00
Sam McCall b5bbfef6cd [cland] Dex: fix/simplify short-trigram generation
Summary:
1) Instead of x$$ for a short-query trigram, just use x
2) Make rules more coherent: prefixes of length 1-2, and first char + next head
3) Fix Dex::fuzzyFind to mark results as incomplete, because
   short-trigram rules only yield a subset of results.

Reviewers: ioeric

Subscribers: ilya-biryukov, jkorous, mgrang, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52808

llvm-svn: 343775
2018-10-04 14:01:55 +00:00
Sam McCall 87f69eaf4e [clangd] Dex: FALSE iterator, peephole optimizations, fix AND bug
Summary:
The FALSE iterator will be used in a followup patch to fix a logic bug in Dex
(currently, tokens that don't have posting lists in the index are simply dropped
from the query, changing semantics).

It can usually be optimized away, so added the following opmitizations:
 - simplify booleans inside AND/OR
 - replace effectively-empty AND/OR with booleans
 - flatten nested AND/ORs

While working on this, found a bug in the AND iterator: its constructor sync()
assumes that ReachedEnd is set if applicable, but the constructor never sets it.
This crashes if a non-first iterator is nonempty.

Reviewers: ilya-biryukov

Subscribers: ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52789

llvm-svn: 343774
2018-10-04 13:12:23 +00:00
Sam McCall 71ea753a80 [clangd] Revert accidental flag change
llvm-svn: 343764
2018-10-04 10:10:35 +00:00
Haojian Wu d592498978 [clangd] Use canonical declarations in ReferenceFinder.
Summary:
handleDeclOccurrencce reports a canonical declartion, so stick to use
canonical declarations to determine whether a declaration is in the
target set.

Also fix a previous ref test which misses a matched label (it fails without this
patch).

Reviewers: sammccall

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52871

llvm-svn: 343763
2018-10-04 09:56:08 +00:00
Sam McCall d9eae39800 [clangd] Support refs() in dex. Largely cloned from MemIndex.
Reviewers: hokein

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52726

llvm-svn: 343760
2018-10-04 09:16:12 +00:00
Sam McCall 41e6d76c22 [clangd] clangd-indexer: Drop support for MR-via-YAML
Summary:
It's slow, and the open-source reduce implementation doesn't scale properly.
While here, tidy up some dead headers and comments.

Reviewers: kadircet

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D52517

llvm-svn: 343759
2018-10-04 08:30:03 +00:00
Mikael Holmen 101d9eae23 Fix compilation warning by removing unused variable [NFC]
clang complained with

../tools/clang/tools/extra/clangd/FS.cpp:19:12: error: unused variable 'Err' [-Werror,-Wunused-variable]
  if (auto Err = FS.makeAbsolute(PathStore))
           ^
1 error generated.

llvm-svn: 343661
2018-10-03 05:41:14 +00:00
Sam McCall a659d779f8 Reland r343589 "[clangd] Dex: add Corpus factory for iterators, rename, fold constant. NFC""
This reverts commit r343610.

llvm-svn: 343622
2018-10-02 19:59:23 +00:00
Reid Kleckner 2b5259afb3 Revert r343589 "[clangd] Dex: add Corpus factory for iterators, rename, fold constant. NFC"
Declaring a field with the same name as a type causes GCC to error out:

Dex.h:104:10: error: declaration of 'clang::clangd::dex::Corpus clang::clangd::dex::Dex::Corpus' [-fpermissive]
   Corpus Corpus;
          ^
Iterator.h:127:7: error: changes meaning of 'Corpus' from 'class clang::clangd::dex::Corpus' [-fpermissive]
 class Corpus {

llvm-svn: 343610
2018-10-02 17:31:43 +00:00
Sam McCall 51be55d0ec [clangd] Zap TODONEs
llvm-svn: 343590
2018-10-02 13:51:43 +00:00
Sam McCall a1e7385d5c [clangd] Dex: add Corpus factory for iterators, rename, fold constant. NFC
Summary:
- Corpus avoids having to pass size to the true iterator, and (soon) any
  iterator that might optimize down to true.
- Shorten names of factory functions now they're scoped to the Corpus.
  intersect() and unionOf() rather than createAnd() or createOr() as this
  seems to read better to me, and fits with other short names. Opinion wanted!
- DEFAULT_BOOST_SCORE --> 1. This is a multiplier, don't obfuscate identity.
- Simplify variadic templates in Iterator.h

Reviewers: ioeric

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52711

llvm-svn: 343589
2018-10-02 13:44:26 +00:00
Sam McCall 7402836042 [clangd] Dex iterator printer shows query structure, not iterator state.
Summary:
This makes it suitable for logging (which immediately found a bug, to
be fixed in the next patch...)

Reviewers: ioeric

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52715

llvm-svn: 343580
2018-10-02 11:51:36 +00:00
Eric Liu b1d7542dde [clangd] Cache FS stat() calls when building preamble.
Summary:
The file stats can be reused when preamble is reused (e.g. code
completion). It's safe to assume that cached status is not outdated as we
assume preamble files to remain unchanged.

On real file system, this made code completion ~20% faster on a measured file
(with big preamble). The preamble build time doesn't change much.

Reviewers: sammccall, ilya-biryukov

Reviewed By: sammccall

Subscribers: mgorny, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52419

llvm-svn: 343576
2018-10-02 10:43:55 +00:00
Kadir Cetinkaya b15b8dce39 [clangd] Remove override result handling logic from clangd
Summary:
Since we plan to move handling of override suggestions to Sema with
D52225 this patch just makes sure clangd-side has no logic related to that
anymore and updates tests.

Reviewers: ioeric, ilya-biryukov

Reviewed By: ioeric

Subscribers: MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D52226

llvm-svn: 343567
2018-10-02 09:42:17 +00:00
Sam McCall 329fc143fd [clangd] Query dex index using query-style trigrams, not identifier-style trigrams
llvm-svn: 343453
2018-10-01 10:42:51 +00:00
Eric Liu d5d6a60a78 [clangd] Fix header mapping for std::string. NFC
Some implementation has std::string declared in <iosfwd>.

llvm-svn: 343448
2018-10-01 08:50:49 +00:00
Eric Liu 670c147d83 [clangd] Initial supoprt for cross-namespace global code completion.
Summary:
When no scope qualifier is specified, allow completing index symbols
from any scope and insert proper automatically. This is still experimental and
hidden behind a flag.

Things missing:
- Scope proximity based scoring.
- FuzzyFind supports weighted scopes.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: kbobyrev, ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52364

llvm-svn: 343248
2018-09-27 18:46:00 +00:00
Eric Liu ee7fe93fa8 [clangd] Add more tracing to index queries. NFC
Reviewers: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52611

llvm-svn: 343247
2018-09-27 18:23:23 +00:00
Kadir Cetinkaya 133d46f9a7 Introduce completionItemKind capability support.
Reviewers: sammccall

Reviewed By: sammccall

Subscribers: ilya-biryukov, ioeric, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D52616

llvm-svn: 343237
2018-09-27 17:13:07 +00:00
Eric Liu fd9f426049 [clangd] Make IncludeInserter less slow. NFC
llvm-svn: 343223
2018-09-27 14:27:02 +00:00
Kadir Cetinkaya 0ed5d29b44 Tell whether file/folder for include completions.
Reviewers: sammccall

Reviewed By: sammccall

Subscribers: ilya-biryukov, ioeric, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D52547

llvm-svn: 343221
2018-09-27 14:21:07 +00:00
Kadir Cetinkaya 2e3f1f13b5 Improve diagnostics range reporting.
Summary:
If we have some range information coming from clang diagnostic, promote
that one even if it doesn't contain diagnostic location inside.

Reviewers: sammccall, ioeric

Reviewed By: ioeric

Subscribers: ilya-biryukov, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D52544

llvm-svn: 343197
2018-09-27 12:12:42 +00:00
Kirill Bobyrev ea4f20c6be [clangd] Fix bugs with incorrect memory estimate report
* With the current implementation, `sizeof(std::vector<Chunk>)` is added
twice to the `Dex` memory estimate which is incorrect
* `Dex` logs memory usage estimation before `BackingDataSize` is set and
hence the log report excludes size of the external `SymbolSlab` which is
coupled with `Dex` instance

Reviewed By: ioeric

Differential Revision: https://reviews.llvm.org/D52503

llvm-svn: 343117
2018-09-26 15:06:23 +00:00
Kirill Bobyrev 0cdf629394 [docs] Update PostingList string representation format
Because `PostingList` objects are compressed, it is now impossible to
see elements other than the current one and the documentation doesn't
match implementation anymore.

Reviewed By: ioeric

Differential Revision: https://reviews.llvm.org/D52545

llvm-svn: 343116
2018-09-26 14:59:49 +00:00
Simon Pilgrim 3462e76ba5 Removed extra semicolon to fix Wpedantic. (NFCI).
llvm-svn: 343083
2018-09-26 09:02:45 +00:00
Ilya Biryukov 652364b765 [clangd] Fix crash if pending computations were active on exit
Summary:
Make sure JSONRPCDispatcher outlives the worker threads, they access
its fields to remove the stored cancellations when Context dies.

Reviewers: sammccall, ioeric

Reviewed By: ioeric

Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52420

llvm-svn: 343067
2018-09-26 05:48:29 +00:00
Ilya Biryukov 4f98470dea [clangd] Handle template args for disabled function arg snippets
Reviewers: kadircet, ioeric, sammccall

Reviewed By: kadircet

Subscribers: MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D52422

llvm-svn: 343066
2018-09-26 05:45:31 +00:00
Sam McCall 321d5d4802 [clangd] Extract mapper logic from clangd-indexer into a library.
Summary: Soon we can drop support for MR-via-YAML.
I need to modify some out-of-tree versions to use the library, first.

Reviewers: kadircet

Subscribers: mgorny, ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D52465

llvm-svn: 343019
2018-09-25 20:02:36 +00:00
Sam McCall e38c7f8d96 [clangd] Fix reversed RIFF/YAML serialization
llvm-svn: 343017
2018-09-25 19:53:33 +00:00
Sam McCall 02d600d267 [clangd] Merge binary + YAML serialization behind a (mostly) common interface.
Summary:
Interface is in one file, implementation in two as they have little in common.
A couple of ad-hoc YAML functions left exposed:
 - symbol -> YAML I expect to keep for tools like dexp
 - YAML -> symbol is used for the MR-style indexer, I think we can eliminate
   this (merge-on-the-fly, else use a different serialization)

Reviewers: kbobyrev

Subscribers: mgorny, ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52453

llvm-svn: 342999
2018-09-25 18:06:43 +00:00
Kirill Bobyrev d041f8a9d0 [clangd] NFC: Simplify code, enforce LLVM Coding Standards
For consistency, functional-style code pieces are replaced with their
simple counterparts to improve readability.

Also, file headers are fixed to comply with LLVM Coding Standards.

`static` member of anonymous namespace is not marked `static` anymore,
because it is redundant.

Reviewed By: sammccall

Differential Revision: https://reviews.llvm.org/D52466

llvm-svn: 342974
2018-09-25 13:58:48 +00:00
Kirill Bobyrev 69e6388564 [clangd] Fix some buildbots after r342965
Some compilers fail to parse struct default member initializer.

llvm-svn: 342970
2018-09-25 13:14:11 +00:00
Kirill Bobyrev 6c2f5bd0f1 [clangd] Implement VByte PostingList compression
This patch implements Variable-length Byte compression of `PostingList`s
to sacrifice some performance for lower memory consumption.

`PostingList` compression and decompression was extensively tested using
fuzzer for multiple hours and runnning significant number of realistic
`FuzzyFindRequests`. AddressSanitizer and UndefinedBehaviorSanitizer
were used to ensure the correct behaviour.

Performance evaluation was conducted with recent LLVM symbol index (292k
symbols) and the collection of user-recorded queries (7751
`FuzzyFindRequest` JSON dumps):

| Metrics | Before| After | Change (%)
| -----  | -----  | -----   | -----
| Memory consumption (posting lists only), MB  |  54.4 | 23.5 | -60%
| Time to process queries, sec | 7.70 | 9.4 | +25%

Reviewers: sammccall, ioeric

Reviewed By: sammccall

Differential Revision: https://reviews.llvm.org/D52300

llvm-svn: 342965
2018-09-25 11:54:51 +00:00
Eric Liu a9d41f94b9 [clangd] Fix build bot after r342961
Use llvm::isAlpha instead of std::isalpha etc. This should fix bot failure:
http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/20180

llvm-svn: 342964
2018-09-25 11:47:14 +00:00
Eric Liu e631d09fc5 [clangd] Check that scheme is valid when parsing URI.
Reviewers: sammccall

Reviewed By: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52455

llvm-svn: 342961
2018-09-25 10:47:46 +00:00
Sylvestre Ledru cba939a74b Fix a typo in the help of clangd
llvm-svn: 342960
2018-09-25 10:36:57 +00:00
Sam McCall 3ca9759a21 [clangd] Fix uninit bool in r342888
llvm-svn: 342903
2018-09-24 16:52:48 +00:00
Sam McCall 8fb7bb2482 [clangd] Do bounds checks while reading data, otherwise var-length records are too painful. NFC
llvm-svn: 342888
2018-09-24 14:51:15 +00:00
Kirill Bobyrev 94af0612e0 [clangd] Force Dex to respect symbol collector flags
`Dex` should utilize `FuzzyFindRequest.RestrictForCodeCompletion` flags
and omit symbols not meant for code completion when asked for it.

The measurements below were conducted with setting
`FuzzyFindRequest.RestrictForCodeCompletion` to `true` (so that it's
more realistic). Sadly, the average latency goes down, I suspect that is
mostly because of the empty queries where the number of posting lists is
critical.

| Metrics  | Before | After | Relative difference
| -----  | -----  | -----   | -----
| Cumulative query latency (7000 `FuzzyFindRequest`s over LLVM static index)  | 6182735043 ns    | 7202442053 ns | +16%
| Whole Index size | 81.24 MB    | 81.79 MB | +0.6%

Out of 292252 symbols collected from LLVM codebase 136926 appear to be
restricted for code completion.

Reviewers: ioeric

Differential Revision: https://reviews.llvm.org/D52357

llvm-svn: 342866
2018-09-24 08:45:18 +00:00
Eric Liu c275fb2a5d [clangd] Remember to serialize symbol origin in YAML.
llvm-svn: 342730
2018-09-21 13:04:57 +00:00
Kadir Cetinkaya e8d8aee537 [clangd] Add option to enable/disable function argument snippets.
Summary:
Currently LSP clients cannot directly change EnableFunctionArgSnippets parameter.
This patch is to provide them with a way to enable/disable that functionality.

Reviewers: hokein, ioeric, ilya-biryukov

Reviewed By: ilya-biryukov

Subscribers: sammccall, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D51214

llvm-svn: 342533
2018-09-19 10:16:44 +00:00
Eric Liu 467c5f9ce0 [clangd] Store preamble macros in dynamic index.
Summary:
Pros:
o Loading macros from preamble for every completion is slow (see profile).
o Calculating macro USR is also slow (see profile).
o Sema can provide a lot of macro completion results (e.g. when filter is empty,
60k for some large TUs!).

Cons:
o Slight memory increase in dynamic index (~1%).
o Some extra work during preamble build (should be fine as preamble build and
indexAST is way slower).

Before:
{F7195645}

After:
{F7195646}

Reviewers: ilya-biryukov, sammccall

Reviewed By: sammccall

Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52078

llvm-svn: 342529
2018-09-19 09:35:04 +00:00
Sam McCall 46b5555844 [clangd] Fix error handling for SymbolID parsing (notably YAML and dexp)
llvm-svn: 342505
2018-09-18 19:00:59 +00:00
Eric Liu 764f461f9c [clangd] Get rid of Decls parameter in indexMainDecls. NFC
It's already available in ParsedAST.

llvm-svn: 342473
2018-09-18 13:35:16 +00:00
Eric Liu 821a116818 [clangd] Merge ClangdServer::DynamicIndex into FileIndex. NFC.
Summary:
FileIndex now provides explicit interfaces for preamble and main file updates.
This avoids growing parameter list when preamble and main symbols diverge
further (e.g. D52078). This also gets rid of the hack in `indexAST` that
inferred main file index based on `TopLevelDecls`.

Also separate `indexMainDecls` from `indexAST`.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52222

llvm-svn: 342460
2018-09-18 10:30:44 +00:00
Sam McCall 3bf9b6d920 [clangd] dexp tool uses llvm::cl to parse its flags.
Summary:
We can use cl::ResetCommandLineParser() to support different types of
command-lines, as long as we're careful about option lifetimes.
(I tried using subcommands, but the error messages were bad)
I found a mostly-reasonable pattern to isolate the fiddly parts.

Added -scope and -limit flags to the `find` command to demonstrate.
(Note that scope support seems to be broken in dex?)

Fixed symbol lookup to parse symbol IDs.

Caveats:
 - with command help (e.g. `find -help`), you also get some spam
   about required arguments. This is a bug in llvm::cl, which prints
   these to errs() rather than the designated stream.

Reviewers: kbobyrev

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D51989

llvm-svn: 342456
2018-09-18 09:49:57 +00:00
Sam McCall 4c077f987c [clangd] Update code completion for #include completions in r342449
llvm-svn: 342453
2018-09-18 09:08:28 +00:00
Eric Liu f736766659 [clangd] Adapt API change after 342451.
llvm-svn: 342452
2018-09-18 08:52:14 +00:00
Eric Liu a57afd091f [clangd] Get rid of AST matchers in SymbolCollector. NFC
Reviewers: ilya-biryukov, kadircet

Subscribers: MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D52089

llvm-svn: 342362
2018-09-17 07:43:49 +00:00
Sam McCall ebef81227b [clangd] Don't override the preamble while completing inside it, it doesn't work.
Summary:
To stay fast, enable single-file-mode instead. This is fine since completions
in the preamble are simple.

The net effect for now is to suppress the spurious TopLevel completions when
completing inside the preamble.
Once Sema has include directive completion, this will be more important.

Reviewers: ilya-biryukov

Subscribers: ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D52071

llvm-svn: 342230
2018-09-14 12:36:06 +00:00
Sam McCall 053c9ee826 [clangd] Don't double-infer compile commands after r342228
llvm-svn: 342229
2018-09-14 12:32:08 +00:00
Kirill Bobyrev 2fcdf76eef [clangd] NFC: Fix IndexBenchmark CLI arguments handling
llvm-svn: 342227
2018-09-14 12:21:09 +00:00
Ilya Biryukov 58382fab77 [clangd] Update IndexerMain.cpp file comment after rename. NFC
llvm-svn: 342226
2018-09-14 11:39:05 +00:00
Fangrui Song 847bd33166 [clangd] Fix TUScheduler typos
llvm-svn: 342198
2018-09-14 00:56:11 +00:00
Kirill Bobyrev 249c5864cf [clangd] Introduce PostingList interface
This patch abstracts `PostingList` interface and reuses existing
implementation. It will be used later to test different `PostingList`
representations.

No functionality change is introduced, this patch is mostly refactoring
so that the following patches could focus on functionality while not
being too hard to review.

Reviewed By: sammccall, ioeric

Differential Revision: https://reviews.llvm.org/D51982

llvm-svn: 342155
2018-09-13 17:11:03 +00:00
Kirill Bobyrev bd72b08eb3 [clangd] Fix Dexp build
%s/MaxCandidateCount/Limit/g after rL342138.

llvm-svn: 342143
2018-09-13 15:35:55 +00:00
Kirill Bobyrev e6dd0806c7 [clangd] Cleanup FuzzyFindRequest filtering limit semantics
As discussed during D51860 review, it is better to use `llvm::Optional`
here as it has clear semantics which reflect intended behavior.

Reviewed By: sammccall

Differential Revision: https://reviews.llvm.org/D52028

llvm-svn: 342138
2018-09-13 14:27:03 +00:00
Kirill Bobyrev 60be1f59bd [clangd] Use JSON format in benchmark requests reader
After `FuzzyFindRequest` JSON (de)serialization was introduced, it
should replace ad-hoc fuzzy-find request parsing implemented in the
IndexBenchmark driver.

Reviewed By: ilya-biryukov

Differential Revision: https://reviews.llvm.org/D51971

llvm-svn: 342137
2018-09-13 14:21:50 +00:00
Sam McCall 45be5cf0ed [clangd] Allow all LSP methods to signal cancellation via $/cancelRequest
Summary:
The cancelable scopes are managed by JSONRPCDispatcher so that all Handlers
run in cancelable contexts.
(Previously ClangdServer did this, for code completion only).

Cancellation request processing is therefore also in JSONRPCDispatcher.
(Previously it was in ClangdLSPServer).

This doesn't actually make any new commands *respect* cancellation - they'd
need to check isCancelled() and bail out. But it opens the door to doing
this incrementally, and putting such logic in common machinery like TUScheduler.

I also rewrote the ClangdServer class/threading comments because I wanted to
add to it and I got carried away.

Reviewers: ilya-biryukov, kadircet

Subscribers: ioeric, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D52004

llvm-svn: 342135
2018-09-13 12:58:36 +00:00
Eric Liu 76b88d8e98 [clangd] Clarify and hide -index flag.
Summary:
The wording implies global index support, which is confusing.
As most users shouldn't care about this flag, also make it hidden to avoid
further confusion.

Reviewers: sammccall, ilya-biryukov

Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D51977

llvm-svn: 342134
2018-09-13 12:53:23 +00:00
Sam McCall c8758406b5 [clangd] Simplify cancellation public API
Summary:
Task is no longer exposed:
 - task cancellation is hidden as a std::function
 - task creation returns the new context directly
 - checking is via free function only, with no way to avoid the context lookup
The implementation is essentially the same, but a bit terser as it's hidden.

isCancelled() is now safe to use outside any task (it returns false).
This will leave us free to sprinkle cancellation in e.g. TUScheduler without
needing elaborate test setup, and lets callers that don't cancel "just work".

Updated the docs to describe the new expected use pattern.
One thing I noticed: there's nothing async-specific about the cancellation.
Async tasks can be cancelled from any thread (typically the one that created
them), sync tasks can be cancelled from any *other* thread in the same way.
So the docs now refer to "long-running" tasks instead of async ones.

Updated usage in code complete, without any structural changes.
I didn't update all the names of the helpers in ClangdLSPServer (these will
likely be moved to JSONRPCDispatcher anyway).

Reviewers: ilya-biryukov, kadircet

Subscribers: ioeric, MaskRay, jkorous, arphaman, jfb, cfe-commits

Differential Revision: https://reviews.llvm.org/D51996

llvm-svn: 342130
2018-09-13 11:47:48 +00:00
Kirill Bobyrev d9f33b129c [clangd] Don't create child AND and OR iterators with one posting list
`AND( AND( Child ) ... )` -> `AND( Child ... )`
`AND( OR( Child ) ... )` -> `AND( Child ... )`

This simple optimization results in 5-6% performance improvement in the
benchmark with 2000 serialized `FuzzyFindRequest`s.

Reviewed By: ilya-biryukov

Differential Revision: https://reviews.llvm.org/D52016

llvm-svn: 342124
2018-09-13 10:02:48 +00:00
Ilya Biryukov 33b72d8d34 [clangd] Rename global-symbol-builder to clangd-indexer.
Summary:
Given that the indexer binary is put directly into ./bin directory
when built, 'clangd-' prefix seems to provide better context to the
reader than 'global-'.

The new name is also shorter and easier to type.

Reviewers: ioeric, sammccall, kadircet

Reviewed By: ioeric, sammccall

Subscribers: kbobyrev, ilya-biryukov, mgorny, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D51987

llvm-svn: 342123
2018-09-13 09:44:11 +00:00
Heejin Ahn 386d272387 [clangd] Add missing clangBasic target_link_libraries
Without this, builds with `-DSHARED_LIB=ON` fail.

llvm-svn: 342037
2018-09-12 09:40:13 +00:00
Kirill Bobyrev 73c201da51 [clangd] Add index benchmarks
This patch introduces index benchmarks on top of the proposed LLVM
benchmark pull.

Reviewed By: sammccall, lebedev.ri

Differential Revision: https://reviews.llvm.org/D51090

llvm-svn: 342026
2018-09-12 07:49:44 +00:00
Kirill Bobyrev e1e19c7b75 [clangd] Implement a Proof-of-Concept tool for symbol index exploration
Reviewed By: sammccall, ilya-biryukov

Differential Revision: https://reviews.llvm.org/D51628

llvm-svn: 342025
2018-09-12 07:32:54 +00:00
Simon Pilgrim 008da027fc Remove unnecessary semicolon to silence -Wpedantic warning. NFCI.
llvm-svn: 341938
2018-09-11 13:42:15 +00:00
Kirill Bobyrev 0dee397e06 [clangd] NFC: Use uint32_t for FuzzyFindRequest limits
Reviewed By: ioeric

Differential Revision: https://reviews.llvm.org/D51860

llvm-svn: 341921
2018-09-11 10:31:38 +00:00
Kirill Bobyrev 5faf8a3d84 [clangd] Unbreak buildbots after r341802
Solution: use std::move when returning result from toJSON(...).
llvm-svn: 341832
2018-09-10 14:31:38 +00:00
Kirill Bobyrev 09f00dcf69 [clangd] Implement FuzzyFindRequest JSON (de)serialization
JSON (de)serialization of `FuzzyFindRequest` might be useful for both
D51090 and D51628. Also, this allows precise logging of the fuzzy find
requests.

Reviewed By: sammccall

Differential Revision: https://reviews.llvm.org/D51852

llvm-svn: 341802
2018-09-10 11:51:05 +00:00
Kirill Bobyrev 38a889c185 [clangd] Add symbol slab size to index memory consumption estimates
Currently, `SymbolIndex::estimateMemoryUsage()` returns the "overhead"
estimate, i.e. the estimate of the Index data structure excluding
backing data (such as Symbol Slab and Reference Slab). This patch
propagates information about paired data size where necessary.

Reviewed By: ioeric, sammccall

Differential Revision: https://reviews.llvm.org/D51539

llvm-svn: 341800
2018-09-10 11:46:07 +00:00
Sam McCall f469c64fef [clangd] Fix async index loading (from r341376).
Summary:
This wasn't actually async (due to std::future destructor blocking).
If it were, we would have clean shutdown issues if main returned
and destroyed Placeholder before the thread is done with it.

We could attempt to avoid any blocking by using shared_ptr or weak_ptr tricks so
the thread can detect Placeholder's destruction, but there are other potential
issues (e.g. loadIndex does tracing, and we'll destroy the tracer...)
Instead, once LSPServer::run returns, we wait for the index to finish loading
before exiting. Performance is not critical in this situation.

Reviewers: ilya-biryukov

Subscribers: ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D51674

llvm-svn: 341797
2018-09-10 10:00:47 +00:00
Kirill Bobyrev 5abe478a3d [clangd] NFC: Rename DexIndex to Dex
Also, cleanup some redundant includes.

Reviewed By: sammccall

Differential Revision: https://reviews.llvm.org/D51774

llvm-svn: 341784
2018-09-10 08:23:53 +00:00
Kirill Bobyrev 59491a1fa9 [clangd] Make advanceTo() faster on Posting Lists
If the current element is already beyond advanceTo()'s DocID, just
return instead of doing binary search. This simple optimization saves up
to 6-7% performance,

Reviewed By: ilya-biryukov

Differential Revision: https://reviews.llvm.org/D51802

llvm-svn: 341781
2018-09-10 07:57:28 +00:00
Eric Liu f76886859f [clangd] Canonicalize include paths in clangd.
Get rid of "../"  and "../../".

llvm-svn: 341645
2018-09-07 09:40:36 +00:00
Kirill Bobyrev e5536b972f [clangd] NFC: Document URIDistance
`URIDistance` constructor should mention that `Sources` must contain
*absolute paths*, not URIs. This is not very clear when looking at the
interface, especially given that `distance(...)` accepts `URI`, not an
absolute path which can give the wrong impression.

Reviewed By: sammccall

Differential Revision: https://reviews.llvm.org/D51691

llvm-svn: 341639
2018-09-07 09:18:58 +00:00