Commit Graph

429 Commits

Author SHA1 Message Date
Sam McCall 8faffec4e2 [clangd] Also accept flags from CLANGD_FLAGS variable.
This simplifies various workflows, particularly in debugging/development.
e.g. editors will tend to propagate flags, so you can run
`env CLANGD_FLAGS=-input-mirror-file=/tmp/mirror vim foo.cc` rather than
change the configuration in a persistent way.
(This also gives us a generic lever when we don't know how to customize
the flags in some particular LSP client).

While here, add a test for this and other startup logging, and fix a
couple of direct writes to errs() that should have been logs.

Differential Revision: https://reviews.llvm.org/D65153

llvm-svn: 366991
2019-07-25 07:54:48 +00:00
Kadir Cetinkaya 91e5f4b46b Revert "Revert r366458, r366467 and r366468"
This reverts commit 9c377105da.

[clangd][BackgroundIndexLoader] Directly store DependentTU while loading shard

Summary:
We were deferring the population of DependentTU field in LoadedShard
until BackgroundIndexLoader was consumed. This actually triggers a use after
free since the shards FileToTU was pointing at could've been moved while
consuming the Loader.

Reviewers: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64980

llvm-svn: 366559
2019-07-19 10:18:52 +00:00
Azharuddin Mohammed 9c377105da Revert r366458, r366467 and r366468
r366458 is causing test failures. r366467 and r366468 had to be reverted as
they were casuing conflict while reverting r366458.

r366468 [clangd] Remove dead code from BackgroundIndex
r366467 [clangd] BackgroundIndex stores shards to the closest project
r366458 [clangd] Refactor background-index shard loading

llvm-svn: 366551
2019-07-19 09:26:33 +00:00
Haojian Wu 6ae86ea275 [clangd] cleanup: unify the implemenation of checking a location is inside main file.
Summary: We have variant implementations in the codebase, this patch unifies them.

Reviewers: ilya-biryukov, kadircet

Subscribers: MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64915

llvm-svn: 366541
2019-07-19 08:33:39 +00:00
Kadir Cetinkaya cfa14ac2a7 [clangd] Remove dead code from BackgroundIndex
llvm-svn: 366468
2019-07-18 17:25:57 +00:00
Kadir Cetinkaya 006d1915e2 [clangd] BackgroundIndex stores shards to the closest project
Summary:
Changes persistance logic to store shards at the directory of closest
CDB. Previously we were storing all shards to directory of the CDB that
triggered indexing, it had its downsides.

For example, if you had two TUs coming from a different CDB but depending on the
same header foo.h, we will store the foo.h only for the first CDB, and it would
be missing for the second and we would never persist it since it was actually
present in the memory and persisted before.

This patch still stores only a single copy of a shard, but makes the directory a
function of the file name. So that the shard place will be unique even with
multiple CDBs accessing the file. This directory is determined as the first
directory containing a CDB in the file's parent directories, if no such
directory exists we make use of the home directory.

Reviewers: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64745

llvm-svn: 366467
2019-07-18 17:20:41 +00:00
Kadir Cetinkaya 40073f922a [clangd] Refactor background-index shard loading
Reviewers: sammccall

Subscribers: mgorny, ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64712

llvm-svn: 366458
2019-07-18 16:25:36 +00:00
Sam McCall 06377ae2e5 [clangd] Don't rebuild background index until we indexed one TU per thread.
Summary:
This increases the odds that the boosted file (cpp file matching header)
will be ready. (It always enqueues first, so it'll be present unless
another thread indexes *two* files before the first thread indexes one.)

Reviewers: kadircet

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D64682

llvm-svn: 366199
2019-07-16 10:17:06 +00:00
Sam McCall 697de1c24e [clangd] Fix off-by-one in CodeComplete and assertion in Dex
llvm-svn: 365955
2019-07-12 20:35:41 +00:00
Sam McCall 0f7146db9b [clangd] Prioritize indexing of files that share a basename with the open file.
Summary:
In practice, opening Foo.h will still often result in Foo.cpp making the
second index build instead of the first, as the rebuild policy doesn't
know to wait.

Reviewers: kadircet

Subscribers: ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D64575

llvm-svn: 365888
2019-07-12 10:18:42 +00:00
Sam McCall ad37ae18e2 [clangd] Avoid template in Task constructor, hopefully fix MSVC build
llvm-svn: 365794
2019-07-11 16:26:53 +00:00
Haojian Wu f3661a78bf [clangd] Remove an extra ";", NFC
llvm-svn: 365778
2019-07-11 13:58:56 +00:00
Sam McCall 7e27d86afb [clangd] Add priorities to background index queue, extract to separate class
Reviewers: kadircet

Subscribers: mgorny, ilya-biryukov, MaskRay, jkorous, arphaman, jfb, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D64560

llvm-svn: 365773
2019-07-11 13:34:38 +00:00
Kadir Cetinkaya ad54935c77 [clangd] Reland rL365634
This was reverted in rL365678, the failure was due to YAML parsing of
compile_commands.json.

Converting backslashes to forward slashes to fix the issue in unittest.

llvm-svn: 365748
2019-07-11 09:54:31 +00:00
Haojian Wu e6695821e5 Revert Recommit "[CommandLine] Remove OptionCategory and SubCommand caches from the Option class."
This reverts r365675 (git commit 43d75f9778)

The patch causes a crash in SupportTests (CommandLineTest.AliasesWithArguments).

llvm-svn: 365742
2019-07-11 08:54:28 +00:00
Matthew Voss 6d1a64e489 Revert "[clangd] Filter out non-governed files from broadcast"
This reverts commit d5214dfa7b.

It's causing failures, both in our local CI and the PS4 Windows bot.

http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-windows10pro-fast/builds/26872/steps/test/logs/stdio

llvm-svn: 365678
2019-07-10 18:16:35 +00:00
Don Hinton 43d75f9778 Recommit "[CommandLine] Remove OptionCategory and SubCommand caches from the Option class."
Previously reverted in 364141 due to buildbot breakage, and fixed here
by making GeneralCategory global a ManagedStatic.

Summary:
This change processes `OptionCategory`s and `SubCommand`s as they
are seen instead of caching them in the Option class and processing
them later.  Doing so simplifies the work needed to be done by the Global
parser and significantly reduces the size of the Option class to a mere 64
bytes.

Removing  the `OptionCategory` cache saved 24 bytes, and removing
the `SubCommand` cache saved an additional 48 bytes, for a total of a
72 byte reduction.

Reviewed By: serge-sans-paille

Tags: #llvm, #clang

Differential Revision: https://reviews.llvm.org/D62105

llvm-svn: 365675
2019-07-10 17:57:05 +00:00
Kadir Cetinkaya d5214dfa7b [clangd] Filter out non-governed files from broadcast
Summary:
This also turns off implicit discovery of additional compilation
databases.

Reviewers: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64247

llvm-svn: 365634
2019-07-10 14:11:46 +00:00
Sam McCall 2f760c44e6 [clangd] Rewrite of logic to rebuild the background index serving structures.
Summary:
Previously it was rebuilding every 5s by default, which was much too frequent
in the long run - the goal was to provide an early build. There were also some
bugs. There were also some bugs, and a dedicated thread was used in production
but not tested.

 - rebuilds are triggered by #TUs built, rather than time. This should scale
   more sensibly to fast vs slow machines.
 - there are two separate indexed-TU thresholds to trigger index build: 5 TUs
   for the first build, 100 for subsequent rebuilds.
 - rebuild is always done on the regular indexing threads, and is affected by
   blockUntilIdle. This means unit/lit tests run the production configuration.
 - fixed a bug where we'd rebuild after attempting to load shards, even if there
   were no shards.
 - the BackgroundIndexTests don't really test the subtleties of the rebuild
   policy (for determinism, we call blockUntilIdle, so rebuild-on-idle is enough
   to pass the tests). Instead, we expose the rebuilder as a separate class and
   have fine-grained tests for it.

Reviewers: kadircet

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, jfb, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D64291

llvm-svn: 365531
2019-07-09 18:30:49 +00:00
Sam McCall 674d8a947c [clangd] Use xxhash instead of SHA1 for background index file digests.
Summary:
Currently SHA1 is about 10% of our CPU, this patch reduces it to ~1%.

xxhash is a well-defined (stable) non-cryptographic hash optimized for
fast checksums (like crc32).
Collisions shouldn't be a problem, despite the reduced length:
 - for actual file content (used to invalidate bg index shards), there
   are only two versions that can collide (new shard and old shard).
 - for file paths in bg index shard filenames, we would need 2^32 files
   with the same filename to expect a collision. Imperfect hashing may
   reduce this a bit but it's well beyond what's plausible.

This will invalidate shards on disk (as usual; I bumped the version),
but this time the filenames are changing so the old files will stick
around :-( So this is more expensive than the usual bump, but would be
good to land before the v9 branch when everyone will start using bg index.

Reviewers: kadircet

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D64306

llvm-svn: 365311
2019-07-08 11:33:17 +00:00
Sam McCall 8f9e4d92e0 [clangd] Encapsulate fields in dex token. NFC
llvm-svn: 365288
2019-07-08 02:37:06 +00:00
Kadir Cetinkaya 51702765b4 [clangd] Fix breakage on gcc 5.4
llvm-svn: 365140
2019-07-04 13:47:51 +00:00
Kadir Cetinkaya 12b14869dc [clangd] Make HadErrors part of background index's internal state
Reviewers: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64147

llvm-svn: 365123
2019-07-04 09:52:12 +00:00
Kadir Cetinkaya adbb347ffe [clangd] Add HadErrors field into shards
Reviewers: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64133

llvm-svn: 365122
2019-07-04 09:52:04 +00:00
Kadir Cetinkaya 11e1c50b08 [clangd] Store hash of command line in index shards.
Summary: This is to enable cache invalidation when command line flags changes.

Reviewers: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D64018

llvm-svn: 365121
2019-07-04 09:51:53 +00:00
Kadir Cetinkaya a6fedc8bd6 [clangd] Also cache failures while indexing
Summary:
Clangd currently doesn't cache any indexing failures, which results in
retrying those failed files even if their contents haven't changed.

Reviewers: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D63986

llvm-svn: 365120
2019-07-04 09:51:43 +00:00
Haojian Wu 7c251fa069 [clangd] Collect the refs when the main file is header.
Summary:
Previously, we only collect refs of the symbols which are declared in
the preamble and referenced in the main file, it works well when the
main file is .cpp file.

However, when the main file is .h file (when opening a .h file in the
editor), we don't collect refs of the symbol declared in this file, so we miss
these refs in our dynamic index.

A typical scenario:

1. Open Foo.h (which contains class Foo)
2. Open Foo.cpp, call find references for Foo

And we only get refs from Foo.cpp.

Reviewers: ilya-biryukov

Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D63818

llvm-svn: 364893
2019-07-02 09:16:21 +00:00
Fangrui Song 78ee2fbf98 Cleanup: llvm::bsearch -> llvm::partition_point after r364719
llvm-svn: 364720
2019-06-30 11:19:56 +00:00
Fangrui Song 2d2cb77e45 [ADT] Implement llvm::bsearch() with std::partition_point()
Summary:
Delete the begin-end form because the standard std::partition_point
can be easily used as a replacement.

The ranges-style llvm::bsearch will be renamed to llvm::partition_point
in the next clean-up patch.

The name "bsearch" doesn't meet people's expectation because in C:

> If two or more members compare equal, which member is returned is unspecified.

Reviewed By: sammccall

Differential Revision: https://reviews.llvm.org/D63718

llvm-svn: 364719
2019-06-30 09:17:59 +00:00
JF Bastien 0e82895826 BitStream reader: propagate errors
The bitstream reader handles errors poorly. This has two effects:

 * Bugs in file handling (especially modules) manifest as an "unexpected end of
   file" crash
 * Users of clang as a library end up aborting because the code unconditionally
   calls `report_fatal_error`

The bitstream reader should be more resilient and return Expected / Error as
soon as an error is encountered, not way late like it does now. This patch
starts doing so and adopting the error handling where I think it makes sense.
There's plenty more to do: this patch propagates errors to be minimally useful,
and follow-ups will propagate them further and improve diagnostics.

https://bugs.llvm.org/show_bug.cgi?id=42311
<rdar://problem/33159405>

Differential Revision: https://reviews.llvm.org/D63518

llvm-svn: 364464
2019-06-26 19:50:12 +00:00
Haojian Wu 34f5188d0f [clangd] Add include-mapping for C symbols.
Summary:
This resolves the issue of introducing c++-style includes for C files.

- refactor the gen_std.py, make it reusable for parsing C symbols.
- add a language mode to the mapping method to use different mapping for
  C and C++ files.

Reviewers: kadircet

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, jfb, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D63270

llvm-svn: 364044
2019-06-21 13:32:18 +00:00
Haojian Wu b5ce4e5ea3 [clangd] Perform merge for main file symbols.
Summary:
Previously, we randomly pick one main file symbol in dynamic index, we
may loose the ideal symbol (with definition location) in the index.

It fixes the issue where sometimes we fail to go to the symbol definition, see:

1. call go-to-decl on Foo in Foo.cpp
2. jump to Foo.h, call go-to-def on Foo in Foo.h

we can't go back to Foo.cpp -- because we open Foo.cpp, Foo.h in clangd, both
files have Foo symbol (one with def&decl, one with decl only), we randomely
choose one.

Reviewers: kadircet

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D63425

llvm-svn: 363568
2019-06-17 14:49:18 +00:00
Don Hinton b3fc9fde2c Fix gcc-05.4 bot failures caused by in r363481 "[clangd] Index API and implementations for relations"
Use std::make_tuple instead of initializer list to make gcc-5.4 happy.

See https://reviews.llvm.org/D62839 for details.

llvm-svn: 363504
2019-06-16 01:09:41 +00:00
Nathan Ridge f1e6f5713c [clangd] Index API and implementations for relations
Summary:
This builds on the relations support added in D59407, D62459,
and D62471 to expose relations in SymbolIndex and its
implementations.

Reviewers: kadircet

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D62839

llvm-svn: 363481
2019-06-15 02:26:47 +00:00
Nathan Ridge 73e6f47da2 [clangd] SymbolCollector support for relations
Summary:
The only relation currently collected is RelationBaseOf, because this is
all we need for type hierarchy subtypes. Additional relations can be
collected in the future as the need arises.

This patch builds on D59407 and D62459.

Reviewers: kadircet

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D62471

llvm-svn: 362467
2019-06-04 04:25:44 +00:00
Nathan Ridge 92524f9bf8 [clangd] Serialization support for RelationSlab
Summary: This builds on D59407 to provide YAML and RIFF serialization support.

Reviewers: kadircet

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D62459

llvm-svn: 362353
2019-06-03 05:07:52 +00:00
Nathan Ridge 3fc299df3d [clangd] Add RelationSlab
Summary:
RelationSlab is a new index data structure that stores relations between
symbols.

Reviewers: kadircet

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D59407

llvm-svn: 362352
2019-06-03 04:55:46 +00:00
Nathan Ridge b2f45ac299 [clangd] clang-format SymbolCollector.cpp
llvm-svn: 362176
2019-05-30 23:54:43 +00:00
Haojian Wu 228b130a4b [clangd] Fix buildbot error.
llvm-svn: 361960
2019-05-29 14:11:53 +00:00
Haojian Wu 4c5a0d1683 [clangd] Remove the whitelist std symbols in CanonicalIncludes.
Summary: These symbols have been included via StdSymbolMap.inc.

Reviewers: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D62527

llvm-svn: 361952
2019-05-29 12:08:11 +00:00
Kadir Cetinkaya 40177ac6d1 [clangd] Bump index version and get rid of wrong assertion
Summary:
After rL360344, BackgroundIndex expects symbols with zero refcounts.
Therefore existing index files are no longer valid.

Assertion regarding finding target of a reference was wrong, since
background-index might still be going on or we might've loaded only some part of
the shards and might be missing the declaring shards for the symbol.

Reviewers: ilya-biryukov

Subscribers: MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D61734

llvm-svn: 360349
2019-05-09 15:07:53 +00:00
Kadir Cetinkaya 70674549f1 [clangd] Count number of references while merging RefSlabs inside FileIndex
Summary:
For counting number of references clangd was relying on merging every
duplication of a symbol. Unfortunately this does not apply to FileIndex(and one
of its users' BackgroundIndex), since we get rid of duplication by simply
dropping symbols coming from non-canonical locations. So only one or two(coming
from canonical declaration header and defined source file, if exists)
replications of the same symbol reaches merging step.

This patch changes reference counting logic to rather count number of different
RefSlabs a given SymbolID exists.

Reviewers: ilya-biryukov

Subscribers: ioeric, MaskRay, jkorous, mgrang, arphaman, jdoerfert, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D59481

llvm-svn: 360344
2019-05-09 14:22:07 +00:00
Ilya Biryukov db68b104d8 [clangd] Use AsyncTaskRunner in BackgroundIndex instead of std::thread
Summary:
To unify the way we create threads in clangd.
This should simplify landing D50993.

Reviewers: kadircet

Reviewed By: kadircet

Subscribers: MaskRay, jkorous, arphaman, jfb, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D61724

llvm-svn: 360332
2019-05-09 12:04:07 +00:00
Sam McCall ec026532d6 [clangd] Fix header-guard check for include insertion, and don't index header guards.
Summary:
Both of these attempt to check whether a header guard exists while parsing the
file. However the file is only marked as guarded once clang finishes processing
it. We defer the checks and work until SymbolCollector::finish().

This is ugly and ad-hoc, deferring *all* work might be cleaner.

Reviewers: kadircet

Subscribers: ilya-biryukov, MaskRay, jkorous, mgrang, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D61442

llvm-svn: 359880
2019-05-03 13:17:29 +00:00
Sam McCall 1b29dec05f Reapply r359778: [clangd] Fix code completion of macros defined in the preamble region of the main file.
The bad assert has been removed, and updateOutOfDateIdentifier has been guarded.
This reverts commit r359796.

llvm-svn: 359799
2019-05-02 16:12:36 +00:00
Simon Pilgrim 73c44e45ec Revert rL359778 : [clangd] Fix code completion of macros defined in the preamble region of the main file.
Summary:
This is a tricky case (we baked the assumption that symbols come from
the preamble xor mainfile pretty deeply) and the fix is a bit of a hack:
We look at the code to guess the macro names, and deserialize them from
the preamble "by hand".

Reviewers: ilya-biryukov

Subscribers: ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D60937
........
Fix buildbots http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-scei-ps4-ubuntu-fast/builds/47684/

llvm-svn: 359796
2019-05-02 15:47:33 +00:00
Sam McCall 929f639eb8 [clangd] Fix code completion of macros defined in the preamble region of the main file.
Summary:
This is a tricky case (we baked the assumption that symbols come from
the preamble xor mainfile pretty deeply) and the fix is a bit of a hack:
We look at the code to guess the macro names, and deserialize them from
the preamble "by hand".

Reviewers: ilya-biryukov

Subscribers: ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D60937

llvm-svn: 359778
2019-05-02 11:06:44 +00:00
Fangrui Song e54a93fe6f [clangd] Delete an unused declaration
llvm-svn: 359674
2019-05-01 12:16:37 +00:00
Kadir Cetinkaya 250eae2452 [clangd] Fix serialization logic for Origin and Flags.
llvm-svn: 359470
2019-04-29 17:25:58 +00:00
Sam McCall e3559eee63 [clangd] Optimize "don't include me" check.
Summary:
llvm::Regex is really slow, and regex evaluation during preamble indexing was
showing up as 25% on a profile of clangd in a codebase with large preambles.

Reviewers: ilya-biryukov

Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D61120

llvm-svn: 359214
2019-04-25 17:47:07 +00:00
Kadir Cetinkaya f8537b3c69 [clangd] Use llvm::set_thread_priority in background-index
Reviewers: gribozavr

Subscribers: krytarowski, ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, jfb, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D60865

llvm-svn: 358664
2019-04-18 13:46:40 +00:00
Sam McCall a96efb654e [clangd] Recognize "don't include me directly" pattern, and suppress include insertion.
Summary:
Typically used with umbrella headers, e.g. GTK:

 #if !defined (__GTK_H_INSIDE__) && !defined (GTK_COMPILATION)
 #error "Only <gtk/gtk.h> can be included directly."
 #endif

Heuristic is fairly conservative, a quick code search over github showed
a fair number of hits and few/no false positives. (Not all were umbrella
headers, but I'd be happy avoiding include insertion for all of them).

We may want to relax the heuristic later to catch more cases.

Reviewers: ioeric

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D60815

llvm-svn: 358605
2019-04-17 18:33:07 +00:00
Sam McCall 62e2472321 [clangd] Include insertion: require header guards, drop other heuristics, treat .def like .inc.
Summary:
We do have some reports of include insertion behaving badly in some
codebases. Requiring header guards both makes sense in principle, and is
likely to disable this "nice-to-have" feature in codebases where headers don't
follow the expected pattern.

With this we can drop some other heuristics, such as looking at file
extensions to detect known non-headers - implementation files have no guards.

One wrinkle here is #import - objc headers may not have guards because
they're intended to be used via #import. If the header is the main file
or is #included, we won't collect locations - merge should take care of
this if we see the file #imported somewhere. Seems likely to be OK.

Headers which have a canonicalization (stdlib, IWYU) are exempt from this check.
*.inc files continue to be handled by looking up to the including file.
This patch also adds *.def here - tablegen wants this pattern too.

In terms of code structure, the division between SymbolCollector and
CanonicalIncludes has shifted: SymbolCollector is responsible for more.
This is because SymbolCollector has all the SourceManager/HeaderSearch access
needed for checking for guards, and we interleave these checks with the *.def
checks in a loop (potentially).
We could hand all the info into CanonicalIncludes and put the logic there
if that's preferable.

Reviewers: ioeric

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D60316

llvm-svn: 358571
2019-04-17 10:36:02 +00:00
Sam McCall 277754c71d [clangd] lower_bound -> bsearch, NFC
llvm-svn: 358561
2019-04-17 07:00:36 +00:00
Kadir Cetinkaya bb6cd8254c [clangd] Fallback to OrigD when SLoc is invalid
Summary:
Some implicit/built-in decls lack the source location
information. Fallback to OrigD that we've seen in the source code
instead of the canonical one in those cases.

Reviewers: sammccall

Subscribers: cfe-commits, arphaman, jkorous, MaskRay, ioeric, ilya-biryukov

Tags: #clang

Differential Revision: https://reviews.llvm.org/D60689

llvm-svn: 358413
2019-04-15 14:38:46 +00:00
Kadir Cetinkaya a9ce09be03 [clangd] Bump clangd-index version for TemplateArgument changes
llvm-svn: 358383
2019-04-15 09:18:57 +00:00
Kadir Cetinkaya 4f789e1b39 [clangd] Show template argument list in workspacesymbols and documentsymbols responses
Summary:
Last part of re-landing rC356541. Puts TemplateArgumentsList into
responses of the above mentioned two requests.

Reviewers: ioeric, ilya-biryukov

Subscribers: MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D59641

llvm-svn: 358274
2019-04-12 10:09:37 +00:00
Kadir Cetinkaya 79063de95c [clangd] Add TemplateArgumentList into Symbol
Summary:
Part of re-landing rC356541 with D59599. Changes the way we store
template arguments, previous patch was storing them inside Name field of Symbol.
Which was violating the assumption:
```Symbol::Scope+Symbol::Name == clang::clangd::printQualifiedName```
which was made in multiple places inside codebase. This patch instead moves
those arguments into their own field. Currently the field is meant to be
human-readable, can be made structured if need be.

Reviewers: ioeric, ilya-biryukov, gribozavr

Subscribers: MaskRay, jkorous, arphaman, jdoerfert, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D59640

llvm-svn: 358273
2019-04-12 10:09:24 +00:00
Eric Liu 00d99bd1c4 [clangd] Use identifiers in file as completion candidates when build is not ready.
Summary:
o Lex the code to get the identifiers and put them into a "symbol" index.
o Adds a new completion mode without compilation/sema into code completion workflow.
o Make IncludeInserter work even when no compile command is present, by avoiding
inserting non-verbatim headers.

Reviewers: sammccall

Reviewed By: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, jdoerfert, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D60126

llvm-svn: 358159
2019-04-11 09:36:36 +00:00
Sam McCall 2d02c6df6b [clangd] Fix non-indexing of builtin functions like printf when the TU is C
llvm-svn: 358098
2019-04-10 16:26:58 +00:00
Sam McCall 4e56502be2 [clangd] Stop passing around PCHContainerOperations, just create it in place. NFC
llvm-svn: 357689
2019-04-04 12:56:03 +00:00
Ilya Biryukov bea52e5289 [clangd] Use capacity() instead of size() in RefSlab::bytes()
Patch by Nathan Ridge.

Reviewers: gribozavr

Reviewed By: gribozavr

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D60040

llvm-svn: 357454
2019-04-02 08:24:37 +00:00
Sam McCall 4180a7cd83 Disable warnings when indexing as a standalone action.
Summary:
- we don't record the warnings at all
- we don't want to stop indexing if we hit error-limit due to warnings
- this allows some analyses to be skipped which can save some CPU

https://github.com/clangd/clangd/issues/24

Reviewers: hokein

Subscribers: ilya-biryukov, ioeric, jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D59935

llvm-svn: 357186
2019-03-28 17:07:28 +00:00
Sam McCall a69698f45f [clangd] Support utf-8 offsets (rather than utf-16) as a protocol extension
Summary:
Still some pieces to go here: unit tests for new SourceCode functionality and
a command-line flag to force utf-8 mode. But wanted to get early feedback.

Reviewers: hokein

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, jdoerfert, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D58275

llvm-svn: 357102
2019-03-27 17:47:49 +00:00
Jordan Rupprecht ce3d670097 Revert "[clangd] Print arguments in template specializations"
This reverts commit 44a63f6a15. It segfaults on an internal test case (will follow up off thread).

llvm-svn: 356623
2019-03-20 22:51:56 +00:00
Kadir Cetinkaya 44a63f6a15 [clangd] Print arguments in template specializations
Reviewers: ilya-biryukov

Subscribers: ioeric, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D59354

llvm-svn: 356541
2019-03-20 09:43:38 +00:00
Kadir Cetinkaya 8665802202 [clangd] Add support for type hierarchy (super types only for now)
Summary:
Patch by Nathan Ridge(@nridge)!

This is an LSP extension proposed here:
https://github.com/Microsoft/vscode-languageserver-node/pull/426

An example client implementation can be found here:
https://github.com/theia-ide/theia/pull/3802

Reviewers: kadircet, sammccall

Reviewed By: kadircet

Subscribers: jdoerfert, sammccall, cfe-commits, mgorny, dschaefer, simark, ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet

Tags: #clang

Differential Revision: https://reviews.llvm.org/D56370

llvm-svn: 356445
2019-03-19 09:27:04 +00:00
Haojian Wu 7f51b5dc32 [clangd] Using symbol name to map includes for STL symbols.
Summary:
Using suffix path mapping relies on the STL implementations, and it is
not portable. This patch is using symbol name mapping, which should
work with different STL implementations, fix clangd/clangd#9.

To generate the symbol mapping, we parse the cppreference symbol index
page to build a lookup table.

The mapping is not completed, a few TODOs:
  - support symbols from different headers (e.g. std::move)
  - support STL macros
  - support symbols from std's sub-namespaces (e.g. chrono)

Reviewers: ioeric, jfb, serge-sans-paille

Reviewed By: ioeric

Subscribers: sammccall, klimek, ilya-biryukov, ioeric, MaskRay, jkorous, mgrang, arphaman, kadircet, jfb, jdoerfert, cfe-commits

Tags: #clang-tools-extra, #clang

Differential Revision: https://reviews.llvm.org/D58345

llvm-svn: 356134
2019-03-14 11:25:26 +00:00
Haojian Wu 1f28a4a17f [clangd] Fix an out-of-date FIXME, NFC.
llvm-svn: 356127
2019-03-14 10:01:07 +00:00
Haojian Wu 7ce0d6cc33 [clangd] Build Dex index after loading all shards in BackgroundIndex.
Summary:
Currently after loadding all shards, we use MemIndex which has poor query
performance, we should use Dex.

Reviewers: kadircet

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D59350

llvm-svn: 356126
2019-03-14 09:57:10 +00:00
Kadir Cetinkaya d9c174648e [clangd] Store explicit template specializations in index for code navigation purposes
Summary:
This introduces ~4k new symbols, and ~10k refs for LLVM. We need that
information for providing better code navigation support:
- When references for a class template is requested, we should return these specializations as well.
- When children of a specialization is requested, we should be able to query for those symbols(instead of just class template)

Number of symbols: 378574 -> 382784
Number of refs: 5098857 -> 5110689

Reviewers: hokein, gribozavr

Reviewed By: gribozavr

Subscribers: nridge, ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, jdoerfert, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D59083

llvm-svn: 356125
2019-03-14 08:35:17 +00:00
Kadir Cetinkaya 3d8e289f71 [clangd] Default initialize SymInfo
llvm-svn: 356032
2019-03-13 08:42:15 +00:00
Kadir Cetinkaya a7cdfd9cb5 [clangd] Respect Origin option in createStaticIndexingAction
Summary:
Currently createStaticIndexingAction always set Origin to Static, which
makes it hard to change it later on by different indexers(One needs to go over
each symbol making a new copy).

This patch changes that behavior to rather respect it if set by user.

Reviewers: ioeric

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D59205

llvm-svn: 355820
2019-03-11 11:01:14 +00:00
Kadir Cetinkaya 890dfddef0 [clangd] Remove ./ and ../ in the file paths
Reviewers: hokein

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D59084

llvm-svn: 355681
2019-03-08 09:57:33 +00:00
Kadir Cetinkaya 017cc6c190 [clangd] Make sure constructors do not reference class
Reviewers: gribozavr

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D58815

llvm-svn: 355679
2019-03-08 09:54:37 +00:00
Haojian Wu f334231176 [clangd] Deduplicate Refs on the fly.
Summary:
Currently, we only do deduplication when we flush final results. We may
have huge duplications (refs from headers) during the indexing period (running
clangd-indexer on Chromium).

With this change, clangd-indexer can index the whole chromium projects
(48 threads, 40 GB peak memory usage).

Reviewers: kadircet

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, mgrang, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D59092

llvm-svn: 355676
2019-03-08 09:26:30 +00:00
Fangrui Song ffe9f00cfe Fix file headers. NFC
llvm-svn: 355188
2019-03-01 09:52:53 +00:00
Dmitri Gribenko 2098b86b96 Added missing license headers
Reviewers: ioeric

Subscribers: jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D58781

llvm-svn: 355092
2019-02-28 14:01:11 +00:00
Dmitri Gribenko 7344188b72 Use ArrayRef::copy, instead of copying data manually
Reviewers: ioeric

Subscribers: jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D58782

llvm-svn: 355091
2019-02-28 14:00:26 +00:00
Dmitri Gribenko cb83ea6274 Moved Ref into its own header and implementation file
Reviewers: ioeric

Subscribers: mgorny, jkorous, mgrang, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D58778

llvm-svn: 355090
2019-02-28 13:49:25 +00:00
Dmitri Gribenko 08b49b5338 Moved Symbol into its own header and implementation file
Reviewers: ioeric

Subscribers: mgorny, jkorous, arphaman, kadircet, jdoerfert, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D58774

llvm-svn: 355088
2019-02-28 13:23:03 +00:00
Dmitri Gribenko dba22a3908 Moved SymbolOrigin into its own header and implementation file
Reviewers: ioeric

Subscribers: mgorny, jkorous, arphaman, kadircet, jdoerfert, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D58773

llvm-svn: 355086
2019-02-28 12:31:49 +00:00
Dmitri Gribenko 5306a71480 Moved SymbolLocation into its own header and implementation file
Reviewers: ioeric

Subscribers: mgorny, jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D58768

llvm-svn: 355082
2019-02-28 11:02:01 +00:00
Dmitri Gribenko ed2539a608 Moved DenseMap support for SymbolID into SymbolID.h
llvm-svn: 355081
2019-02-28 11:00:44 +00:00
Haojian Wu da79dcc317 [clangd] Drop documentation in static index if symbols are not indexed for completion.
Summary:
This is a further optimization of r350803, we drop docs in static index for
symbols not being indexed for completion, while keeping the docs in dynamic
index (we rely on dynamic index to get docs for class members).

Reviewers: ilya-biryukov

Reviewed By: ilya-biryukov

Subscribers: ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D56539

llvm-svn: 354792
2019-02-25 16:00:00 +00:00
Ilya Biryukov 2754942cba [clangd] Store index in '.clangd/index' instead of '.clangd-index'
Summary: To take up the .clangd folder for other potential uses in the future.

Reviewers: kadircet, sammccall

Reviewed By: kadircet

Subscribers: ioeric, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D58440

llvm-svn: 354505
2019-02-20 19:08:06 +00:00
Eric Liu d11fbf523d [clangd] Prefer location from codegen files when merging symbols.
Summary:
For example, if an index symbol has location in a .proto file and an AST symbol
has location in a generated .proto.h file, then we prefer location in .proto
which is more meaningful to users.

Also use `mergeSymbols` to get the preferred location between AST location and index location in go-to-def.

Reviewers: sammccall

Subscribers: ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D58037

llvm-svn: 353708
2019-02-11 15:05:29 +00:00
Kadir Cetinkaya ee23633eab [clangd] Reduce number of threads used by BackgroundIndex to number of physical cores.
Summary:
clangd is using as many threads as logical cores for BackgroundIndex
by default. We observed that it increases latency of foreground tasks.

This patch aims to change that default to number of physical cores to get rid of
that extra latency.

Reviewers: ilya-biryukov

Reviewed By: ilya-biryukov

Subscribers: ioeric, MaskRay, jkorous, arphaman, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D57819

llvm-svn: 353422
2019-02-07 16:04:30 +00:00
Eric Liu d7cb5b8a33 [clangd] Add type boost to fuzzy find in Dex.
Summary:
No noticeable impact on code completions overall except some improvement on
cross-namespace completion.

Reviewers: sammccall, ilya-biryukov

Reviewed By: sammccall

Subscribers: MaskRay, jkorous, arphaman, kadircet, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D57815

llvm-svn: 353310
2019-02-06 15:36:23 +00:00
Kadir Cetinkaya c3dcd2673e [clangd] Enable include insertion for static index
Summary:
This enables include insertion by adding canonical includes into
preambledata.

Reviewers: ioeric, ilya-biryukov

Subscribers: javed.absar, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D57508

llvm-svn: 353054
2019-02-04 16:19:57 +00:00
Haojian Wu 7b6f874717 [clangd] Index main-file macros (bug 39761)
Patch by Nathan Ridge!

Differential Revision: https://reviews.llvm.org/D55739

llvm-svn: 352367
2019-01-28 14:11:49 +00:00
Haojian Wu aa3ed5a983 [clangd] NFC: fix clang-tidy warnings.
Most are about llvm code style violation (found via
readability-identifier-naming check).

llvm-svn: 352205
2019-01-25 15:14:03 +00:00
Ilya Biryukov 4e0c400a47 [clangd] Fix crash due to ObjCPropertyDecl
With ObjCPropertyDecl, ASTNode.OrigD can be a ObjCPropertyImplDecl
which is not a NamedDecl, leading to a crash since the code
incorrectly assumes ASTNode.OrigD will always be a NamedDecl.

Change by dgoldman (David Goldman)!

Differential Revision: https://reviews.llvm.org/D56916

llvm-svn: 351941
2019-01-23 10:35:12 +00:00
Haojian Wu 72ef4510b6 [clangd] Fix the `-Wtype-limits` warning, NFC
The assertion is always true, and triggers a compiler warning, so remove it.

llvm-svn: 351809
2019-01-22 12:21:25 +00:00
Kadir Cetinkaya be6b35dac4 [clangd] Filter out plugin related flags and move all commandline manipulations into OverlayCDB.
Summary:
Some projects make use of clang plugins when building, but clangd is
not aware of those plugins therefore can't work with the same compile command
arguments.

There were multiple places clangd performed commandline manipulations,
 this one also moves them all into OverlayCDB.

Reviewers: ilya-biryukov

Subscribers: klimek, sammccall, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D56841

llvm-svn: 351788
2019-01-22 09:10:20 +00:00
Chandler Carruth 2946cd7010 Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636
2019-01-19 08:50:56 +00:00
Ilya Biryukov 1c48f0383c [clangd] Make background index less chatty
Summary:
It is producing too much input in non-verbose mode,
i.e. a message per indexed file

Reviewers: sammccall, kadircet

Reviewed By: sammccall

Subscribers: ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D56915

llvm-svn: 351563
2019-01-18 17:04:26 +00:00
Kadir Cetinkaya 226af75a02 [clangd] Fix updated file detection logic in indexing
Summary:
Files without any symbols were never marked as updated during indexing, which resulted in failure while writing shards for these files.

This patch fixes the logic to mark files that are seen for the first time but don't contain any symbols as updated.

Reviewers: ilya-biryukov

Reviewed By: ilya-biryukov

Subscribers: ioeric, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D56592

llvm-svn: 351170
2019-01-15 09:03:33 +00:00
Haojian Wu c34f022bfe [clangd] Add Limit parameter for xref.
Reviewers: sammccall

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, kadircet, cfe-commits

Differential Revision: https://reviews.llvm.org/D56597

llvm-svn: 351081
2019-01-14 18:11:09 +00:00
Kadir Cetinkaya 560b853ccf [clangd] Fix a reference invalidation
Summary: Fix for the breakage in http://green.lab.llvm.org/green/job/clang-stage1-configure-RA/52811/consoleFull#-42777206a1ca8a51-895e-46c6-af87-ce24fa4cd561

Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, arphaman, cfe-commits

Differential Revision: https://reviews.llvm.org/D56656

llvm-svn: 351052
2019-01-14 11:24:07 +00:00