[clangd] Introduce a "Symbol" class.
Summary:
* The "Symbol" class represents a C++ symbol in the codebase, containing all the
information of a C++ symbol needed by clangd. clangd will use it in clangd's
AST/dynamic index and global/static index (code completion and code
navigation).
* The SymbolCollector (another IndexAction) will be used to recollect the
symbols when the source file is changed (for ASTIndex), or to generate
all C++ symbols for the whole project.
In the long term (when index-while-building is ready), clangd should share a
same "Symbol" structure and IndexAction with index-while-building, but
for now we want to have some stuff working in clangd.
Reviewers: ioeric, sammccall, ilya-biryukov, malaperle
Reviewed By: sammccall
Subscribers: malaperle, klimek, mgorny, cfe-commits
Differential Revision: https://reviews.llvm.org/D40897
llvm-svn: 320486
2017-12-12 23:42:10 +08:00
|
|
|
//===--- Index.cpp -----------------------------------------------*- C++-*-===//
|
|
|
|
//
|
2019-01-19 16:50:56 +08:00
|
|
|
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
|
|
|
|
// See https://llvm.org/LICENSE.txt for license information.
|
|
|
|
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
[clangd] Introduce a "Symbol" class.
Summary:
* The "Symbol" class represents a C++ symbol in the codebase, containing all the
information of a C++ symbol needed by clangd. clangd will use it in clangd's
AST/dynamic index and global/static index (code completion and code
navigation).
* The SymbolCollector (another IndexAction) will be used to recollect the
symbols when the source file is changed (for ASTIndex), or to generate
all C++ symbols for the whole project.
In the long term (when index-while-building is ready), clangd should share a
same "Symbol" structure and IndexAction with index-while-building, but
for now we want to have some stuff working in clangd.
Reviewers: ioeric, sammccall, ilya-biryukov, malaperle
Reviewed By: sammccall
Subscribers: malaperle, klimek, mgorny, cfe-commits
Differential Revision: https://reviews.llvm.org/D40897
llvm-svn: 320486
2017-12-12 23:42:10 +08:00
|
|
|
//
|
|
|
|
//===----------------------------------------------------------------------===//
|
|
|
|
|
|
|
|
#include "Index.h"
|
[clangd] Move non-clang base pieces into separate support/ lib. NFCI
Summary:
This enforces layering, reduces a sprawling clangd/ directory, and makes life
easier for embedders.
Reviewers: kbobyrev
Subscribers: mgorny, ilya-biryukov, javed.absar, MaskRay, jkorous, arphaman, jfb, kadircet, usaxena95, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D79014
2020-04-28 23:49:17 +08:00
|
|
|
#include "support/Logger.h"
|
2017-12-14 20:17:14 +08:00
|
|
|
#include "llvm/ADT/StringExtras.h"
|
[clangd] Support multiple #include headers in one symbol.
Summary:
Currently, a symbol can have only one #include header attached, which
might not work well if the symbol can be imported via different #includes depending
on where it's used. This patch stores multiple #include headers (with # references)
for each symbol, so that CodeCompletion can decide which include to insert.
In this patch, code completion simply picks the most popular include as the default inserted header. We also return all possible includes and their edits in the `CodeCompletion` results.
Reviewers: sammccall
Reviewed By: sammccall
Subscribers: mgrang, ilya-biryukov, MaskRay, jkorous, arphaman, kadircet, cfe-commits
Differential Revision: https://reviews.llvm.org/D51291
llvm-svn: 341304
2018-09-03 18:18:21 +08:00
|
|
|
#include "llvm/ADT/StringRef.h"
|
[clangd] Define a compact binary serialization fomat for symbol slab/index.
Summary:
This is intended to replace the current YAML format for general use.
It's ~10x more compact than YAML, and ~40% more compact than gzipped YAML:
llvmidx.riff = 20M, llvmidx.yaml = 272M, llvmidx.yaml.gz = 32M
It's also simpler/faster to read and write.
The format is a RIFF container (chunks of (type, size, data)) with:
- a compressed string table
- simple binary encoding of symbols (with varints for compactness)
It can be extended to include occurrences, Dex posting lists, etc.
There's no rich backwards-compatibility scheme, but a version number is included
so we can detect incompatible files and do ad-hoc back-compat.
Alternatives considered:
- compressed YAML or JSON: bulky and slow to load
- llvm bitstream: confusing model and libraries are hard to use. My attempt
produced slightly larger files, and the code was longer and slower.
- protobuf or similar: would be really nice (esp for back-compat) but the
dependency is a big hassle
- ad-hoc binary format without a container: it seems clear we're going
to add posting lists and occurrences here, and that they will benefit
from sharing a string table. The container makes it easy to debug
these pieces in isolation, and make them optional.
Reviewers: ioeric
Subscribers: mgorny, ilya-biryukov, MaskRay, jkorous, mgrang, arphaman, kadircet, cfe-commits
Differential Revision: https://reviews.llvm.org/D51585
llvm-svn: 341375
2018-09-05 00:16:50 +08:00
|
|
|
#include "llvm/Support/Error.h"
|
2018-02-09 22:42:01 +08:00
|
|
|
#include "llvm/Support/raw_ostream.h"
|
2019-02-28 21:49:25 +08:00
|
|
|
#include <limits>
|
[clangd] Introduce a "Symbol" class.
Summary:
* The "Symbol" class represents a C++ symbol in the codebase, containing all the
information of a C++ symbol needed by clangd. clangd will use it in clangd's
AST/dynamic index and global/static index (code completion and code
navigation).
* The SymbolCollector (another IndexAction) will be used to recollect the
symbols when the source file is changed (for ASTIndex), or to generate
all C++ symbols for the whole project.
In the long term (when index-while-building is ready), clangd should share a
same "Symbol" structure and IndexAction with index-while-building, but
for now we want to have some stuff working in clangd.
Reviewers: ioeric, sammccall, ilya-biryukov, malaperle
Reviewed By: sammccall
Subscribers: malaperle, klimek, mgorny, cfe-commits
Differential Revision: https://reviews.llvm.org/D40897
llvm-svn: 320486
2017-12-12 23:42:10 +08:00
|
|
|
|
|
|
|
namespace clang {
|
|
|
|
namespace clangd {
|
|
|
|
|
[clangd] Factor out the data-swapping functionality from MemIndex/DexIndex.
Summary:
This is now handled by a wrapper class SwapIndex, so MemIndex/DexIndex can be
immutable and focus on their job.
Old and busted:
I have a MemIndex, which holds a shared_ptr<vector<Symbol*>>, which keeps the
symbol slab alive. I update by calling build(shared_ptr<vector<Symbol*>>).
New hotness: I have a SwapIndex, which holds a unique_ptr<SymbolIndex>, which
holds a MemIndex, which holds a shared_ptr<void>, which keeps backing
data alive.
I update by building a new MemIndex and calling SwapIndex::reset().
Reviewers: kbobyrev, ioeric
Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, mgrang, arphaman, kadircet, cfe-commits
Differential Revision: https://reviews.llvm.org/D51422
llvm-svn: 341318
2018-09-03 22:37:43 +08:00
|
|
|
void SwapIndex::reset(std::unique_ptr<SymbolIndex> Index) {
|
|
|
|
// Keep the old index alive, so we don't destroy it under lock (may be slow).
|
|
|
|
std::shared_ptr<SymbolIndex> Pin;
|
|
|
|
{
|
|
|
|
std::lock_guard<std::mutex> Lock(Mutex);
|
|
|
|
Pin = std::move(this->Index);
|
|
|
|
this->Index = std::move(Index);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
std::shared_ptr<SymbolIndex> SwapIndex::snapshot() const {
|
|
|
|
std::lock_guard<std::mutex> Lock(Mutex);
|
|
|
|
return Index;
|
|
|
|
}
|
|
|
|
|
[JSON] Add error reporting to fromJSON and ObjectMapper
Translating between JSON objects and C++ strutctures is common.
From experience in clangd, fromJSON/ObjectMapper work well and save a lot of
code, but aren't adopted elsewhere at least partly due to total lack of error
reporting beyond "ok"/"bad".
The recently-added error model should be rich enough for most applications.
It requires tracking the path within the root object and reporting local
errors at appropriate places.
To do this, we exploit the fact that the call graph of recursive
parse functions mirror the structure of the JSON itself.
The current path is represented as a linked list of segments, each of which is
on the stack as a parameter. Concretely, fromJSON now looks like:
bool fromJSON(const Value&, T&, Path);
Beyond the signature change, this is reasonably unobtrusive: building
the path segments is mostly handled by ObjectMapper and the vector<T> fromJSON.
However the root caller of fromJSON must now create a Root object to
store the errors, which is a little clunky.
I've added high-level parse<T>(StringRef) -> Expected<T>, but it's not
general enough to be the primary interface I think (at least, not usable in
clangd).
All existing users (mostly just clangd) are updated in this patch,
making this change backwards-compatible is a bit hairy.
Differential Revision: https://reviews.llvm.org/D88103
2020-09-24 07:14:12 +08:00
|
|
|
bool fromJSON(const llvm::json::Value &Parameters, FuzzyFindRequest &Request,
|
|
|
|
llvm::json::Path P) {
|
|
|
|
llvm::json::ObjectMapper O(Parameters, P);
|
2018-09-13 22:27:03 +08:00
|
|
|
int64_t Limit;
|
2018-09-10 19:51:05 +08:00
|
|
|
bool OK =
|
|
|
|
O && O.map("Query", Request.Query) && O.map("Scopes", Request.Scopes) &&
|
2018-11-12 20:24:08 +08:00
|
|
|
O.map("AnyScope", Request.AnyScope) && O.map("Limit", Limit) &&
|
2018-09-10 19:51:05 +08:00
|
|
|
O.map("RestrictForCodeCompletion", Request.RestrictForCodeCompletion) &&
|
2019-02-06 23:36:23 +08:00
|
|
|
O.map("ProximityPaths", Request.ProximityPaths) &&
|
|
|
|
O.map("PreferredTypes", Request.PreferredTypes);
|
2018-09-13 22:27:03 +08:00
|
|
|
if (OK && Limit <= std::numeric_limits<uint32_t>::max())
|
|
|
|
Request.Limit = Limit;
|
2018-09-10 19:51:05 +08:00
|
|
|
return OK;
|
|
|
|
}
|
|
|
|
|
2019-01-07 23:45:19 +08:00
|
|
|
llvm::json::Value toJSON(const FuzzyFindRequest &Request) {
|
|
|
|
return llvm::json::Object{
|
2018-09-10 19:51:05 +08:00
|
|
|
{"Query", Request.Query},
|
2019-01-07 23:45:19 +08:00
|
|
|
{"Scopes", llvm::json::Array{Request.Scopes}},
|
2018-11-12 20:24:08 +08:00
|
|
|
{"AnyScope", Request.AnyScope},
|
2018-09-13 22:27:03 +08:00
|
|
|
{"Limit", Request.Limit},
|
2018-09-10 19:51:05 +08:00
|
|
|
{"RestrictForCodeCompletion", Request.RestrictForCodeCompletion},
|
2019-01-07 23:45:19 +08:00
|
|
|
{"ProximityPaths", llvm::json::Array{Request.ProximityPaths}},
|
2019-02-06 23:36:23 +08:00
|
|
|
{"PreferredTypes", llvm::json::Array{Request.PreferredTypes}},
|
2018-09-10 19:51:05 +08:00
|
|
|
};
|
|
|
|
}
|
|
|
|
|
[clangd] Factor out the data-swapping functionality from MemIndex/DexIndex.
Summary:
This is now handled by a wrapper class SwapIndex, so MemIndex/DexIndex can be
immutable and focus on their job.
Old and busted:
I have a MemIndex, which holds a shared_ptr<vector<Symbol*>>, which keeps the
symbol slab alive. I update by calling build(shared_ptr<vector<Symbol*>>).
New hotness: I have a SwapIndex, which holds a unique_ptr<SymbolIndex>, which
holds a MemIndex, which holds a shared_ptr<void>, which keeps backing
data alive.
I update by building a new MemIndex and calling SwapIndex::reset().
Reviewers: kbobyrev, ioeric
Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, mgrang, arphaman, kadircet, cfe-commits
Differential Revision: https://reviews.llvm.org/D51422
llvm-svn: 341318
2018-09-03 22:37:43 +08:00
|
|
|
bool SwapIndex::fuzzyFind(const FuzzyFindRequest &R,
|
2019-01-07 23:45:19 +08:00
|
|
|
llvm::function_ref<void(const Symbol &)> CB) const {
|
[clangd] Factor out the data-swapping functionality from MemIndex/DexIndex.
Summary:
This is now handled by a wrapper class SwapIndex, so MemIndex/DexIndex can be
immutable and focus on their job.
Old and busted:
I have a MemIndex, which holds a shared_ptr<vector<Symbol*>>, which keeps the
symbol slab alive. I update by calling build(shared_ptr<vector<Symbol*>>).
New hotness: I have a SwapIndex, which holds a unique_ptr<SymbolIndex>, which
holds a MemIndex, which holds a shared_ptr<void>, which keeps backing
data alive.
I update by building a new MemIndex and calling SwapIndex::reset().
Reviewers: kbobyrev, ioeric
Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, mgrang, arphaman, kadircet, cfe-commits
Differential Revision: https://reviews.llvm.org/D51422
llvm-svn: 341318
2018-09-03 22:37:43 +08:00
|
|
|
return snapshot()->fuzzyFind(R, CB);
|
|
|
|
}
|
|
|
|
void SwapIndex::lookup(const LookupRequest &R,
|
2019-01-07 23:45:19 +08:00
|
|
|
llvm::function_ref<void(const Symbol &)> CB) const {
|
[clangd] Factor out the data-swapping functionality from MemIndex/DexIndex.
Summary:
This is now handled by a wrapper class SwapIndex, so MemIndex/DexIndex can be
immutable and focus on their job.
Old and busted:
I have a MemIndex, which holds a shared_ptr<vector<Symbol*>>, which keeps the
symbol slab alive. I update by calling build(shared_ptr<vector<Symbol*>>).
New hotness: I have a SwapIndex, which holds a unique_ptr<SymbolIndex>, which
holds a MemIndex, which holds a shared_ptr<void>, which keeps backing
data alive.
I update by building a new MemIndex and calling SwapIndex::reset().
Reviewers: kbobyrev, ioeric
Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, mgrang, arphaman, kadircet, cfe-commits
Differential Revision: https://reviews.llvm.org/D51422
llvm-svn: 341318
2018-09-03 22:37:43 +08:00
|
|
|
return snapshot()->lookup(R, CB);
|
|
|
|
}
|
2019-11-13 21:42:26 +08:00
|
|
|
bool SwapIndex::refs(const RefsRequest &R,
|
2019-01-07 23:45:19 +08:00
|
|
|
llvm::function_ref<void(const Ref &)> CB) const {
|
[clangd] SymbolOccurrences -> Refs and cleanup
Summary:
A few things that I noticed while merging the SwapIndex patch:
- SymbolOccurrences and particularly SymbolOccurrenceSlab are unwieldy names,
and these names appear *a lot*. Ref, RefSlab, etc seem clear enough
and read/format much better.
- The asymmetry between SymbolSlab and RefSlab (build() vs freeze()) is
confusing and irritating, and doesn't even save much code.
Avoiding RefSlab::Builder was my idea, but it was a bad one; add it.
- DenseMap<SymbolID, ArrayRef<Ref>> seems like a reasonable compromise for
constructing MemIndex - and means many less wasted allocations than the
current DenseMap<SymbolID, vector<Ref*>> for FileIndex, and none for
slabs.
- RefSlab::find() is not actually used for anything, so we can throw
away the DenseMap and keep the representation much more compact.
- A few naming/consistency fixes: e.g. Slabs,Refs -> Symbols,Refs.
Reviewers: ioeric
Subscribers: ilya-biryukov, MaskRay, jkorous, mgrang, arphaman, kadircet, cfe-commits
Differential Revision: https://reviews.llvm.org/D51605
llvm-svn: 341368
2018-09-04 22:39:56 +08:00
|
|
|
return snapshot()->refs(R, CB);
|
[clangd] Factor out the data-swapping functionality from MemIndex/DexIndex.
Summary:
This is now handled by a wrapper class SwapIndex, so MemIndex/DexIndex can be
immutable and focus on their job.
Old and busted:
I have a MemIndex, which holds a shared_ptr<vector<Symbol*>>, which keeps the
symbol slab alive. I update by calling build(shared_ptr<vector<Symbol*>>).
New hotness: I have a SwapIndex, which holds a unique_ptr<SymbolIndex>, which
holds a MemIndex, which holds a shared_ptr<void>, which keeps backing
data alive.
I update by building a new MemIndex and calling SwapIndex::reset().
Reviewers: kbobyrev, ioeric
Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, mgrang, arphaman, kadircet, cfe-commits
Differential Revision: https://reviews.llvm.org/D51422
llvm-svn: 341318
2018-09-03 22:37:43 +08:00
|
|
|
}
|
2019-06-15 10:26:47 +08:00
|
|
|
void SwapIndex::relations(
|
|
|
|
const RelationsRequest &R,
|
|
|
|
llvm::function_ref<void(const SymbolID &, const Symbol &)> CB) const {
|
|
|
|
return snapshot()->relations(R, CB);
|
|
|
|
}
|
|
|
|
|
[clangd] Factor out the data-swapping functionality from MemIndex/DexIndex.
Summary:
This is now handled by a wrapper class SwapIndex, so MemIndex/DexIndex can be
immutable and focus on their job.
Old and busted:
I have a MemIndex, which holds a shared_ptr<vector<Symbol*>>, which keeps the
symbol slab alive. I update by calling build(shared_ptr<vector<Symbol*>>).
New hotness: I have a SwapIndex, which holds a unique_ptr<SymbolIndex>, which
holds a MemIndex, which holds a shared_ptr<void>, which keeps backing
data alive.
I update by building a new MemIndex and calling SwapIndex::reset().
Reviewers: kbobyrev, ioeric
Subscribers: ilya-biryukov, ioeric, MaskRay, jkorous, mgrang, arphaman, kadircet, cfe-commits
Differential Revision: https://reviews.llvm.org/D51422
llvm-svn: 341318
2018-09-03 22:37:43 +08:00
|
|
|
size_t SwapIndex::estimateMemoryUsage() const {
|
|
|
|
return snapshot()->estimateMemoryUsage();
|
|
|
|
}
|
|
|
|
|
[clangd] Introduce a "Symbol" class.
Summary:
* The "Symbol" class represents a C++ symbol in the codebase, containing all the
information of a C++ symbol needed by clangd. clangd will use it in clangd's
AST/dynamic index and global/static index (code completion and code
navigation).
* The SymbolCollector (another IndexAction) will be used to recollect the
symbols when the source file is changed (for ASTIndex), or to generate
all C++ symbols for the whole project.
In the long term (when index-while-building is ready), clangd should share a
same "Symbol" structure and IndexAction with index-while-building, but
for now we want to have some stuff working in clangd.
Reviewers: ioeric, sammccall, ilya-biryukov, malaperle
Reviewed By: sammccall
Subscribers: malaperle, klimek, mgorny, cfe-commits
Differential Revision: https://reviews.llvm.org/D40897
llvm-svn: 320486
2017-12-12 23:42:10 +08:00
|
|
|
} // namespace clangd
|
|
|
|
} // namespace clang
|