maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/clang/lib/Interpreter/IncrementalParser.cpp

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

295 lines
10 KiB
C++
Raw Normal View History

[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
//===--------- IncrementalParser.cpp - Incremental Compilation -----------===//
//
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
//===----------------------------------------------------------------------===//
//
// This file implements the class which performs incremental code compilation.
//
//===----------------------------------------------------------------------===//
#include "IncrementalParser.h"
Reland "[clang-repl] Implement partial translation units and error recovery." Original commit message: [clang-repl] Implement partial translation units and error recovery. https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas: Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it. The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments. The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back. Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it. Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists. It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though. We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are. We add a different kind of translation unit `TU_Incremental` which is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols. This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance: ```./clang-repl clang-repl> int i = 12; error; In file included from <<< inputs >>>:1: input_line_0:1:13: error: C++ requires a type specifier for all declarations int i = 12; error; ^ error: Parsing failed. clang-repl> int i = 13; extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=13 clang-repl> quit ``` Differential revision: https://reviews.llvm.org/D104918
2021-07-12 21:35:20 +08:00
#include "clang/AST/DeclContextInternals.h"
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
#include "clang/CodeGen/BackendUtil.h"
#include "clang/CodeGen/CodeGenAction.h"
#include "clang/CodeGen/ModuleBuilder.h"
#include "clang/Frontend/CompilerInstance.h"
#include "clang/Frontend/FrontendAction.h"
#include "clang/FrontendTool/Utils.h"
#include "clang/Parse/Parser.h"
#include "clang/Sema/Sema.h"
#include "llvm/Option/ArgList.h"
#include "llvm/Support/CrashRecoveryContext.h"
#include "llvm/Support/Error.h"
#include "llvm/Support/Timer.h"
#include <sstream>
namespace clang {
/// A custom action enabling the incremental processing functionality.
///
/// The usual \p FrontendAction expects one call to ExecuteAction and once it
/// sees a call to \p EndSourceFile it deletes some of the important objects
/// such as \p Preprocessor and \p Sema assuming no further input will come.
///
/// \p IncrementalAction ensures it keep its underlying action's objects alive
/// as long as the \p IncrementalParser needs them.
///
class IncrementalAction : public WrapperFrontendAction {
private:
bool IsTerminating = false;
public:
IncrementalAction(CompilerInstance &CI, llvm::LLVMContext &LLVMCtx,
llvm::Error &Err)
: WrapperFrontendAction([&]() {
llvm::ErrorAsOutParameter EAO(&Err);
std::unique_ptr<FrontendAction> Act;
switch (CI.getFrontendOpts().ProgramAction) {
default:
Err = llvm::createStringError(
std::errc::state_not_recoverable,
"Driver initialization failed. "
[clang-repl] Better match the underlying architecture. In cases where -fno-integrated-as is specified we should overwrite the EmitAssembly action as well. We also should rely on the target triple from the process at least until we implement out-of-process execution. This patch should improve clang-repl on AIX. Discussion available at: https://reviews.llvm.org/D96033 Differential revision: https://reviews.llvm.org/D102688
2021-05-19 01:53:54 +08:00
"Incremental mode for action %d is not supported",
CI.getFrontendOpts().ProgramAction);
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
return Act;
case frontend::ASTDump:
LLVM_FALLTHROUGH;
case frontend::ASTPrint:
LLVM_FALLTHROUGH;
case frontend::ParseSyntaxOnly:
Act = CreateFrontendAction(CI);
break;
Reland "[clang-repl] Allow loading of plugins in clang-repl." Differential revision: https://reviews.llvm.org/D110484
2021-10-05 14:11:42 +08:00
case frontend::PluginAction:
LLVM_FALLTHROUGH;
[clang-repl] Better match the underlying architecture. In cases where -fno-integrated-as is specified we should overwrite the EmitAssembly action as well. We also should rely on the target triple from the process at least until we implement out-of-process execution. This patch should improve clang-repl on AIX. Discussion available at: https://reviews.llvm.org/D96033 Differential revision: https://reviews.llvm.org/D102688
2021-05-19 01:53:54 +08:00
case frontend::EmitAssembly:
LLVM_FALLTHROUGH;
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
case frontend::EmitObj:
LLVM_FALLTHROUGH;
case frontend::EmitLLVMOnly:
Act.reset(new EmitLLVMOnlyAction(&LLVMCtx));
break;
}
return Act;
}()) {}
FrontendAction *getWrapped() const { return WrappedAction.get(); }
Reland "[clang-repl] Implement partial translation units and error recovery." Original commit message: [clang-repl] Implement partial translation units and error recovery. https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas: Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it. The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments. The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back. Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it. Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists. It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though. We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are. We add a different kind of translation unit `TU_Incremental` which is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols. This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance: ```./clang-repl clang-repl> int i = 12; error; In file included from <<< inputs >>>:1: input_line_0:1:13: error: C++ requires a type specifier for all declarations int i = 12; error; ^ error: Parsing failed. clang-repl> int i = 13; extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=13 clang-repl> quit ``` Differential revision: https://reviews.llvm.org/D104918
2021-07-12 21:35:20 +08:00
TranslationUnitKind getTranslationUnitKind() override {
return TU_Incremental;
}
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
void ExecuteAction() override {
CompilerInstance &CI = getCompilerInstance();
assert(CI.hasPreprocessor() && "No PP!");
// FIXME: Move the truncation aspect of this into Sema, we delayed this till
// here so the source manager would be initialized.
if (hasCodeCompletionSupport() &&
!CI.getFrontendOpts().CodeCompletionAt.FileName.empty())
CI.createCodeCompletionConsumer();
// Use a code completion consumer?
CodeCompleteConsumer *CompletionConsumer = nullptr;
if (CI.hasCodeCompletionConsumer())
CompletionConsumer = &CI.getCodeCompletionConsumer();
Preprocessor &PP = CI.getPreprocessor();
PP.enableIncrementalProcessing();
PP.EnterMainSourceFile();
if (!CI.hasSema())
CI.createSema(getTranslationUnitKind(), CompletionConsumer);
}
// Do not terminate after processing the input. This allows us to keep various
// clang objects alive and to incrementally grow the current TU.
void EndSourceFile() override {
// The WrappedAction can be nullptr if we issued an error in the ctor.
if (IsTerminating && getWrapped())
WrapperFrontendAction::EndSourceFile();
}
void FinalizeAction() {
assert(!IsTerminating && "Already finalized!");
IsTerminating = true;
EndSourceFile();
}
};
IncrementalParser::IncrementalParser(std::unique_ptr<CompilerInstance> Instance,
llvm::LLVMContext &LLVMCtx,
llvm::Error &Err)
: CI(std::move(Instance)) {
llvm::ErrorAsOutParameter EAO(&Err);
Act = std::make_unique<IncrementalAction>(*CI, LLVMCtx, Err);
if (Err)
return;
CI->ExecuteAction(*Act);
Consumer = &CI->getASTConsumer();
P.reset(
new Parser(CI->getPreprocessor(), CI->getSema(), /*SkipBodies=*/false));
P->Initialize();
}
IncrementalParser::~IncrementalParser() { Act->FinalizeAction(); }
Reland "[clang-repl] Implement partial translation units and error recovery." Original commit message: [clang-repl] Implement partial translation units and error recovery. https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas: Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it. The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments. The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back. Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it. Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists. It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though. We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are. We add a different kind of translation unit `TU_Incremental` which is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols. This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance: ```./clang-repl clang-repl> int i = 12; error; In file included from <<< inputs >>>:1: input_line_0:1:13: error: C++ requires a type specifier for all declarations int i = 12; error; ^ error: Parsing failed. clang-repl> int i = 13; extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=13 clang-repl> quit ``` Differential revision: https://reviews.llvm.org/D104918
2021-07-12 21:35:20 +08:00
llvm::Expected<PartialTranslationUnit &>
IncrementalParser::ParseOrWrapTopLevelDecl() {
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
// Recover resources if we crash before exiting this method.
Sema &S = CI->getSema();
llvm::CrashRecoveryContextCleanupRegistrar<Sema> CleanupSema(&S);
Sema::GlobalEagerInstantiationScope GlobalInstantiations(S, /*Enabled=*/true);
Sema::LocalEagerInstantiationScope LocalInstantiations(S);
Reland "[clang-repl] Implement partial translation units and error recovery." Original commit message: [clang-repl] Implement partial translation units and error recovery. https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas: Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it. The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments. The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back. Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it. Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists. It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though. We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are. We add a different kind of translation unit `TU_Incremental` which is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols. This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance: ```./clang-repl clang-repl> int i = 12; error; In file included from <<< inputs >>>:1: input_line_0:1:13: error: C++ requires a type specifier for all declarations int i = 12; error; ^ error: Parsing failed. clang-repl> int i = 13; extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=13 clang-repl> quit ``` Differential revision: https://reviews.llvm.org/D104918
2021-07-12 21:35:20 +08:00
PTUs.emplace_back(PartialTranslationUnit());
PartialTranslationUnit &LastPTU = PTUs.back();
// Add a new PTU.
ASTContext &C = S.getASTContext();
C.addTranslationUnitDecl();
LastPTU.TUPart = C.getTranslationUnitDecl();
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
// Skip previous eof due to last incremental input.
Reland "[clang-repl] Implement partial translation units and error recovery." Original commit message: [clang-repl] Implement partial translation units and error recovery. https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas: Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it. The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments. The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back. Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it. Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists. It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though. We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are. We add a different kind of translation unit `TU_Incremental` which is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols. This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance: ```./clang-repl clang-repl> int i = 12; error; In file included from <<< inputs >>>:1: input_line_0:1:13: error: C++ requires a type specifier for all declarations int i = 12; error; ^ error: Parsing failed. clang-repl> int i = 13; extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=13 clang-repl> quit ``` Differential revision: https://reviews.llvm.org/D104918
2021-07-12 21:35:20 +08:00
if (P->getCurToken().is(tok::eof)) {
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
P->ConsumeToken();
Reland "[clang-repl] Implement partial translation units and error recovery." Original commit message: [clang-repl] Implement partial translation units and error recovery. https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas: Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it. The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments. The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back. Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it. Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists. It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though. We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are. We add a different kind of translation unit `TU_Incremental` which is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols. This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance: ```./clang-repl clang-repl> int i = 12; error; In file included from <<< inputs >>>:1: input_line_0:1:13: error: C++ requires a type specifier for all declarations int i = 12; error; ^ error: Parsing failed. clang-repl> int i = 13; extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=13 clang-repl> quit ``` Differential revision: https://reviews.llvm.org/D104918
2021-07-12 21:35:20 +08:00
// FIXME: Clang does not call ExitScope on finalizing the regular TU, we
// might want to do that around HandleEndOfTranslationUnit.
P->ExitScope();
S.CurContext = nullptr;
// Start a new PTU.
P->EnterScope(Scope::DeclScope);
S.ActOnTranslationUnitScope(P->getCurScope());
}
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
Parser::DeclGroupPtrTy ADecl;
for (bool AtEOF = P->ParseFirstTopLevelDecl(ADecl); !AtEOF;
AtEOF = P->ParseTopLevelDecl(ADecl)) {
// If we got a null return and something *was* parsed, ignore it. This
// is due to a top-level semicolon, an action override, or a parse error
// skipping something.
if (ADecl && !Consumer->HandleTopLevelDecl(ADecl.get()))
return llvm::make_error<llvm::StringError>("Parsing failed. "
"The consumer rejected a decl",
std::error_code());
Reland "[clang-repl] Implement partial translation units and error recovery." Original commit message: [clang-repl] Implement partial translation units and error recovery. https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas: Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it. The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments. The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back. Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it. Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists. It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though. We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are. We add a different kind of translation unit `TU_Incremental` which is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols. This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance: ```./clang-repl clang-repl> int i = 12; error; In file included from <<< inputs >>>:1: input_line_0:1:13: error: C++ requires a type specifier for all declarations int i = 12; error; ^ error: Parsing failed. clang-repl> int i = 13; extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=13 clang-repl> quit ``` Differential revision: https://reviews.llvm.org/D104918
2021-07-12 21:35:20 +08:00
}
DiagnosticsEngine &Diags = getCI()->getDiagnostics();
if (Diags.hasErrorOccurred()) {
TranslationUnitDecl *MostRecentTU = C.getTranslationUnitDecl();
TranslationUnitDecl *PreviousTU = MostRecentTU->getPreviousDecl();
assert(PreviousTU && "Must have a TU from the ASTContext initialization!");
TranslationUnitDecl *FirstTU = MostRecentTU->getFirstDecl();
assert(FirstTU);
FirstTU->RedeclLink.setLatest(PreviousTU);
C.TUDecl = PreviousTU;
S.TUScope->setEntity(PreviousTU);
// Clean up the lookup table
if (StoredDeclsMap *Map = PreviousTU->getLookupPtr()) {
for (auto I = Map->begin(); I != Map->end(); ++I) {
StoredDeclsList &List = I->second;
DeclContextLookupResult R = List.getLookupResult();
for (NamedDecl *D : R)
if (D->getTranslationUnitDecl() == MostRecentTU)
List.remove(D);
if (List.isNull())
Map->erase(I);
}
}
// FIXME: Do not reset the pragma handlers.
Diags.Reset();
return llvm::make_error<llvm::StringError>("Parsing failed.",
std::error_code());
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
}
// Process any TopLevelDecls generated by #pragma weak.
for (Decl *D : S.WeakTopLevelDecls()) {
DeclGroupRef DGR(D);
Consumer->HandleTopLevelDecl(DGR);
}
LocalInstantiations.perform();
GlobalInstantiations.perform();
Reland "[clang-repl] Implement partial translation units and error recovery." Original commit message: [clang-repl] Implement partial translation units and error recovery. https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas: Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it. The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments. The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back. Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it. Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists. It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though. We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are. We add a different kind of translation unit `TU_Incremental` which is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols. This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance: ```./clang-repl clang-repl> int i = 12; error; In file included from <<< inputs >>>:1: input_line_0:1:13: error: C++ requires a type specifier for all declarations int i = 12; error; ^ error: Parsing failed. clang-repl> int i = 13; extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=13 clang-repl> quit ``` Differential revision: https://reviews.llvm.org/D104918
2021-07-12 21:35:20 +08:00
Consumer->HandleTranslationUnit(C);
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
Reland "[clang-repl] Implement partial translation units and error recovery." Original commit message: [clang-repl] Implement partial translation units and error recovery. https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas: Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it. The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments. The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back. Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it. Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists. It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though. We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are. We add a different kind of translation unit `TU_Incremental` which is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols. This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance: ```./clang-repl clang-repl> int i = 12; error; In file included from <<< inputs >>>:1: input_line_0:1:13: error: C++ requires a type specifier for all declarations int i = 12; error; ^ error: Parsing failed. clang-repl> int i = 13; extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=13 clang-repl> quit ``` Differential revision: https://reviews.llvm.org/D104918
2021-07-12 21:35:20 +08:00
return LastPTU;
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
}
static CodeGenerator *getCodeGen(FrontendAction *Act) {
IncrementalAction *IncrAct = static_cast<IncrementalAction *>(Act);
FrontendAction *WrappedAct = IncrAct->getWrapped();
if (!WrappedAct->hasIRSupport())
return nullptr;
return static_cast<CodeGenAction *>(WrappedAct)->getCodeGenerator();
}
Reland "[clang-repl] Implement partial translation units and error recovery." Original commit message: [clang-repl] Implement partial translation units and error recovery. https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas: Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it. The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments. The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back. Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it. Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists. It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though. We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are. We add a different kind of translation unit `TU_Incremental` which is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols. This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance: ```./clang-repl clang-repl> int i = 12; error; In file included from <<< inputs >>>:1: input_line_0:1:13: error: C++ requires a type specifier for all declarations int i = 12; error; ^ error: Parsing failed. clang-repl> int i = 13; extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=13 clang-repl> quit ``` Differential revision: https://reviews.llvm.org/D104918
2021-07-12 21:35:20 +08:00
llvm::Expected<PartialTranslationUnit &>
IncrementalParser::Parse(llvm::StringRef input) {
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
Preprocessor &PP = CI->getPreprocessor();
assert(PP.isIncrementalProcessingEnabled() && "Not in incremental mode!?");
std::ostringstream SourceName;
SourceName << "input_line_" << InputCount++;
// Create an uninitialized memory buffer, copy code in and append "\n"
size_t InputSize = input.size(); // don't include trailing 0
// MemBuffer size should *not* include terminating zero
std::unique_ptr<llvm::MemoryBuffer> MB(
llvm::WritableMemoryBuffer::getNewUninitMemBuffer(InputSize + 1,
SourceName.str()));
char *MBStart = const_cast<char *>(MB->getBufferStart());
memcpy(MBStart, input.data(), InputSize);
MBStart[InputSize] = '\n';
SourceManager &SM = CI->getSourceManager();
// FIXME: Create SourceLocation, which will allow clang to order the overload
// candidates for example
SourceLocation NewLoc = SM.getLocForStartOfFile(SM.getMainFileID());
// Create FileID for the current buffer.
FileID FID = SM.createFileID(std::move(MB), SrcMgr::C_User, /*LoadedID=*/0,
/*LoadedOffset=*/0, NewLoc);
// NewLoc only used for diags.
if (PP.EnterSourceFile(FID, /*DirLookup=*/0, NewLoc))
return llvm::make_error<llvm::StringError>("Parsing failed. "
"Cannot enter source file.",
std::error_code());
Reland "[clang-repl] Implement partial translation units and error recovery." Original commit message: [clang-repl] Implement partial translation units and error recovery. https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas: Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it. The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments. The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back. Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it. Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists. It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though. We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are. We add a different kind of translation unit `TU_Incremental` which is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols. This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance: ```./clang-repl clang-repl> int i = 12; error; In file included from <<< inputs >>>:1: input_line_0:1:13: error: C++ requires a type specifier for all declarations int i = 12; error; ^ error: Parsing failed. clang-repl> int i = 13; extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=13 clang-repl> quit ``` Differential revision: https://reviews.llvm.org/D104918
2021-07-12 21:35:20 +08:00
auto PTU = ParseOrWrapTopLevelDecl();
if (!PTU)
return PTU.takeError();
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
if (PP.getLangOpts().DelayedTemplateParsing) {
// Microsoft-specific:
// Late parsed templates can leave unswallowed "macro"-like tokens.
// They will seriously confuse the Parser when entering the next
// source file. So lex until we are EOF.
Token Tok;
do {
PP.Lex(Tok);
} while (Tok.isNot(tok::eof));
}
Token AssertTok;
PP.Lex(AssertTok);
assert(AssertTok.is(tok::eof) &&
"Lexer must be EOF when starting incremental parse!");
if (CodeGenerator *CG = getCodeGen(Act.get())) {
std::unique_ptr<llvm::Module> M(CG->ReleaseModule());
Reland "[clang-repl] Implement partial translation units and error recovery." Original commit message: [clang-repl] Implement partial translation units and error recovery. https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas: Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it. The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments. The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back. Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it. Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists. It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though. We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are. We add a different kind of translation unit `TU_Incremental` which is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols. This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance: ```./clang-repl clang-repl> int i = 12; error; In file included from <<< inputs >>>:1: input_line_0:1:13: error: C++ requires a type specifier for all declarations int i = 12; error; ^ error: Parsing failed. clang-repl> int i = 13; extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=13 clang-repl> quit ``` Differential revision: https://reviews.llvm.org/D104918
2021-07-12 21:35:20 +08:00
CG->StartModule("incr_module_" + std::to_string(PTUs.size()),
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
M->getContext());
Reland "[clang-repl] Implement partial translation units and error recovery." Original commit message: [clang-repl] Implement partial translation units and error recovery. https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas: Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it. The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments. The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back. Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it. Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists. It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though. We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are. We add a different kind of translation unit `TU_Incremental` which is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols. This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance: ```./clang-repl clang-repl> int i = 12; error; In file included from <<< inputs >>>:1: input_line_0:1:13: error: C++ requires a type specifier for all declarations int i = 12; error; ^ error: Parsing failed. clang-repl> int i = 13; extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=13 clang-repl> quit ``` Differential revision: https://reviews.llvm.org/D104918
2021-07-12 21:35:20 +08:00
PTU->TheModule = std::move(M);
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
}
Reland "[clang-repl] Implement partial translation units and error recovery." Original commit message: [clang-repl] Implement partial translation units and error recovery. https://reviews.llvm.org/D96033 contained a discussion regarding efficient modeling of error recovery. @rjmccall has outlined the key ideas: Conceptually, we can split the translation unit into a sequence of partial translation units (PTUs). Every declaration will be associated with a unique PTU that owns it. The first key insight here is that the owning PTU isn't always the "active" (most recent) PTU, and it isn't always the PTU that the declaration "comes from". A new declaration (that isn't a redeclaration or specialization of anything) does belong to the active PTU. A template specialization, however, belongs to the most recent PTU of all the declarations in its signature - mostly that means that it can be pulled into a more recent PTU by its template arguments. The second key insight is that processing a PTU might extend an earlier PTU. Rolling back the later PTU shouldn't throw that extension away. For example, if the second PTU defines a template, and the third PTU requires that template to be instantiated at float, that template specialization is still part of the second PTU. Similarly, if the fifth PTU uses an inline function belonging to the fourth, that definition still belongs to the fourth. When we go to emit code in a new PTU, we map each declaration we have to emit back to its owning PTU and emit it in a new module for just the extensions to that PTU. We keep track of all the modules we've emitted for a PTU so that we can unload them all if we decide to roll it back. Most declarations/definitions will only refer to entities from the same or earlier PTUs. However, it is possible (primarily by defining a previously-declared entity, but also through templates or ADL) for an entity that belongs to one PTU to refer to something from a later PTU. We will have to keep track of this and prevent unwinding to later PTU when we recognize it. Fortunately, this should be very rare; and crucially, we don't have to do the bookkeeping for this if we've only got one PTU, e.g. in normal compilation. Otherwise, PTUs after the first just need to record enough metadata to be able to revert any changes they've made to declarations belonging to earlier PTUs, e.g. to redeclaration chains or template specialization lists. It should even eventually be possible for PTUs to provide their own slab allocators which can be thrown away as part of rolling back the PTU. We can maintain a notion of the active allocator and allocate things like Stmt/Expr nodes in it, temporarily changing it to the appropriate PTU whenever we go to do something like instantiate a function template. More care will be required when allocating declarations and types, though. We would want the PTU to be efficiently recoverable from a Decl; I'm not sure how best to do that. An easy option that would cover most declarations would be to make multiple TranslationUnitDecls and parent the declarations appropriately, but I don't think that's good enough for things like member function templates, since an instantiation of that would still be parented by its original class. Maybe we can work this into the DC chain somehow, like how lexical DCs are. We add a different kind of translation unit `TU_Incremental` which is a complete translation unit that we might nonetheless incrementally extend later. Because it is complete (and we might want to generate code for it), we do perform template instantiation, but because it might be extended later, we don't warn if it declares or uses undefined internal-linkage symbols. This patch teaches clang-repl how to recover from errors by disconnecting the most recent PTU and update the primary PTU lookup tables. For instance: ```./clang-repl clang-repl> int i = 12; error; In file included from <<< inputs >>>:1: input_line_0:1:13: error: C++ requires a type specifier for all declarations int i = 12; error; ^ error: Parsing failed. clang-repl> int i = 13; extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=13 clang-repl> quit ``` Differential revision: https://reviews.llvm.org/D104918
2021-07-12 21:35:20 +08:00
return PTU;
[clang-repl] Recommit "Land initial infrastructure for incremental parsing" Original commit message: In http://lists.llvm.org/pipermail/llvm-dev/2020-July/143257.html we have mentioned our plans to make some of the incremental compilation facilities available in llvm mainline. This patch proposes a minimal version of a repl, clang-repl, which enables interpreter-like interaction for C++. For instance: ./bin/clang-repl clang-repl> int i = 42; clang-repl> extern "C" int printf(const char*,...); clang-repl> auto r1 = printf("i=%d\n", i); i=42 clang-repl> quit The patch allows very limited functionality, for example, it crashes on invalid C++. The design of the proposed patch follows closely the design of cling. The idea is to gather feedback and gradually evolve both clang-repl and cling to what the community agrees upon. The IncrementalParser class is responsible for driving the clang parser and codegen and allows the compiler infrastructure to process more than one input. Every input adds to the “ever-growing” translation unit. That model is enabled by an IncrementalAction which prevents teardown when HandleTranslationUnit. The IncrementalExecutor class hides some of the underlying implementation details of the concrete JIT infrastructure. It exposes the minimal set of functionality required by our incremental compiler/interpreter. The Transaction class keeps track of the AST and the LLVM IR for each incremental input. That tracking information will be later used to implement error recovery. The Interpreter class orchestrates the IncrementalParser and the IncrementalExecutor to model interpreter-like behavior. It provides the public API which can be used (in future) when using the interpreter library. Differential revision: https://reviews.llvm.org/D96033
2021-05-13 13:41:44 +08:00
}
} // end namespace clang