forked from OSchip/llvm-project
164580 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Roman Tereshin
|
fedae33efa |
[GlobalISel][InstructionSelect] MatchTable second level grouping, perf patch 5
This patch continues a series of patches started by r332907 (reapplied as r332917) In this commit we start grouping rules with common first condition on the second level of the table. This is expected to decrease time GlobalISel spends in its InstructionSelect pass by roughly 13% for an -O0 build as measured on sqlite3-amalgamation (http://sqlite.org/download.html) targeting AArch64. Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar Reviewed By: qcolombet Subscribers: rovka, llvm-commits, kristof.beyls Differential Revision: https://reviews.llvm.org/D44700 llvm-svn: 333053 |
|
|
Heejin Ahn
|
1e4d35044f |
[WebAssembly] Add functions for EHScopes
Summary: There are functions using the term 'funclet' to refer to both 1. an EH scopes, the structure of BBs that starts with catchpad/cleanuppad and ends with catchret/cleanupret, and 2. a small function that gets outlined in AsmPrinter, which is the original meaning of 'funclet'. So far the two have been the same thing; EH scopes are always outlined in AsmPrinter as funclets at the end of the compilation pipeline. But now wasm also uses scope-based EH but does not outline those, so we now need to correctly distinguish those two use cases in functions. This patch splits `MachineBasicBlock::isFuncletEntry` into `isFuncletEntry` and `isEHScopeEntry`, and `MachineFunction::hasFunclets` into `hasFunclets` and `hasEHScopes`, in order to distinguish the two different use cases. And this also changes some uses of the term 'funclet' to 'scope' in `getFuncletMembership` and change the function name to `getEHScopeMembership` because this function is not about outlined funclets but about EH scope memberships. This change is in the same vein as D45559. Reviewers: majnemer, dschuff Subscribers: sbc100, jgravelle-google, sunfish, llvm-commits Differential Revision: https://reviews.llvm.org/D47005 llvm-svn: 333045 |
|
|
Sanjay Patel
|
4b96935bd7 |
[InstCombine] use nsw negation for abs libcalls
Also, produce the canonical IR abs (s<0) to be more efficient. This is the libcall equivalent of the clang builtin change from: rL333038 Pasting from that commit message: The stdlib functions are defined in section 7.20.6.1 of the C standard with: "If the result cannot be represented, the behavior is undefined." That lets us mark the negation with 'nsw' because "sub i32 0, INT_MIN" would be UB/poison. llvm-svn: 333042 |
|
|
Sanjay Patel
|
3ef8f858da |
[InstCombine] move misplaced test file and regenerate checks; NFC
llvm-svn: 333039 |
|
|
Eli Friedman
|
35756e8984 |
[docs] Clarify usage of "vector" in Programmer's Manual.
The explanation is specifically referring to std::vector; this might not be clear from the context. llvm-svn: 333036 |
|
|
Max Moroz
|
5b1083d76f |
[Coverage] Update CSS to make HTML reports copy-paste friendly.
Summary: This minor change allows to copy snippets from HTML reports so they will be pasted in the following format: %LineNumber%\t%HitCount%\t%CodeLine% rather then being split onto multiple lines. To see this in action, try copy pasting from https://chromium-coverage.appspot.com/reports/560344/linux/chromium/src/third_party/zlib/compress.c.html Requested in https://bugs.chromium.org/p/chromium/issues/detail?id=845571 Reviewers: vsk Reviewed By: vsk Subscribers: llvm-commits, morehouse, kcc Differential Revision: https://reviews.llvm.org/D47231 llvm-svn: 333034 |
|
|
David Bolvansky
|
88e262bcdd |
Delete empty test file
Differential Revision: https://reviews.llvm.org/D47230 llvm-svn: 333031 |
|
|
Matt Davis
|
bd12532300 |
[llvm-mca] Move DispatchStage::cycleEvent to preExecute. NFC.
Summary:
This is an intermediate change, it moves the non-notification logic from
Backend::notifyCycleBegin to runCycle().
Once the scheduler becomes part of the Execution stage
the explicit call to Scheduler::cycleEvent will disappear.
The logic for Dispatch::cycleEvent() can be in
the preExecute phase, which this patch addresses.
Reviewers: andreadb, RKSimon, courbet
Reviewed By: andreadb
Subscribers: tschuett, gbedwell, llvm-commits
Differential Revision: https://reviews.llvm.org/D47213
llvm-svn: 333029
|
|
|
Lang Hames
|
a65d39e479 |
[ORC] Add some comments to Layer.h.
llvm-svn: 333028 |
|
|
Matt Arsenault
|
f0c5c6b3f3 |
AMDGPU: Fix missing test coverage for some 16-bit and packed ops
llvm-svn: 333024 |
|
|
David Bolvansky
|
1f343fa0e0 |
[InstCombine] Remove calloc transformations
Summary: Previous patch does not care if a value is changed between calloc and strlen. This needs to be removed from InstCombine and maybe moved to DSE later after some rework. Reviewers: efriedma Reviewed By: efriedma Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D47218 llvm-svn: 333022 |
|
|
Matt Arsenault
|
606bc315d6 |
AMDGPU: Fix v2f16 fneg/fabs pattern
The integer operation convertion for some reason only happens if the source is a bitcast from an integer, which happens to always be the situation when the result is loaded. Add an additional pattern for when the source operation is really an FP operation. llvm-svn: 333019 |
|
|
Eli Friedman
|
785acce51d |
Delete unused variable from r333015.
(The assertion suppressed the unused variable warning on Release+Asserts builds, so I didn't notice.) llvm-svn: 333018 |
|
|
Roman Tereshin
|
0ee082f3b9 |
[GlobalISel][InstructionSelect] Switching MatchTable over opcodes, perf patch 4
This patch continues a series of patches started by r332907 (reapplied as r332917) In this commit we introduce a new matching opcode GIM_SwitchOpcode that implements a jump table over opcodes and start emitting them for root instructions. This is expected to decrease time GlobalISel spends in its InstructionSelect pass by roughly 20% for an -O0 build as measured on sqlite3-amalgamation (http://sqlite.org/download.html) targeting AArch64. To some degree, we assume here that the opcodes form a dense set, which is true at the moment for all upstream targets given the limitations of our rule importing mechanism. It might not be true for out of tree targets, specifically due to pseudo's. If so, we might noticeably increase the size of the MatchTable with this patch due to padding zeros. This will be addressed later. Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar Reviewed By: qcolombet Subscribers: rovka, llvm-commits, kristof.beyls Differential Revision: https://reviews.llvm.org/D44700 llvm-svn: 333017 |
|
|
Tom Stellard
|
b12f4dec08 |
AMDGPU: Move AMDGPUTargetLowering::isFPExtFoldable() into SITargetLowering
Summary: This is always false for R600. Reviewers: arsenm, nhaehnle Reviewed By: arsenm Subscribers: kzhuravl, wdng, yaxunl, dstuttard, tpr, t-tye, llvm-commits Differential Revision: https://reviews.llvm.org/D47180 llvm-svn: 333016 |
|
|
Eli Friedman
|
042dc9e092 |
[MachineOutliner] Add "thunk" outlining for AArch64.
When we're outlining a sequence that ends in a call, we can save up to three instructions in the outlined function by turning the call into a tail-call. I refer to this as thunk outlining because the resulting outlined function looks like a thunk; suggestions welcome for a better name. In addition to making the outlined function shorter, thunk outlining allows outlining calls which would otherwise be illegal to outline: we don't need to save/restore LR, so we don't need to prove anything about the stack access patterns of the callee. To make this work effectively, I also added MachineOutlinerInstrType::LegalTerminator to the generic MachineOutliner code; this allows treating an arbitrary instruction as a terminator in the suffix tree. Differential Revision: https://reviews.llvm.org/D47173 llvm-svn: 333015 |
|
|
Sanjay Patel
|
9781679f0f |
[InstCombine] move/add tests for sub with bool op; NFC
llvm-svn: 333012 |
|
|
Krzysztof Parzyszek
|
840b02bccf |
[Hexagon] Add patterns for accumulating HVX compares
llvm-svn: 333009 |
|
|
Alexander Shaposhnikov
|
6e7814c484 |
[llvm-objcopy] Fix the behavior of --strip-* and --keep-symbol
If one runs llvm-objcopy --strip-all --keep-symbol foo and the symbol table indeed contains the symbol "foo" then it should not be removed. Test plan: make check-all Differential revision: https://reviews.llvm.org/D47052 llvm-svn: 333008 |
|
|
Florian Hahn
|
a6e63f176c |
[NewGVN] Fix handling of assumes
This patch fixes two bugs:
* test1: Previously assume(a >= 5) concluded that a == 5. That's only
valid for assume(a == 5)...
* test2: If operands were swapped, additional users were added to the
wrong cmp operand. This resulted in an "unsettled iteration"
assertion failure.
Patch by Nikita Popov
Differential Revision: https://reviews.llvm.org/D46974
llvm-svn: 333007
|
|
|
Jonas Devlieghere
|
63eca15e95 |
[DebugInfo] Invert DIE order for range errors.
When printing an error for an invalid address range in a DIE, we used to print the child above the parent, which is counter intuitive. This patch reverses the order and indents the child to mimic the way we print the debug info section. llvm-svn: 333006 |
|
|
Jonas Devlieghere
|
7e0b023302 |
[DebugInfo] Fix location list check in the verifier
We weren't properly verifying location lists because we tried obtaining the offset as a constant. llvm-svn: 333005 |
|
|
Paul Robinson
|
543c0e1d50 |
[DWARFv5] Put the DWO ID in its place.
In DWARF v5, the DWO ID is in the (split/skeleton) CU header, not an attribute on the CU DIE. This changes the size of those headers, so use the parsed size whenever we have one, for simplicitly. Differential Revision: https://reviews.llvm.org/D47158 llvm-svn: 333004 |
|
|
Roman Tereshin
|
7701360308 |
[GlobalISel][InstructionSelect] Sorting MatchTable's first level by opcodes and num operands, perf patch 3
This patch continues a series of patches started by r332907 (reapplied as r332917) In this commit we start sorting the rules by the opcode first, and if the same, by the number of operands of the root instructions. This allows better grouping and safe as patterns with different opcodes are mutually exclusive. This is expected to decrease time GlobalISel spends in its InstructionSelect pass by roughly 18% for an -O0 build as measured on sqlite3-amalgamation (http://sqlite.org/download.html) targeting AArch64. I'm also removing RuleMatcher::getFirstConditionAsRootType() function here and moving it to a later patch within the series as it's not used yet and was causing a warning on sanitizer-ppc64le-linux bot. Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar Reviewed By: qcolombet Subscribers: rovka, llvm-commits, kristof.beyls Differential Revision: https://reviews.llvm.org/D44700 llvm-svn: 333001 |
|
|
Roman Tereshin
|
dad5d6f347 |
Reverting 332999 to get it a proper commit message
llvm-svn: 333000 |
|
|
Roman Tereshin
|
a7b5d45f9b |
diff --git a/utils/TableGen/GlobalISelEmitter.cpp b/utils/TableGen/GlobalISelEmitter.cpp
index cdc9df7bf6b..be08165a200 100644
--- a/utils/TableGen/GlobalISelEmitter.cpp
+++ b/utils/TableGen/GlobalISelEmitter.cpp
@@ -1,4531 +1,4539 @@
//===- GlobalISelEmitter.cpp - Generate an instruction selector -----------===//
//
// The LLVM Compiler Infrastructure
//
// This file is distributed under the University of Illinois Open Source
// License. See LICENSE.TXT for details.
//
//===----------------------------------------------------------------------===//
//
/// \file
/// This tablegen backend emits code for use by the GlobalISel instruction
/// selector. See include/llvm/CodeGen/TargetGlobalISel.td.
///
/// This file analyzes the patterns recognized by the SelectionDAGISel tablegen
/// backend, filters out the ones that are unsupported, maps
/// SelectionDAG-specific constructs to their GlobalISel counterpart
/// (when applicable: MVT to LLT; SDNode to generic Instruction).
///
/// Not all patterns are supported: pass the tablegen invocation
/// "-warn-on-skipped-patterns" to emit a warning when a pattern is skipped,
/// as well as why.
///
/// The generated file defines a single method:
/// bool <Target>InstructionSelector::selectImpl(MachineInstr &I) const;
/// intended to be used in InstructionSelector::select as the first-step
/// selector for the patterns that don't require complex C++.
///
/// FIXME: We'll probably want to eventually define a base
/// "TargetGenInstructionSelector" class.
///
//===----------------------------------------------------------------------===//
#include "CodeGenDAGPatterns.h"
#include "SubtargetFeatureInfo.h"
#include "llvm/ADT/Optional.h"
#include "llvm/ADT/SmallSet.h"
#include "llvm/ADT/Statistic.h"
#include "llvm/Support/CodeGenCoverage.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Error.h"
#include "llvm/Support/LowLevelTypeImpl.h"
#include "llvm/Support/MachineValueType.h"
#include "llvm/Support/ScopedPrinter.h"
#include "llvm/TableGen/Error.h"
#include "llvm/TableGen/Record.h"
#include "llvm/TableGen/TableGenBackend.h"
#include <numeric>
#include <string>
using namespace llvm;
#define DEBUG_TYPE "gisel-emitter"
STATISTIC(NumPatternTotal, "Total number of patterns");
STATISTIC(NumPatternImported, "Number of patterns imported from SelectionDAG");
STATISTIC(NumPatternImportsSkipped, "Number of SelectionDAG imports skipped");
STATISTIC(NumPatternsTested, "Number of patterns executed according to coverage information");
STATISTIC(NumPatternEmitted, "Number of patterns emitted");
cl::OptionCategory GlobalISelEmitterCat("Options for -gen-global-isel");
static cl::opt<bool> WarnOnSkippedPatterns(
"warn-on-skipped-patterns",
cl::desc("Explain why a pattern was skipped for inclusion "
"in the GlobalISel selector"),
cl::init(false), cl::cat(GlobalISelEmitterCat));
static cl::opt<bool> GenerateCoverage(
"instrument-gisel-coverage",
cl::desc("Generate coverage instrumentation for GlobalISel"),
cl::init(false), cl::cat(GlobalISelEmitterCat));
static cl::opt<std::string> UseCoverageFile(
"gisel-coverage-file", cl::init(""),
cl::desc("Specify file to retrieve coverage information from"),
cl::cat(GlobalISelEmitterCat));
static cl::opt<bool> OptimizeMatchTable(
"optimize-match-table",
cl::desc("Generate an optimized version of the match table"),
cl::init(true), cl::cat(GlobalISelEmitterCat));
namespace {
//===- Helper functions ---------------------------------------------------===//
/// Get the name of the enum value used to number the predicate function.
std::string getEnumNameForPredicate(const TreePredicateFn &Predicate) {
return "GIPFP_" + Predicate.getImmTypeIdentifier().str() + "_" +
Predicate.getFnName();
}
/// Get the opcode used to check this predicate.
std::string getMatchOpcodeForPredicate(const TreePredicateFn &Predicate) {
return "GIM_Check" + Predicate.getImmTypeIdentifier().str() + "ImmPredicate";
}
/// This class stands in for LLT wherever we want to tablegen-erate an
/// equivalent at compiler run-time.
class LLTCodeGen {
private:
LLT Ty;
public:
LLTCodeGen() = default;
LLTCodeGen(const LLT &Ty) : Ty(Ty) {}
std::string getCxxEnumValue() const {
std::string Str;
raw_string_ostream OS(Str);
emitCxxEnumValue(OS);
return OS.str();
}
void emitCxxEnumValue(raw_ostream &OS) const {
if (Ty.isScalar()) {
OS << "GILLT_s" << Ty.getSizeInBits();
return;
}
if (Ty.isVector()) {
OS << "GILLT_v" << Ty.getNumElements() << "s" << Ty.getScalarSizeInBits();
return;
}
if (Ty.isPointer()) {
OS << "GILLT_p" << Ty.getAddressSpace();
if (Ty.getSizeInBits() > 0)
OS << "s" << Ty.getSizeInBits();
return;
}
llvm_unreachable("Unhandled LLT");
}
void emitCxxConstructorCall(raw_ostream &OS) const {
if (Ty.isScalar()) {
OS << "LLT::scalar(" << Ty.getSizeInBits() << ")";
return;
}
if (Ty.isVector()) {
OS << "LLT::vector(" << Ty.getNumElements() << ", "
<< Ty.getScalarSizeInBits() << ")";
return;
}
if (Ty.isPointer() && Ty.getSizeInBits() > 0) {
OS << "LLT::pointer(" << Ty.getAddressSpace() << ", "
<< Ty.getSizeInBits() << ")";
return;
}
llvm_unreachable("Unhandled LLT");
}
const LLT &get() const { return Ty; }
/// This ordering is used for std::unique() and llvm::sort(). There's no
/// particular logic behind the order but either A < B or B < A must be
/// true if A != B.
bool operator<(const LLTCodeGen &Other) const {
if (Ty.isValid() != Other.Ty.isValid())
return Ty.isValid() < Other.Ty.isValid();
if (!Ty.isValid())
return false;
if (Ty.isVector() != Other.Ty.isVector())
return Ty.isVector() < Other.Ty.isVector();
if (Ty.isScalar() != Other.Ty.isScalar())
return Ty.isScalar() < Other.Ty.isScalar();
if (Ty.isPointer() != Other.Ty.isPointer())
return Ty.isPointer() < Other.Ty.isPointer();
if (Ty.isPointer() && Ty.getAddressSpace() != Other.Ty.getAddressSpace())
return Ty.getAddressSpace() < Other.Ty.getAddressSpace();
if (Ty.isVector() && Ty.getNumElements() != Other.Ty.getNumElements())
return Ty.getNumElements() < Other.Ty.getNumElements();
return Ty.getSizeInBits() < Other.Ty.getSizeInBits();
}
bool operator==(const LLTCodeGen &B) const { return Ty == B.Ty; }
};
// Track all types that are used so we can emit the corresponding enum.
std::set<LLTCodeGen> KnownTypes;
class InstructionMatcher;
/// Convert an MVT to an equivalent LLT if possible, or the invalid LLT() for
/// MVTs that don't map cleanly to an LLT (e.g., iPTR, *any, ...).
static Optional<LLTCodeGen> MVTToLLT(MVT::SimpleValueType SVT) {
MVT VT(SVT);
if (VT.isVector() && VT.getVectorNumElements() != 1)
return LLTCodeGen(
LLT::vector(VT.getVectorNumElements(), VT.getScalarSizeInBits()));
if (VT.isInteger() || VT.isFloatingPoint())
return LLTCodeGen(LLT::scalar(VT.getSizeInBits()));
return None;
}
static std::string explainPredicates(const TreePatternNode *N) {
std::string Explanation = "";
StringRef Separator = "";
for (const auto &P : N->getPredicateFns()) {
Explanation +=
(Separator + P.getOrigPatFragRecord()->getRecord()->getName()).str();
Separator = ", ";
if (P.isAlwaysTrue())
Explanation += " always-true";
if (P.isImmediatePattern())
Explanation += " immediate";
if (P.isUnindexed())
Explanation += " unindexed";
if (P.isNonExtLoad())
Explanation += " non-extload";
if (P.isAnyExtLoad())
Explanation += " extload";
if (P.isSignExtLoad())
Explanation += " sextload";
if (P.isZeroExtLoad())
Explanation += " zextload";
if (P.isNonTruncStore())
Explanation += " non-truncstore";
if (P.isTruncStore())
Explanation += " truncstore";
if (Record *VT = P.getMemoryVT())
Explanation += (" MemVT=" + VT->getName()).str();
if (Record *VT = P.getScalarMemoryVT())
Explanation += (" ScalarVT(MemVT)=" + VT->getName()).str();
if (P.isAtomicOrderingMonotonic())
Explanation += " monotonic";
if (P.isAtomicOrderingAcquire())
Explanation += " acquire";
if (P.isAtomicOrderingRelease())
Explanation += " release";
if (P.isAtomicOrderingAcquireRelease())
Explanation += " acq_rel";
if (P.isAtomicOrderingSequentiallyConsistent())
Explanation += " seq_cst";
if (P.isAtomicOrderingAcquireOrStronger())
Explanation += " >=acquire";
if (P.isAtomicOrderingWeakerThanAcquire())
Explanation += " <acquire";
if (P.isAtomicOrderingReleaseOrStronger())
Explanation += " >=release";
if (P.isAtomicOrderingWeakerThanRelease())
Explanation += " <release";
}
return Explanation;
}
std::string explainOperator(Record *Operator) {
if (Operator->isSubClassOf("SDNode"))
return (" (" + Operator->getValueAsString("Opcode") + ")").str();
if (Operator->isSubClassOf("Intrinsic"))
return (" (Operator is an Intrinsic, " + Operator->getName() + ")").str();
if (Operator->isSubClassOf("ComplexPattern"))
return (" (Operator is an unmapped ComplexPattern, " + Operator->getName() +
")")
.str();
if (Operator->isSubClassOf("SDNodeXForm"))
return (" (Operator is an unmapped SDNodeXForm, " + Operator->getName() +
")")
.str();
return (" (Operator " + Operator->getName() + " not understood)").str();
}
/// Helper function to let the emitter report skip reason error messages.
static Error failedImport(const Twine &Reason) {
return make_error<StringError>(Reason, inconvertibleErrorCode());
}
static Error isTrivialOperatorNode(const TreePatternNode *N) {
std::string Explanation = "";
std::string Separator = "";
bool HasUnsupportedPredicate = false;
for (const auto &Predicate : N->getPredicateFns()) {
if (Predicate.isAlwaysTrue())
continue;
if (Predicate.isImmediatePattern())
continue;
if (Predicate.isNonExtLoad() || Predicate.isAnyExtLoad() ||
Predicate.isSignExtLoad() || Predicate.isZeroExtLoad())
continue;
if (Predicate.isNonTruncStore())
continue;
if (Predicate.isLoad() && Predicate.getMemoryVT())
continue;
if (Predicate.isLoad() || Predicate.isStore()) {
if (Predicate.isUnindexed())
continue;
}
if (Predicate.isAtomic() && Predicate.getMemoryVT())
continue;
if (Predicate.isAtomic() &&
(Predicate.isAtomicOrderingMonotonic() ||
Predicate.isAtomicOrderingAcquire() ||
Predicate.isAtomicOrderingRelease() ||
Predicate.isAtomicOrderingAcquireRelease() ||
Predicate.isAtomicOrderingSequentiallyConsistent() ||
Predicate.isAtomicOrderingAcquireOrStronger() ||
Predicate.isAtomicOrderingWeakerThanAcquire() ||
Predicate.isAtomicOrderingReleaseOrStronger() ||
Predicate.isAtomicOrderingWeakerThanRelease()))
continue;
HasUnsupportedPredicate = true;
Explanation = Separator + "Has a predicate (" + explainPredicates(N) + ")";
Separator = ", ";
Explanation += (Separator + "first-failing:" +
Predicate.getOrigPatFragRecord()->getRecord()->getName())
.str();
break;
}
if (!HasUnsupportedPredicate)
return Error::success();
return failedImport(Explanation);
}
static Record *getInitValueAsRegClass(Init *V) {
if (DefInit *VDefInit = dyn_cast<DefInit>(V)) {
if (VDefInit->getDef()->isSubClassOf("RegisterOperand"))
return VDefInit->getDef()->getValueAsDef("RegClass");
if (VDefInit->getDef()->isSubClassOf("RegisterClass"))
return VDefInit->getDef();
}
return nullptr;
}
std::string
getNameForFeatureBitset(const std::vector<Record *> &FeatureBitset) {
std::string Name = "GIFBS";
for (const auto &Feature : FeatureBitset)
Name += ("_" + Feature->getName()).str();
return Name;
}
//===- MatchTable Helpers -------------------------------------------------===//
class MatchTable;
/// A record to be stored in a MatchTable.
///
/// This class represents any and all output that may be required to emit the
/// MatchTable. Instances are most often configured to represent an opcode or
/// value that will be emitted to the table with some formatting but it can also
/// represent commas, comments, and other formatting instructions.
struct MatchTableRecord {
enum RecordFlagsBits {
MTRF_None = 0x0,
/// Causes EmitStr to be formatted as comment when emitted.
MTRF_Comment = 0x1,
/// Causes the record value to be followed by a comma when emitted.
MTRF_CommaFollows = 0x2,
/// Causes the record value to be followed by a line break when emitted.
MTRF_LineBreakFollows = 0x4,
/// Indicates that the record defines a label and causes an additional
/// comment to be emitted containing the index of the label.
MTRF_Label = 0x8,
/// Causes the record to be emitted as the index of the label specified by
/// LabelID along with a comment indicating where that label is.
MTRF_JumpTarget = 0x10,
/// Causes the formatter to add a level of indentation before emitting the
/// record.
MTRF_Indent = 0x20,
/// Causes the formatter to remove a level of indentation after emitting the
/// record.
MTRF_Outdent = 0x40,
};
/// When MTRF_Label or MTRF_JumpTarget is used, indicates a label id to
/// reference or define.
unsigned LabelID;
/// The string to emit. Depending on the MTRF_* flags it may be a comment, a
/// value, a label name.
std::string EmitStr;
private:
/// The number of MatchTable elements described by this record. Comments are 0
/// while values are typically 1. Values >1 may occur when we need to emit
/// values that exceed the size of a MatchTable element.
unsigned NumElements;
public:
/// A bitfield of RecordFlagsBits flags.
unsigned Flags;
/// The actual run-time value, if known
int64_t RawValue;
MatchTableRecord(Optional<unsigned> LabelID_, StringRef EmitStr,
unsigned NumElements, unsigned Flags,
int64_t RawValue = std::numeric_limits<int64_t>::min())
: LabelID(LabelID_.hasValue() ? LabelID_.getValue() : ~0u),
EmitStr(EmitStr), NumElements(NumElements), Flags(Flags),
RawValue(RawValue) {
assert((!LabelID_.hasValue() || LabelID != ~0u) &&
"This value is reserved for non-labels");
}
MatchTableRecord(const MatchTableRecord &Other) = default;
MatchTableRecord(MatchTableRecord &&Other) = default;
/// Useful if a Match Table Record gets optimized out
void turnIntoComment() {
Flags |= MTRF_Comment;
Flags &= ~MTRF_CommaFollows;
NumElements = 0;
}
/// For Jump Table generation purposes
bool operator<(const MatchTableRecord &Other) const {
return RawValue < Other.RawValue;
}
int64_t getRawValue() const { return RawValue; }
void emit(raw_ostream &OS, bool LineBreakNextAfterThis,
const MatchTable &Table) const;
unsigned size() const { return NumElements; }
};
class Matcher;
/// Holds the contents of a generated MatchTable to enable formatting and the
/// necessary index tracking needed to support GIM_Try.
class MatchTable {
/// An unique identifier for the table. The generated table will be named
/// MatchTable${ID}.
unsigned ID;
/// The records that make up the table. Also includes comments describing the
/// values being emitted and line breaks to format it.
std::vector<MatchTableRecord> Contents;
/// The currently defined labels.
DenseMap<unsigned, unsigned> LabelMap;
/// Tracks the sum of MatchTableRecord::NumElements as the table is built.
unsigned CurrentSize = 0;
/// A unique identifier for a MatchTable label.
unsigned CurrentLabelID = 0;
/// Determines if the table should be instrumented for rule coverage tracking.
bool IsWithCoverage;
public:
static MatchTableRecord LineBreak;
static MatchTableRecord Comment(StringRef Comment) {
return MatchTableRecord(None, Comment, 0, MatchTableRecord::MTRF_Comment);
}
static MatchTableRecord Opcode(StringRef Opcode, int IndentAdjust = 0) {
unsigned ExtraFlags = 0;
if (IndentAdjust > 0)
ExtraFlags |= MatchTableRecord::MTRF_Indent;
if (IndentAdjust < 0)
ExtraFlags |= MatchTableRecord::MTRF_Outdent;
return MatchTableRecord(None, Opcode, 1,
MatchTableRecord::MTRF_CommaFollows | ExtraFlags);
}
static MatchTableRecord NamedValue(StringRef NamedValue) {
return MatchTableRecord(None, NamedValue, 1,
MatchTableRecord::MTRF_CommaFollows);
}
static MatchTableRecord NamedValue(StringRef NamedValue, int64_t RawValue) {
return MatchTableRecord(None, NamedValue, 1,
MatchTableRecord::MTRF_CommaFollows, RawValue);
}
static MatchTableRecord NamedValue(StringRef Namespace,
StringRef NamedValue) {
return MatchTableRecord(None, (Namespace + "::" + NamedValue).str(), 1,
MatchTableRecord::MTRF_CommaFollows);
}
static MatchTableRecord NamedValue(StringRef Namespace, StringRef NamedValue,
int64_t RawValue) {
return MatchTableRecord(None, (Namespace + "::" + NamedValue).str(), 1,
MatchTableRecord::MTRF_CommaFollows, RawValue);
}
static MatchTableRecord IntValue(int64_t IntValue) {
return MatchTableRecord(None, llvm::to_string(IntValue), 1,
MatchTableRecord::MTRF_CommaFollows);
}
static MatchTableRecord Label(unsigned LabelID) {
return MatchTableRecord(LabelID, "Label " + llvm::to_string(LabelID), 0,
MatchTableRecord::MTRF_Label |
MatchTableRecord::MTRF_Comment |
MatchTableRecord::MTRF_LineBreakFollows);
}
static MatchTableRecord JumpTarget(unsigned LabelID) {
return MatchTableRecord(LabelID, "Label " + llvm::to_string(LabelID), 1,
MatchTableRecord::MTRF_JumpTarget |
MatchTableRecord::MTRF_Comment |
MatchTableRecord::MTRF_CommaFollows);
}
static MatchTable buildTable(ArrayRef<Matcher *> Rules, bool WithCoverage);
MatchTable(bool WithCoverage, unsigned ID = 0)
: ID(ID), IsWithCoverage(WithCoverage) {}
bool isWithCoverage() const { return IsWithCoverage; }
void push_back(const MatchTableRecord &Value) {
if (Value.Flags & MatchTableRecord::MTRF_Label)
defineLabel(Value.LabelID);
Contents.push_back(Value);
CurrentSize += Value.size();
}
unsigned allocateLabelID() { return CurrentLabelID++; }
void defineLabel(unsigned LabelID) {
LabelMap.insert(std::make_pair(LabelID, CurrentSize));
}
unsigned getLabelIndex(unsigned LabelID) const {
const auto I = LabelMap.find(LabelID);
assert(I != LabelMap.end() && "Use of undeclared label");
return I->second;
}
void emitUse(raw_ostream &OS) const { OS << "MatchTable" << ID; }
void emitDeclaration(raw_ostream &OS) const {
unsigned Indentation = 4;
OS << " constexpr static int64_t MatchTable" << ID << "[] = {";
LineBreak.emit(OS, true, *this);
OS << std::string(Indentation, ' ');
for (auto I = Contents.begin(), E = Contents.end(); I != E;
++I) {
bool LineBreakIsNext = false;
const auto &NextI = std::next(I);
if (NextI != E) {
if (NextI->EmitStr == "" &&
NextI->Flags == MatchTableRecord::MTRF_LineBreakFollows)
LineBreakIsNext = true;
}
if (I->Flags & MatchTableRecord::MTRF_Indent)
Indentation += 2;
I->emit(OS, LineBreakIsNext, *this);
if (I->Flags & MatchTableRecord::MTRF_LineBreakFollows)
OS << std::string(Indentation, ' ');
if (I->Flags & MatchTableRecord::MTRF_Outdent)
Indentation -= 2;
}
OS << "};\n";
}
};
MatchTableRecord MatchTable::LineBreak = {
None, "" /* Emit String */, 0 /* Elements */,
MatchTableRecord::MTRF_LineBreakFollows};
void MatchTableRecord::emit(raw_ostream &OS, bool LineBreakIsNextAfterThis,
const MatchTable &Table) const {
bool UseLineComment =
LineBreakIsNextAfterThis | (Flags & MTRF_LineBreakFollows);
if (Flags & (MTRF_JumpTarget | MTRF_CommaFollows))
UseLineComment = false;
if (Flags & MTRF_Comment)
OS << (UseLineComment ? "// " : "/*");
OS << EmitStr;
if (Flags & MTRF_Label)
OS << ": @" << Table.getLabelIndex(LabelID);
if (Flags & MTRF_Comment && !UseLineComment)
OS << "*/";
if (Flags & MTRF_JumpTarget) {
if (Flags & MTRF_Comment)
OS << " ";
OS << Table.getLabelIndex(LabelID);
}
if (Flags & MTRF_CommaFollows) {
OS << ",";
if (!LineBreakIsNextAfterThis && !(Flags & MTRF_LineBreakFollows))
OS << " ";
}
if (Flags & MTRF_LineBreakFollows)
OS << "\n";
}
MatchTable &operator<<(MatchTable &Table, const MatchTableRecord &Value) {
Table.push_back(Value);
return Table;
}
//===- Matchers -----------------------------------------------------------===//
class OperandMatcher;
class MatchAction;
class PredicateMatcher;
class RuleMatcher;
class Matcher {
public:
virtual ~Matcher() = default;
virtual void optimize() {}
virtual void emit(MatchTable &Table) = 0;
virtual bool hasFirstCondition() const = 0;
virtual const PredicateMatcher &getFirstCondition() const = 0;
virtual std::unique_ptr<PredicateMatcher> popFirstCondition() = 0;
};
MatchTable MatchTable::buildTable(ArrayRef<Matcher *> Rules,
bool WithCoverage) {
MatchTable Table(WithCoverage);
for (Matcher *Rule : Rules)
Rule->emit(Table);
return Table << MatchTable::Opcode("GIM_Reject") << MatchTable::LineBreak;
}
class GroupMatcher final : public Matcher {
/// Conditions that form a common prefix of all the matchers contained.
SmallVector<std::unique_ptr<PredicateMatcher>, 1> Conditions;
/// All the nested matchers, sharing a common prefix.
std::vector<Matcher *> Matchers;
/// An owning collection for any auxiliary matchers created while optimizing
/// nested matchers contained.
std::vector<std::unique_ptr<Matcher>> MatcherStorage;
public:
/// Add a matcher to the collection of nested matchers if it meets the
/// requirements, and return true. If it doesn't, do nothing and return false.
///
/// Expected to preserve its argument, so it could be moved out later on.
bool addMatcher(Matcher &Candidate);
/// Mark the matcher as fully-built and ensure any invariants expected by both
/// optimize() and emit(...) methods. Generally, both sequences of calls
/// are expected to lead to a sensible result:
///
/// addMatcher(...)*; finalize(); optimize(); emit(...); and
/// addMatcher(...)*; finalize(); emit(...);
///
/// or generally
///
/// addMatcher(...)*; finalize(); { optimize()*; emit(...); }*
///
/// Multiple calls to optimize() are expected to be handled gracefully, though
/// optimize() is not expected to be idempotent. Multiple calls to finalize()
/// aren't generally supported. emit(...) is expected to be non-mutating and
/// producing the exact same results upon repeated calls.
///
/// addMatcher() calls after the finalize() call are not supported.
///
/// finalize() and optimize() are both allowed to mutate the contained
/// matchers, so moving them out after finalize() is not supported.
void finalize();
void optimize() override {}
void emit(MatchTable &Table) override;
/// Could be used to move out the matchers added previously, unless finalize()
/// has been already called. If any of the matchers are moved out, the group
/// becomes safe to destroy, but not safe to re-use for anything else.
iterator_range<std::vector<Matcher *>::iterator> matchers() {
return make_range(Matchers.begin(), Matchers.end());
}
size_t size() const { return Matchers.size(); }
bool empty() const { return Matchers.empty(); }
std::unique_ptr<PredicateMatcher> popFirstCondition() override {
assert(!Conditions.empty() &&
"Trying to pop a condition from a condition-less group");
std::unique_ptr<PredicateMatcher> P = std::move(Conditions.front());
Conditions.erase(Conditions.begin());
return P;
}
const PredicateMatcher &getFirstCondition() const override {
assert(!Conditions.empty() &&
"Trying to get a condition from a condition-less group");
return *Conditions.front();
}
bool hasFirstCondition() const override { return !Conditions.empty(); }
private:
/// See if a candidate matcher could be added to this group solely by
/// analyzing its first condition.
bool candidateConditionMatches(const PredicateMatcher &Predicate) const;
};
/// Generates code to check that a match rule matches.
class RuleMatcher : public Matcher {
public:
using ActionList = std::list<std::unique_ptr<MatchAction>>;
using action_iterator = ActionList::iterator;
protected:
/// A list of matchers that all need to succeed for the current rule to match.
/// FIXME: This currently supports a single match position but could be
/// extended to support multiple positions to support div/rem fusion or
/// load-multiple instructions.
using MatchersTy = std::vector<std::unique_ptr<InstructionMatcher>> ;
MatchersTy Matchers;
/// A list of actions that need to be taken when all predicates in this rule
/// have succeeded.
ActionList Actions;
using DefinedInsnVariablesMap = std::map<InstructionMatcher *, unsigned>;
/// A map of instruction matchers to the local variables
DefinedInsnVariablesMap InsnVariableIDs;
using MutatableInsnSet = SmallPtrSet<InstructionMatcher *, 4>;
// The set of instruction matchers that have not yet been claimed for mutation
// by a BuildMI.
MutatableInsnSet MutatableInsns;
/// A map of named operands defined by the matchers that may be referenced by
/// the renderers.
StringMap<OperandMatcher *> DefinedOperands;
/// ID for the next instruction variable defined with implicitlyDefineInsnVar()
unsigned NextInsnVarID;
/// ID for the next output instruction allocated with allocateOutputInsnID()
unsigned NextOutputInsnID;
/// ID for the next temporary register ID allocated with allocateTempRegID()
unsigned NextTempRegID;
std::vector<Record *> RequiredFeatures;
std::vector<std::unique_ptr<PredicateMatcher>> EpilogueMatchers;
ArrayRef<SMLoc> SrcLoc;
typedef std::tuple<Record *, unsigned, unsigned>
DefinedComplexPatternSubOperand;
typedef StringMap<DefinedComplexPatternSubOperand>
DefinedComplexPatternSubOperandMap;
/// A map of Symbolic Names to ComplexPattern sub-operands.
DefinedComplexPatternSubOperandMap ComplexSubOperands;
uint64_t RuleID;
static uint64_t NextRuleID;
public:
RuleMatcher(ArrayRef<SMLoc> SrcLoc)
: Matchers(), Actions(), InsnVariableIDs(), MutatableInsns(),
DefinedOperands(), NextInsnVarID(0), NextOutputInsnID(0),
NextTempRegID(0), SrcLoc(SrcLoc), ComplexSubOperands(),
RuleID(NextRuleID++) {}
RuleMatcher(RuleMatcher &&Other) = default;
RuleMatcher &operator=(RuleMatcher &&Other) = default;
uint64_t getRuleID() const { return RuleID; }
InstructionMatcher &addInstructionMatcher(StringRef SymbolicName);
void addRequiredFeature(Record *Feature);
const std::vector<Record *> &getRequiredFeatures() const;
template <class Kind, class... Args> Kind &addAction(Args &&... args);
template <class Kind, class... Args>
action_iterator insertAction(action_iterator InsertPt, Args &&... args);
/// Define an instruction without emitting any code to do so.
unsigned implicitlyDefineInsnVar(InstructionMatcher &Matcher);
unsigned getInsnVarID(InstructionMatcher &InsnMatcher) const;
DefinedInsnVariablesMap::const_iterator defined_insn_vars_begin() const {
return InsnVariableIDs.begin();
}
DefinedInsnVariablesMap::const_iterator defined_insn_vars_end() const {
return InsnVariableIDs.end();
}
iterator_range<typename DefinedInsnVariablesMap::const_iterator>
defined_insn_vars() const {
return make_range(defined_insn_vars_begin(), defined_insn_vars_end());
}
MutatableInsnSet::const_iterator mutatable_insns_begin() const {
return MutatableInsns.begin();
}
MutatableInsnSet::const_iterator mutatable_insns_end() const {
return MutatableInsns.end();
}
iterator_range<typename MutatableInsnSet::const_iterator>
mutatable_insns() const {
return make_range(mutatable_insns_begin(), mutatable_insns_end());
}
void reserveInsnMatcherForMutation(InstructionMatcher *InsnMatcher) {
bool R = MutatableInsns.erase(InsnMatcher);
assert(R && "Reserving a mutatable insn that isn't available");
(void)R;
}
action_iterator actions_begin() { return Actions.begin(); }
action_iterator actions_end() { return Actions.end(); }
iterator_range<action_iterator> actions() {
return make_range(actions_begin(), actions_end());
}
void defineOperand(StringRef SymbolicName, OperandMatcher &OM);
void defineComplexSubOperand(StringRef SymbolicName, Record *ComplexPattern,
unsigned RendererID, unsigned SubOperandID) {
assert(ComplexSubOperands.count(SymbolicName) == 0 && "Already defined");
ComplexSubOperands[SymbolicName] =
std::make_tuple(ComplexPattern, RendererID, SubOperandID);
}
Optional<DefinedComplexPatternSubOperand>
getComplexSubOperand(StringRef SymbolicName) const {
const auto &I = ComplexSubOperands.find(SymbolicName);
if (I == ComplexSubOperands.end())
return None;
return I->second;
}
InstructionMatcher &getInstructionMatcher(StringRef SymbolicName) const;
const OperandMatcher &getOperandMatcher(StringRef Name) const;
void optimize() override;
void emit(MatchTable &Table) override;
/// Compare the priority of this object and B.
///
/// Returns true if this object is more important than B.
bool isHigherPriorityThan(const RuleMatcher &B) const;
/// Report the maximum number of temporary operands needed by the rule
/// matcher.
unsigned countRendererFns() const;
std::unique_ptr<PredicateMatcher> popFirstCondition() override;
const PredicateMatcher &getFirstCondition() const override;
- LLTCodeGen getFirstConditionAsRootType();
bool hasFirstCondition() const override;
unsigned getNumOperands() const;
StringRef getOpcode() const;
// FIXME: Remove this as soon as possible
InstructionMatcher &insnmatchers_front() const { return *Matchers.front(); }
unsigned allocateOutputInsnID() { return NextOutputInsnID++; }
unsigned allocateTempRegID() { return NextTempRegID++; }
iterator_range<MatchersTy::iterator> insnmatchers() {
return make_range(Matchers.begin(), Matchers.end());
}
bool insnmatchers_empty() const { return Matchers.empty(); }
void insnmatchers_pop_front() { Matchers.erase(Matchers.begin()); }
};
uint64_t RuleMatcher::NextRuleID = 0;
using action_iterator = RuleMatcher::action_iterator;
template <class PredicateTy> class PredicateListMatcher {
private:
/// Template instantiations should specialize this to return a string to use
/// for the comment emitted when there are no predicates.
std::string getNoPredicateComment() const;
protected:
using PredicatesTy = std::deque<std::unique_ptr<PredicateTy>>;
PredicatesTy Predicates;
/// Track if the list of predicates was manipulated by one of the optimization
/// methods.
bool Optimized = false;
public:
/// Construct a new predicate and add it to the matcher.
template <class Kind, class... Args>
Optional<Kind *> addPredicate(Args &&... args);
typename PredicatesTy::iterator predicates_begin() {
return Predicates.begin();
}
typename PredicatesTy::iterator predicates_end() {
return Predicates.end();
}
iterator_range<typename PredicatesTy::iterator> predicates() {
return make_range(predicates_begin(), predicates_end());
}
typename PredicatesTy::size_type predicates_size() const {
return Predicates.size();
}
bool predicates_empty() const { return Predicates.empty(); }
std::unique_ptr<PredicateTy> predicates_pop_front() {
std::unique_ptr<PredicateTy> Front = std::move(Predicates.front());
Predicates.pop_front();
Optimized = true;
return Front;
}
void prependPredicate(std::unique_ptr<PredicateTy> &&Predicate) {
Predicates.push_front(std::move(Predicate));
}
void eraseNullPredicates() {
const auto NewEnd =
std::stable_partition(Predicates.begin(), Predicates.end(),
std::logical_not<std::unique_ptr<PredicateTy>>());
if (NewEnd != Predicates.begin()) {
Predicates.erase(Predicates.begin(), NewEnd);
Optimized = true;
}
}
/// Emit MatchTable opcodes that tests whether all the predicates are met.
template <class... Args>
void emitPredicateListOpcodes(MatchTable &Table, Args &&... args) {
if (Predicates.empty() && !Optimized) {
Table << MatchTable::Comment(getNoPredicateComment())
<< MatchTable::LineBreak;
return;
}
for (const auto &Predicate : predicates())
Predicate->emitPredicateOpcodes(Table, std::forward<Args>(args)...);
}
};
class PredicateMatcher {
public:
/// This enum is used for RTTI and also defines the priority that is given to
/// the predicate when generating the matcher code. Kinds with higher priority
/// must be tested first.
///
/// The relative priority of OPM_LLT, OPM_RegBank, and OPM_MBB do not matter
/// but OPM_Int must have priority over OPM_RegBank since constant integers
/// are represented by a virtual register defined by a G_CONSTANT instruction.
///
/// Note: The relative priority between IPM_ and OPM_ does not matter, they
/// are currently not compared between each other.
enum PredicateKind {
IPM_Opcode,
IPM_NumOperands,
IPM_ImmPredicate,
IPM_AtomicOrderingMMO,
IPM_MemoryLLTSize,
IPM_MemoryVsLLTSize,
OPM_SameOperand,
OPM_ComplexPattern,
OPM_IntrinsicID,
OPM_Instruction,
OPM_Int,
OPM_LiteralInt,
OPM_LLT,
OPM_PointerToAny,
OPM_RegBank,
OPM_MBB,
};
protected:
PredicateKind Kind;
unsigned InsnVarID;
unsigned OpIdx;
public:
PredicateMatcher(PredicateKind Kind, unsigned InsnVarID, unsigned OpIdx = ~0)
: Kind(Kind), InsnVarID(InsnVarID), OpIdx(OpIdx) {}
unsigned getInsnVarID() const { return InsnVarID; }
unsigned getOpIdx() const { return OpIdx; }
virtual ~PredicateMatcher() = default;
/// Emit MatchTable opcodes that check the predicate for the given operand.
virtual void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const = 0;
PredicateKind getKind() const { return Kind; }
virtual bool isIdentical(const PredicateMatcher &B) const {
return B.getKind() == getKind() && InsnVarID == B.InsnVarID &&
OpIdx == B.OpIdx;
}
virtual bool isIdenticalDownToValue(const PredicateMatcher &B) const {
return hasValue() && PredicateMatcher::isIdentical(B);
}
virtual MatchTableRecord getValue() const {
assert(hasValue() && "Can not get a value of a value-less predicate!");
llvm_unreachable("Not implemented yet");
}
virtual bool hasValue() const { return false; }
/// Report the maximum number of temporary operands needed by the predicate
/// matcher.
virtual unsigned countRendererFns() const { return 0; }
};
/// Generates code to check a predicate of an operand.
///
/// Typical predicates include:
/// * Operand is a particular register.
/// * Operand is assigned a particular register bank.
/// * Operand is an MBB.
class OperandPredicateMatcher : public PredicateMatcher {
public:
OperandPredicateMatcher(PredicateKind Kind, unsigned InsnVarID,
unsigned OpIdx)
: PredicateMatcher(Kind, InsnVarID, OpIdx) {}
virtual ~OperandPredicateMatcher() {}
/// Compare the priority of this object and B.
///
/// Returns true if this object is more important than B.
virtual bool isHigherPriorityThan(const OperandPredicateMatcher &B) const;
};
template <>
std::string
PredicateListMatcher<OperandPredicateMatcher>::getNoPredicateComment() const {
return "No operand predicates";
}
/// Generates code to check that a register operand is defined by the same exact
/// one as another.
class SameOperandMatcher : public OperandPredicateMatcher {
std::string MatchingName;
public:
SameOperandMatcher(unsigned InsnVarID, unsigned OpIdx, StringRef MatchingName)
: OperandPredicateMatcher(OPM_SameOperand, InsnVarID, OpIdx),
MatchingName(MatchingName) {}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == OPM_SameOperand;
}
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override;
bool isIdentical(const PredicateMatcher &B) const override {
return OperandPredicateMatcher::isIdentical(B) &&
MatchingName == cast<SameOperandMatcher>(&B)->MatchingName;
}
};
/// Generates code to check that an operand is a particular LLT.
class LLTOperandMatcher : public OperandPredicateMatcher {
protected:
LLTCodeGen Ty;
public:
static std::map<LLTCodeGen, unsigned> TypeIDValues;
static void initTypeIDValuesMap() {
TypeIDValues.clear();
unsigned ID = 0;
for (const LLTCodeGen LLTy : KnownTypes)
TypeIDValues[LLTy] = ID++;
}
LLTOperandMatcher(unsigned InsnVarID, unsigned OpIdx, const LLTCodeGen &Ty)
: OperandPredicateMatcher(OPM_LLT, InsnVarID, OpIdx), Ty(Ty) {
KnownTypes.insert(Ty);
}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == OPM_LLT;
}
bool isIdentical(const PredicateMatcher &B) const override {
return OperandPredicateMatcher::isIdentical(B) &&
Ty == cast<LLTOperandMatcher>(&B)->Ty;
}
MatchTableRecord getValue() const override {
const auto VI = TypeIDValues.find(Ty);
if (VI == TypeIDValues.end())
return MatchTable::NamedValue(getTy().getCxxEnumValue());
return MatchTable::NamedValue(getTy().getCxxEnumValue(), VI->second);
}
bool hasValue() const override {
if (TypeIDValues.size() != KnownTypes.size())
initTypeIDValuesMap();
return TypeIDValues.count(Ty);
}
LLTCodeGen getTy() const { return Ty; }
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIM_CheckType") << MatchTable::Comment("MI")
<< MatchTable::IntValue(InsnVarID) << MatchTable::Comment("Op")
<< MatchTable::IntValue(OpIdx) << MatchTable::Comment("Type")
<< getValue() << MatchTable::LineBreak;
}
};
std::map<LLTCodeGen, unsigned> LLTOperandMatcher::TypeIDValues;
/// Generates code to check that an operand is a pointer to any address space.
///
/// In SelectionDAG, the types did not describe pointers or address spaces. As a
/// result, iN is used to describe a pointer of N bits to any address space and
/// PatFrag predicates are typically used to constrain the address space. There's
/// no reliable means to derive the missing type information from the pattern so
/// imported rules must test the components of a pointer separately.
///
/// If SizeInBits is zero, then the pointer size will be obtained from the
/// subtarget.
class PointerToAnyOperandMatcher : public OperandPredicateMatcher {
protected:
unsigned SizeInBits;
public:
PointerToAnyOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
unsigned SizeInBits)
: OperandPredicateMatcher(OPM_PointerToAny, InsnVarID, OpIdx),
SizeInBits(SizeInBits) {}
static bool classof(const OperandPredicateMatcher *P) {
return P->getKind() == OPM_PointerToAny;
}
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIM_CheckPointerToAny")
<< MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
<< MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
<< MatchTable::Comment("SizeInBits")
<< MatchTable::IntValue(SizeInBits) << MatchTable::LineBreak;
}
};
/// Generates code to check that an operand is a particular target constant.
class ComplexPatternOperandMatcher : public OperandPredicateMatcher {
protected:
const OperandMatcher &Operand;
const Record &TheDef;
unsigned getAllocatedTemporariesBaseID() const;
public:
bool isIdentical(const PredicateMatcher &B) const override { return false; }
ComplexPatternOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
const OperandMatcher &Operand,
const Record &TheDef)
: OperandPredicateMatcher(OPM_ComplexPattern, InsnVarID, OpIdx),
Operand(Operand), TheDef(TheDef) {}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == OPM_ComplexPattern;
}
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
unsigned ID = getAllocatedTemporariesBaseID();
Table << MatchTable::Opcode("GIM_CheckComplexPattern")
<< MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
<< MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
<< MatchTable::Comment("Renderer") << MatchTable::IntValue(ID)
<< MatchTable::NamedValue(("GICP_" + TheDef.getName()).str())
<< MatchTable::LineBreak;
}
unsigned countRendererFns() const override {
return 1;
}
};
/// Generates code to check that an operand is in a particular register bank.
class RegisterBankOperandMatcher : public OperandPredicateMatcher {
protected:
const CodeGenRegisterClass &RC;
public:
RegisterBankOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
const CodeGenRegisterClass &RC)
: OperandPredicateMatcher(OPM_RegBank, InsnVarID, OpIdx), RC(RC) {}
bool isIdentical(const PredicateMatcher &B) const override {
return OperandPredicateMatcher::isIdentical(B) &&
RC.getDef() == cast<RegisterBankOperandMatcher>(&B)->RC.getDef();
}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == OPM_RegBank;
}
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIM_CheckRegBankForClass")
<< MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
<< MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
<< MatchTable::Comment("RC")
<< MatchTable::NamedValue(RC.getQualifiedName() + "RegClassID")
<< MatchTable::LineBreak;
}
};
/// Generates code to check that an operand is a basic block.
class MBBOperandMatcher : public OperandPredicateMatcher {
public:
MBBOperandMatcher(unsigned InsnVarID, unsigned OpIdx)
: OperandPredicateMatcher(OPM_MBB, InsnVarID, OpIdx) {}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == OPM_MBB;
}
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIM_CheckIsMBB") << MatchTable::Comment("MI")
<< MatchTable::IntValue(InsnVarID) << MatchTable::Comment("Op")
<< MatchTable::IntValue(OpIdx) << MatchTable::LineBreak;
}
};
/// Generates code to check that an operand is a G_CONSTANT with a particular
/// int.
class ConstantIntOperandMatcher : public OperandPredicateMatcher {
protected:
int64_t Value;
public:
ConstantIntOperandMatcher(unsigned InsnVarID, unsigned OpIdx, int64_t Value)
: OperandPredicateMatcher(OPM_Int, InsnVarID, OpIdx), Value(Value) {}
bool isIdentical(const PredicateMatcher &B) const override {
return OperandPredicateMatcher::isIdentical(B) &&
Value == cast<ConstantIntOperandMatcher>(&B)->Value;
}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == OPM_Int;
}
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIM_CheckConstantInt")
<< MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
<< MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
<< MatchTable::IntValue(Value) << MatchTable::LineBreak;
}
};
/// Generates code to check that an operand is a raw int (where MO.isImm() or
/// MO.isCImm() is true).
class LiteralIntOperandMatcher : public OperandPredicateMatcher {
protected:
int64_t Value;
public:
LiteralIntOperandMatcher(unsigned InsnVarID, unsigned OpIdx, int64_t Value)
: OperandPredicateMatcher(OPM_LiteralInt, InsnVarID, OpIdx),
Value(Value) {}
bool isIdentical(const PredicateMatcher &B) const override {
return OperandPredicateMatcher::isIdentical(B) &&
Value == cast<LiteralIntOperandMatcher>(&B)->Value;
}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == OPM_LiteralInt;
}
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIM_CheckLiteralInt")
<< MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
<< MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
<< MatchTable::IntValue(Value) << MatchTable::LineBreak;
}
};
/// Generates code to check that an operand is an intrinsic ID.
class IntrinsicIDOperandMatcher : public OperandPredicateMatcher {
protected:
const CodeGenIntrinsic *II;
public:
IntrinsicIDOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
const CodeGenIntrinsic *II)
: OperandPredicateMatcher(OPM_IntrinsicID, InsnVarID, OpIdx), II(II) {}
bool isIdentical(const PredicateMatcher &B) const override {
return OperandPredicateMatcher::isIdentical(B) &&
II == cast<IntrinsicIDOperandMatcher>(&B)->II;
}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == OPM_IntrinsicID;
}
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIM_CheckIntrinsicID")
<< MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
<< MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
<< MatchTable::NamedValue("Intrinsic::" + II->EnumName)
<< MatchTable::LineBreak;
}
};
/// Generates code to check that a set of predicates match for a particular
/// operand.
class OperandMatcher : public PredicateListMatcher<OperandPredicateMatcher> {
protected:
InstructionMatcher &Insn;
unsigned OpIdx;
std::string SymbolicName;
/// The index of the first temporary variable allocated to this operand. The
/// number of allocated temporaries can be found with
/// countRendererFns().
unsigned AllocatedTemporariesBaseID;
public:
OperandMatcher(InstructionMatcher &Insn, unsigned OpIdx,
const std::string &SymbolicName,
unsigned AllocatedTemporariesBaseID)
: Insn(Insn), OpIdx(OpIdx), SymbolicName(SymbolicName),
AllocatedTemporariesBaseID(AllocatedTemporariesBaseID) {}
bool hasSymbolicName() const { return !SymbolicName.empty(); }
const StringRef getSymbolicName() const { return SymbolicName; }
void setSymbolicName(StringRef Name) {
assert(SymbolicName.empty() && "Operand already has a symbolic name");
SymbolicName = Name;
}
/// Construct a new operand predicate and add it to the matcher.
template <class Kind, class... Args>
Optional<Kind *> addPredicate(Args &&... args) {
if (isSameAsAnotherOperand())
return None;
Predicates.emplace_back(llvm::make_unique<Kind>(
getInsnVarID(), getOpIdx(), std::forward<Args>(args)...));
return static_cast<Kind *>(Predicates.back().get());
}
unsigned getOpIdx() const { return OpIdx; }
unsigned getInsnVarID() const;
std::string getOperandExpr(unsigned InsnVarID) const {
return "State.MIs[" + llvm::to_string(InsnVarID) + "]->getOperand(" +
llvm::to_string(OpIdx) + ")";
}
InstructionMatcher &getInstructionMatcher() const { return Insn; }
Error addTypeCheckPredicate(const TypeSetByHwMode &VTy,
bool OperandIsAPointer);
/// Emit MatchTable opcodes that test whether the instruction named in
/// InsnVarID matches all the predicates and all the operands.
void emitPredicateOpcodes(MatchTable &Table, RuleMatcher &Rule) {
if (!Optimized) {
std::string Comment;
raw_string_ostream CommentOS(Comment);
CommentOS << "MIs[" << getInsnVarID() << "] ";
if (SymbolicName.empty())
CommentOS << "Operand " << OpIdx;
else
CommentOS << SymbolicName;
Table << MatchTable::Comment(CommentOS.str()) << MatchTable::LineBreak;
}
emitPredicateListOpcodes(Table, Rule);
}
/// Compare the priority of this object and B.
///
/// Returns true if this object is more important than B.
bool isHigherPriorityThan(OperandMatcher &B) {
// Operand matchers involving more predicates have higher priority.
if (predicates_size() > B.predicates_size())
return true;
if (predicates_size() < B.predicates_size())
return false;
// This assumes that predicates are added in a consistent order.
for (auto &&Predicate : zip(predicates(), B.predicates())) {
if (std::get<0>(Predicate)->isHigherPriorityThan(*std::get<1>(Predicate)))
return true;
if (std::get<1>(Predicate)->isHigherPriorityThan(*std::get<0>(Predicate)))
return false;
}
return false;
};
/// Report the maximum number of temporary operands needed by the operand
/// matcher.
unsigned countRendererFns() {
return std::accumulate(
predicates().begin(), predicates().end(), 0,
[](unsigned A,
const std::unique_ptr<OperandPredicateMatcher> &Predicate) {
return A + Predicate->countRendererFns();
});
}
unsigned getAllocatedTemporariesBaseID() const {
return AllocatedTemporariesBaseID;
}
bool isSameAsAnotherOperand() {
for (const auto &Predicate : predicates())
if (isa<SameOperandMatcher>(Predicate))
return true;
return false;
}
};
Error OperandMatcher::addTypeCheckPredicate(const TypeSetByHwMode &VTy,
bool OperandIsAPointer) {
if (!VTy.isMachineValueType())
return failedImport("unsupported typeset");
if (VTy.getMachineValueType() == MVT::iPTR && OperandIsAPointer) {
addPredicate<PointerToAnyOperandMatcher>(0);
return Error::success();
}
auto OpTyOrNone = MVTToLLT(VTy.getMachineValueType().SimpleTy);
if (!OpTyOrNone)
return failedImport("unsupported type");
if (OperandIsAPointer)
addPredicate<PointerToAnyOperandMatcher>(OpTyOrNone->get().getSizeInBits());
else
addPredicate<LLTOperandMatcher>(*OpTyOrNone);
return Error::success();
}
unsigned ComplexPatternOperandMatcher::getAllocatedTemporariesBaseID() const {
return Operand.getAllocatedTemporariesBaseID();
}
/// Generates code to check a predicate on an instruction.
///
/// Typical predicates include:
/// * The opcode of the instruction is a particular value.
/// * The nsw/nuw flag is/isn't set.
class InstructionPredicateMatcher : public PredicateMatcher {
public:
InstructionPredicateMatcher(PredicateKind Kind, unsigned InsnVarID)
: PredicateMatcher(Kind, InsnVarID) {}
virtual ~InstructionPredicateMatcher() {}
/// Compare the priority of this object and B.
///
/// Returns true if this object is more important than B.
virtual bool
isHigherPriorityThan(const InstructionPredicateMatcher &B) const {
return Kind < B.Kind;
};
};
template <>
std::string
PredicateListMatcher<PredicateMatcher>::getNoPredicateComment() const {
return "No instruction predicates";
}
/// Generates code to check the opcode of an instruction.
class InstructionOpcodeMatcher : public InstructionPredicateMatcher {
protected:
const CodeGenInstruction *I;
static DenseMap<const CodeGenInstruction *, unsigned> OpcodeValues;
public:
static void initOpcodeValuesMap(const CodeGenTarget &Target) {
OpcodeValues.clear();
unsigned OpcodeValue = 0;
for (const CodeGenInstruction *I : Target.getInstructionsByEnumValue())
OpcodeValues[I] = OpcodeValue++;
}
InstructionOpcodeMatcher(unsigned InsnVarID, const CodeGenInstruction *I)
: InstructionPredicateMatcher(IPM_Opcode, InsnVarID), I(I) {}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == IPM_Opcode;
}
bool isIdentical(const PredicateMatcher &B) const override {
return InstructionPredicateMatcher::isIdentical(B) &&
I == cast<InstructionOpcodeMatcher>(&B)->I;
}
MatchTableRecord getValue() const override {
const auto VI = OpcodeValues.find(I);
if (VI != OpcodeValues.end())
return MatchTable::NamedValue(I->Namespace, I->TheDef->getName(),
VI->second);
return MatchTable::NamedValue(I->Namespace, I->TheDef->getName());
}
bool hasValue() const override { return OpcodeValues.count(I); }
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIM_CheckOpcode") << MatchTable::Comment("MI")
<< MatchTable::IntValue(InsnVarID) << getValue()
<< MatchTable::LineBreak;
}
/// Compare the priority of this object and B.
///
/// Returns true if this object is more important than B.
bool
isHigherPriorityThan(const InstructionPredicateMatcher &B) const override {
if (InstructionPredicateMatcher::isHigherPriorityThan(B))
return true;
if (B.InstructionPredicateMatcher::isHigherPriorityThan(*this))
return false;
// Prioritize opcodes for cosmetic reasons in the generated source. Although
// this is cosmetic at the moment, we may want to drive a similar ordering
// using instruction frequency information to improve compile time.
if (const InstructionOpcodeMatcher *BO =
dyn_cast<InstructionOpcodeMatcher>(&B))
return I->TheDef->getName() < BO->I->TheDef->getName();
return false;
};
bool isConstantInstruction() const {
return I->TheDef->getName() == "G_CONSTANT";
}
StringRef getOpcode() const { return I->TheDef->getName(); }
unsigned getNumOperands() const { return I->Operands.size(); }
StringRef getOperandType(unsigned OpIdx) const {
return I->Operands[OpIdx].OperandType;
}
};
DenseMap<const CodeGenInstruction *, unsigned>
InstructionOpcodeMatcher::OpcodeValues;
class InstructionNumOperandsMatcher final : public InstructionPredicateMatcher {
unsigned NumOperands = 0;
public:
InstructionNumOperandsMatcher(unsigned InsnVarID, unsigned NumOperands)
: InstructionPredicateMatcher(IPM_NumOperands, InsnVarID),
NumOperands(NumOperands) {}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == IPM_NumOperands;
}
bool isIdentical(const PredicateMatcher &B) const override {
return InstructionPredicateMatcher::isIdentical(B) &&
NumOperands == cast<InstructionNumOperandsMatcher>(&B)->NumOperands;
}
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIM_CheckNumOperands")
<< MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
<< MatchTable::Comment("Expected")
<< MatchTable::IntValue(NumOperands) << MatchTable::LineBreak;
}
};
/// Generates code to check that this instruction is a constant whose value
/// meets an immediate predicate.
///
/// Immediates are slightly odd since they are typically used like an operand
/// but are represented as an operator internally. We typically write simm8:$src
/// in a tablegen pattern, but this is just syntactic sugar for
/// (imm:i32)<<P:Predicate_simm8>>:$imm which more directly describes the nodes
/// that will be matched and the predicate (which is attached to the imm
/// operator) that will be tested. In SelectionDAG this describes a
/// ConstantSDNode whose internal value will be tested using the simm8 predicate.
///
/// The corresponding GlobalISel representation is %1 = G_CONSTANT iN Value. In
/// this representation, the immediate could be tested with an
/// InstructionMatcher, InstructionOpcodeMatcher, OperandMatcher, and a
/// OperandPredicateMatcher-subclass to check the Value meets the predicate but
/// there are two implementation issues with producing that matcher
/// configuration from the SelectionDAG pattern:
/// * ImmLeaf is a PatFrag whose root is an InstructionMatcher. This means that
/// were we to sink the immediate predicate to the operand we would have to
/// have two partial implementations of PatFrag support, one for immediates
/// and one for non-immediates.
/// * At the point we handle the predicate, the OperandMatcher hasn't been
/// created yet. If we were to sink the predicate to the OperandMatcher we
/// would also have to complicate (or duplicate) the code that descends and
/// creates matchers for the subtree.
/// Overall, it's simpler to handle it in the place it was found.
class InstructionImmPredicateMatcher : public InstructionPredicateMatcher {
protected:
TreePredicateFn Predicate;
public:
InstructionImmPredicateMatcher(unsigned InsnVarID,
const TreePredicateFn &Predicate)
: InstructionPredicateMatcher(IPM_ImmPredicate, InsnVarID),
Predicate(Predicate) {}
bool isIdentical(const PredicateMatcher &B) const override {
return InstructionPredicateMatcher::isIdentical(B) &&
Predicate.getOrigPatFragRecord() ==
cast<InstructionImmPredicateMatcher>(&B)
->Predicate.getOrigPatFragRecord();
}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == IPM_ImmPredicate;
}
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
Table << MatchTable::Opcode(getMatchOpcodeForPredicate(Predicate))
<< MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
<< MatchTable::Comment("Predicate")
<< MatchTable::NamedValue(getEnumNameForPredicate(Predicate))
<< MatchTable::LineBreak;
}
};
/// Generates code to check that a memory instruction has a atomic ordering
/// MachineMemoryOperand.
class AtomicOrderingMMOPredicateMatcher : public InstructionPredicateMatcher {
public:
enum AOComparator {
AO_Exactly,
AO_OrStronger,
AO_WeakerThan,
};
protected:
StringRef Order;
AOComparator Comparator;
public:
AtomicOrderingMMOPredicateMatcher(unsigned InsnVarID, StringRef Order,
AOComparator Comparator = AO_Exactly)
: InstructionPredicateMatcher(IPM_AtomicOrderingMMO, InsnVarID),
Order(Order), Comparator(Comparator) {}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == IPM_AtomicOrderingMMO;
}
bool isIdentical(const PredicateMatcher &B) const override {
if (!InstructionPredicateMatcher::isIdentical(B))
return false;
const auto &R = *cast<AtomicOrderingMMOPredicateMatcher>(&B);
return Order == R.Order && Comparator == R.Comparator;
}
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
StringRef Opcode = "GIM_CheckAtomicOrdering";
if (Comparator == AO_OrStronger)
Opcode = "GIM_CheckAtomicOrderingOrStrongerThan";
if (Comparator == AO_WeakerThan)
Opcode = "GIM_CheckAtomicOrderingWeakerThan";
Table << MatchTable::Opcode(Opcode) << MatchTable::Comment("MI")
<< MatchTable::IntValue(InsnVarID) << MatchTable::Comment("Order")
<< MatchTable::NamedValue(("(int64_t)AtomicOrdering::" + Order).str())
<< MatchTable::LineBreak;
}
};
/// Generates code to check that the size of an MMO is exactly N bytes.
class MemorySizePredicateMatcher : public InstructionPredicateMatcher {
protected:
unsigned MMOIdx;
uint64_t Size;
public:
MemorySizePredicateMatcher(unsigned InsnVarID, unsigned MMOIdx, unsigned Size)
: InstructionPredicateMatcher(IPM_MemoryLLTSize, InsnVarID),
MMOIdx(MMOIdx), Size(Size) {}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == IPM_MemoryLLTSize;
}
bool isIdentical(const PredicateMatcher &B) const override {
return InstructionPredicateMatcher::isIdentical(B) &&
MMOIdx == cast<MemorySizePredicateMatcher>(&B)->MMOIdx &&
Size == cast<MemorySizePredicateMatcher>(&B)->Size;
}
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIM_CheckMemorySizeEqualTo")
<< MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
<< MatchTable::Comment("MMO") << MatchTable::IntValue(MMOIdx)
<< MatchTable::Comment("Size") << MatchTable::IntValue(Size)
<< MatchTable::LineBreak;
}
};
/// Generates code to check that the size of an MMO is less-than, equal-to, or
/// greater than a given LLT.
class MemoryVsLLTSizePredicateMatcher : public InstructionPredicateMatcher {
public:
enum RelationKind {
GreaterThan,
EqualTo,
LessThan,
};
protected:
unsigned MMOIdx;
RelationKind Relation;
unsigned OpIdx;
public:
MemoryVsLLTSizePredicateMatcher(unsigned InsnVarID, unsigned MMOIdx,
enum RelationKind Relation,
unsigned OpIdx)
: InstructionPredicateMatcher(IPM_MemoryVsLLTSize, InsnVarID),
MMOIdx(MMOIdx), Relation(Relation), OpIdx(OpIdx) {}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == IPM_MemoryVsLLTSize;
}
bool isIdentical(const PredicateMatcher &B) const override {
return InstructionPredicateMatcher::isIdentical(B) &&
MMOIdx == cast<MemoryVsLLTSizePredicateMatcher>(&B)->MMOIdx &&
Relation == cast<MemoryVsLLTSizePredicateMatcher>(&B)->Relation &&
OpIdx == cast<MemoryVsLLTSizePredicateMatcher>(&B)->OpIdx;
}
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
Table << MatchTable::Opcode(Relation == EqualTo
? "GIM_CheckMemorySizeEqualToLLT"
: Relation == GreaterThan
? "GIM_CheckMemorySizeGreaterThanLLT"
: "GIM_CheckMemorySizeLessThanLLT")
<< MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
<< MatchTable::Comment("MMO") << MatchTable::IntValue(MMOIdx)
<< MatchTable::Comment("OpIdx") << MatchTable::IntValue(OpIdx)
<< MatchTable::LineBreak;
}
};
/// Generates code to check that a set of predicates and operands match for a
/// particular instruction.
///
/// Typical predicates include:
/// * Has a specific opcode.
/// * Has an nsw/nuw flag or doesn't.
class InstructionMatcher final : public PredicateListMatcher<PredicateMatcher> {
protected:
typedef std::vector<std::unique_ptr<OperandMatcher>> OperandVec;
RuleMatcher &Rule;
/// The operands to match. All rendered operands must be present even if the
/// condition is always true.
OperandVec Operands;
bool NumOperandsCheck = true;
std::string SymbolicName;
unsigned InsnVarID;
public:
InstructionMatcher(RuleMatcher &Rule, StringRef SymbolicName)
: Rule(Rule), SymbolicName(SymbolicName) {
// We create a new instruction matcher.
// Get a new ID for that instruction.
InsnVarID = Rule.implicitlyDefineInsnVar(*this);
}
/// Construct a new instruction predicate and add it to the matcher.
template <class Kind, class... Args>
Optional<Kind *> addPredicate(Args &&... args) {
Predicates.emplace_back(
llvm::make_unique<Kind>(getInsnVarID(), std::forward<Args>(args)...));
return static_cast<Kind *>(Predicates.back().get());
}
RuleMatcher &getRuleMatcher() const { return Rule; }
unsigned getInsnVarID() const { return InsnVarID; }
/// Add an operand to the matcher.
OperandMatcher &addOperand(unsigned OpIdx, const std::string &SymbolicName,
unsigned AllocatedTemporariesBaseID) {
Operands.emplace_back(new OperandMatcher(*this, OpIdx, SymbolicName,
AllocatedTemporariesBaseID));
if (!SymbolicName.empty())
Rule.defineOperand(SymbolicName, *Operands.back());
return *Operands.back();
}
OperandMatcher &getOperand(unsigned OpIdx) {
auto I = std::find_if(Operands.begin(), Operands.end(),
[&OpIdx](const std::unique_ptr<OperandMatcher> &X) {
return X->getOpIdx() == OpIdx;
});
if (I != Operands.end())
return **I;
llvm_unreachable("Failed to lookup operand");
}
StringRef getSymbolicName() const { return SymbolicName; }
unsigned getNumOperands() const { return Operands.size(); }
OperandVec::iterator operands_begin() { return Operands.begin(); }
OperandVec::iterator operands_end() { return Operands.end(); }
iterator_range<OperandVec::iterator> operands() {
return make_range(operands_begin(), operands_end());
}
OperandVec::const_iterator operands_begin() const { return Operands.begin(); }
OperandVec::const_iterator operands_end() const { return Operands.end(); }
iterator_range<OperandVec::const_iterator> operands() const {
return make_range(operands_begin(), operands_end());
}
bool operands_empty() const { return Operands.empty(); }
void pop_front() { Operands.erase(Operands.begin()); }
void optimize();
/// Emit MatchTable opcodes that test whether the instruction named in
/// InsnVarName matches all the predicates and all the operands.
void emitPredicateOpcodes(MatchTable &Table, RuleMatcher &Rule) {
if (NumOperandsCheck)
InstructionNumOperandsMatcher(InsnVarID, getNumOperands())
.emitPredicateOpcodes(Table, Rule);
emitPredicateListOpcodes(Table, Rule);
for (const auto &Operand : Operands)
Operand->emitPredicateOpcodes(Table, Rule);
}
/// Compare the priority of this object and B.
///
/// Returns true if this object is more important than B.
bool isHigherPriorityThan(InstructionMatcher &B) {
// Instruction matchers involving more operands have higher priority.
if (Operands.size() > B.Operands.size())
return true;
if (Operands.size() < B.Operands.size())
return false;
for (auto &&P : zip(predicates(), B.predicates())) {
auto L = static_cast<InstructionPredicateMatcher *>(std::get<0>(P).get());
auto R = static_cast<InstructionPredicateMatcher *>(std::get<1>(P).get());
if (L->isHigherPriorityThan(*R))
return true;
if (R->isHigherPriorityThan(*L))
return false;
}
for (const auto &Operand : zip(Operands, B.Operands)) {
if (std::get<0>(Operand)->isHigherPriorityThan(*std::get<1>(Operand)))
return true;
if (std::get<1>(Operand)->isHigherPriorityThan(*std::get<0>(Operand)))
return false;
}
return false;
};
/// Report the maximum number of temporary operands needed by the instruction
/// matcher.
unsigned countRendererFns() {
return std::accumulate(
predicates().begin(), predicates().end(), 0,
[](unsigned A,
const std::unique_ptr<PredicateMatcher> &Predicate) {
return A + Predicate->countRendererFns();
}) +
std::accumulate(
Operands.begin(), Operands.end(), 0,
[](unsigned A, const std::unique_ptr<OperandMatcher> &Operand) {
return A + Operand->countRendererFns();
});
}
InstructionOpcodeMatcher &getOpcodeMatcher() {
for (auto &P : predicates())
if (auto *OpMatcher = dyn_cast<InstructionOpcodeMatcher>(P.get()))
return *OpMatcher;
llvm_unreachable("Didn't find an opcode matcher");
}
bool isConstantInstruction() {
return getOpcodeMatcher().isConstantInstruction();
}
StringRef getOpcode() { return getOpcodeMatcher().getOpcode(); }
};
StringRef RuleMatcher::getOpcode() const {
return Matchers.front()->getOpcode();
}
unsigned RuleMatcher::getNumOperands() const {
return Matchers.front()->getNumOperands();
}
-LLTCodeGen RuleMatcher::getFirstConditionAsRootType() {
- InstructionMatcher &InsnMatcher = *Matchers.front();
- if (!InsnMatcher.predicates_empty())
- if (const auto *TM =
- dyn_cast<LLTOperandMatcher>(&**InsnMatcher.predicates_begin()))
- if (TM->getInsnVarID() == 0 && TM->getOpIdx() == 0)
- return TM->getTy();
- return {};
-}
-
/// Generates code to check that the operand is a register defined by an
/// instruction that matches the given instruction matcher.
///
/// For example, the pattern:
/// (set $dst, (G_MUL (G_ADD $src1, $src2), $src3))
/// would use an InstructionOperandMatcher for operand 1 of the G_MUL to match
/// the:
/// (G_ADD $src1, $src2)
/// subpattern.
class InstructionOperandMatcher : public OperandPredicateMatcher {
protected:
std::unique_ptr<InstructionMatcher> InsnMatcher;
public:
InstructionOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
RuleMatcher &Rule, StringRef SymbolicName)
: OperandPredicateMatcher(OPM_Instruction, InsnVarID, OpIdx),
InsnMatcher(new InstructionMatcher(Rule, SymbolicName)) {}
static bool classof(const PredicateMatcher *P) {
return P->getKind() == OPM_Instruction;
}
InstructionMatcher &getInsnMatcher() const { return *InsnMatcher; }
void emitCaptureOpcodes(MatchTable &Table, RuleMatcher &Rule) const {
const unsigned NewInsnVarID = InsnMatcher->getInsnVarID();
Table << MatchTable::Opcode("GIM_RecordInsn")
<< MatchTable::Comment("DefineMI")
<< MatchTable::IntValue(NewInsnVarID) << MatchTable::Comment("MI")
<< MatchTable::IntValue(getInsnVarID())
<< MatchTable::Comment("OpIdx") << MatchTable::IntValue(getOpIdx())
<< MatchTable::Comment("MIs[" + llvm::to_string(NewInsnVarID) + "]")
<< MatchTable::LineBreak;
}
void emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const override {
emitCaptureOpcodes(Table, Rule);
InsnMatcher->emitPredicateOpcodes(Table, Rule);
}
bool isHigherPriorityThan(const OperandPredicateMatcher &B) const override {
if (OperandPredicateMatcher::isHigherPriorityThan(B))
return true;
if (B.OperandPredicateMatcher::isHigherPriorityThan(*this))
return false;
if (const InstructionOperandMatcher *BP =
dyn_cast<InstructionOperandMatcher>(&B))
if (InsnMatcher->isHigherPriorityThan(*BP->InsnMatcher))
return true;
return false;
}
};
void InstructionMatcher::optimize() {
SmallVector<std::unique_ptr<PredicateMatcher>, 8> Stash;
const auto &OpcMatcher = getOpcodeMatcher();
Stash.push_back(predicates_pop_front());
if (Stash.back().get() == &OpcMatcher) {
if (NumOperandsCheck && OpcMatcher.getNumOperands() < getNumOperands())
Stash.emplace_back(
new InstructionNumOperandsMatcher(InsnVarID, getNumOperands()));
NumOperandsCheck = false;
}
if (InsnVarID > 0) {
assert(!Operands.empty() && "Nested instruction is expected to def a vreg");
for (auto &OP : Operands[0]->predicates())
OP.reset();
Operands[0]->eraseNullPredicates();
}
while (!Stash.empty())
prependPredicate(Stash.pop_back_val());
}
//===- Actions ------------------------------------------------------------===//
class OperandRenderer {
public:
enum RendererKind {
OR_Copy,
OR_CopyOrAddZeroReg,
OR_CopySubReg,
OR_CopyConstantAsImm,
OR_CopyFConstantAsFPImm,
OR_Imm,
OR_Register,
OR_TempRegister,
OR_ComplexPattern,
OR_Custom
};
protected:
RendererKind Kind;
public:
OperandRenderer(RendererKind Kind) : Kind(Kind) {}
virtual ~OperandRenderer() {}
RendererKind getKind() const { return Kind; }
virtual void emitRenderOpcodes(MatchTable &Table,
RuleMatcher &Rule) const = 0;
};
/// A CopyRenderer emits code to copy a single operand from an existing
/// instruction to the one being built.
class CopyRenderer : public OperandRenderer {
protected:
unsigned NewInsnID;
/// The name of the operand.
const StringRef SymbolicName;
public:
CopyRenderer(unsigned NewInsnID, StringRef SymbolicName)
: OperandRenderer(OR_Copy), NewInsnID(NewInsnID),
SymbolicName(SymbolicName) {
assert(!SymbolicName.empty() && "Cannot copy from an unspecified source");
}
static bool classof(const OperandRenderer *R) {
return R->getKind() == OR_Copy;
}
const StringRef getSymbolicName() const { return SymbolicName; }
void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
const OperandMatcher &Operand = Rule.getOperandMatcher(SymbolicName);
unsigned OldInsnVarID = Rule.getInsnVarID(Operand.getInstructionMatcher());
Table << MatchTable::Opcode("GIR_Copy") << MatchTable::Comment("NewInsnID")
<< MatchTable::IntValue(NewInsnID) << MatchTable::Comment("OldInsnID")
<< MatchTable::IntValue(OldInsnVarID) << MatchTable::Comment("OpIdx")
<< MatchTable::IntValue(Operand.getOpIdx())
<< MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
}
};
/// A CopyOrAddZeroRegRenderer emits code to copy a single operand from an
/// existing instruction to the one being built. If the operand turns out to be
/// a 'G_CONSTANT 0' then it replaces the operand with a zero register.
class CopyOrAddZeroRegRenderer : public OperandRenderer {
protected:
unsigned NewInsnID;
/// The name of the operand.
const StringRef SymbolicName;
const Record *ZeroRegisterDef;
public:
CopyOrAddZeroRegRenderer(unsigned NewInsnID,
StringRef SymbolicName, Record *ZeroRegisterDef)
: OperandRenderer(OR_CopyOrAddZeroReg), NewInsnID(NewInsnID),
SymbolicName(SymbolicName), ZeroRegisterDef(ZeroRegisterDef) {
assert(!SymbolicName.empty() && "Cannot copy from an unspecified source");
}
static bool classof(const OperandRenderer *R) {
return R->getKind() == OR_CopyOrAddZeroReg;
}
const StringRef getSymbolicName() const { return SymbolicName; }
void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
const OperandMatcher &Operand = Rule.getOperandMatcher(SymbolicName);
unsigned OldInsnVarID = Rule.getInsnVarID(Operand.getInstructionMatcher());
Table << MatchTable::Opcode("GIR_CopyOrAddZeroReg")
<< MatchTable::Comment("NewInsnID") << MatchTable::IntValue(NewInsnID)
<< MatchTable::Comment("OldInsnID")
<< MatchTable::IntValue(OldInsnVarID) << MatchTable::Comment("OpIdx")
<< MatchTable::IntValue(Operand.getOpIdx())
<< MatchTable::NamedValue(
(ZeroRegisterDef->getValue("Namespace")
? ZeroRegisterDef->getValueAsString("Namespace")
: ""),
ZeroRegisterDef->getName())
<< MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
}
};
/// A CopyConstantAsImmRenderer emits code to render a G_CONSTANT instruction to
/// an extended immediate operand.
class CopyConstantAsImmRenderer : public OperandRenderer {
protected:
unsigned NewInsnID;
/// The name of the operand.
const std::string SymbolicName;
bool Signed;
public:
CopyConstantAsImmRenderer(unsigned NewInsnID, StringRef SymbolicName)
: OperandRenderer(OR_CopyConstantAsImm), NewInsnID(NewInsnID),
SymbolicName(SymbolicName), Signed(true) {}
static bool classof(const OperandRenderer *R) {
return R->getKind() == OR_CopyConstantAsImm;
}
const StringRef getSymbolicName() const { return SymbolicName; }
void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
InstructionMatcher &InsnMatcher = Rule.getInstructionMatcher(SymbolicName);
unsigned OldInsnVarID = Rule.getInsnVarID(InsnMatcher);
Table << MatchTable::Opcode(Signed ? "GIR_CopyConstantAsSImm"
: "GIR_CopyConstantAsUImm")
<< MatchTable::Comment("NewInsnID") << MatchTable::IntValue(NewInsnID)
<< MatchTable::Comment("OldInsnID")
<< MatchTable::IntValue(OldInsnVarID)
<< MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
}
};
/// A CopyFConstantAsFPImmRenderer emits code to render a G_FCONSTANT
/// instruction to an extended immediate operand.
class CopyFConstantAsFPImmRenderer : public OperandRenderer {
protected:
unsigned NewInsnID;
/// The name of the operand.
const std::string SymbolicName;
public:
CopyFConstantAsFPImmRenderer(unsigned NewInsnID, StringRef SymbolicName)
: OperandRenderer(OR_CopyFConstantAsFPImm), NewInsnID(NewInsnID),
SymbolicName(SymbolicName) {}
static bool classof(const OperandRenderer *R) {
return R->getKind() == OR_CopyFConstantAsFPImm;
}
const StringRef getSymbolicName() const { return SymbolicName; }
void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
InstructionMatcher &InsnMatcher = Rule.getInstructionMatcher(SymbolicName);
unsigned OldInsnVarID = Rule.getInsnVarID(InsnMatcher);
Table << MatchTable::Opcode("GIR_CopyFConstantAsFPImm")
<< MatchTable::Comment("NewInsnID") << MatchTable::IntValue(NewInsnID)
<< MatchTable::Comment("OldInsnID")
<< MatchTable::IntValue(OldInsnVarID)
<< MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
}
};
/// A CopySubRegRenderer emits code to copy a single register operand from an
/// existing instruction to the one being built and indicate that only a
/// subregister should be copied.
class CopySubRegRenderer : public OperandRenderer {
protected:
unsigned NewInsnID;
/// The name of the operand.
const StringRef SymbolicName;
/// The subregister to extract.
const CodeGenSubRegIndex *SubReg;
public:
CopySubRegRenderer(unsigned NewInsnID, StringRef SymbolicName,
const CodeGenSubRegIndex *SubReg)
: OperandRenderer(OR_CopySubReg), NewInsnID(NewInsnID),
SymbolicName(SymbolicName), SubReg(SubReg) {}
static bool classof(const OperandRenderer *R) {
return R->getKind() == OR_CopySubReg;
}
const StringRef getSymbolicName() const { return SymbolicName; }
void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
const OperandMatcher &Operand = Rule.getOperandMatcher(SymbolicName);
unsigned OldInsnVarID = Rule.getInsnVarID(Operand.getInstructionMatcher());
Table << MatchTable::Opcode("GIR_CopySubReg")
<< MatchTable::Comment("NewInsnID") << MatchTable::IntValue(NewInsnID)
<< MatchTable::Comment("OldInsnID")
<< MatchTable::IntValue(OldInsnVarID) << MatchTable::Comment("OpIdx")
<< MatchTable::IntValue(Operand.getOpIdx())
<< MatchTable::Comment("SubRegIdx")
<< MatchTable::IntValue(SubReg->EnumValue)
<< MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
}
};
/// Adds a specific physical register to the instruction being built.
/// This is typically useful for WZR/XZR on AArch64.
class AddRegisterRenderer : public OperandRenderer {
protected:
unsigned InsnID;
const Record *RegisterDef;
public:
AddRegisterRenderer(unsigned InsnID, const Record *RegisterDef)
: OperandRenderer(OR_Register), InsnID(InsnID), RegisterDef(RegisterDef) {
}
static bool classof(const OperandRenderer *R) {
return R->getKind() == OR_Register;
}
void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIR_AddRegister")
<< MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
<< MatchTable::NamedValue(
(RegisterDef->getValue("Namespace")
? RegisterDef->getValueAsString("Namespace")
: ""),
RegisterDef->getName())
<< MatchTable::LineBreak;
}
};
/// Adds a specific temporary virtual register to the instruction being built.
/// This is used to chain instructions together when emitting multiple
/// instructions.
class TempRegRenderer : public OperandRenderer {
protected:
unsigned InsnID;
unsigned TempRegID;
bool IsDef;
public:
TempRegRenderer(unsigned InsnID, unsigned TempRegID, bool IsDef = false)
: OperandRenderer(OR_Register), InsnID(InsnID), TempRegID(TempRegID),
IsDef(IsDef) {}
static bool classof(const OperandRenderer *R) {
return R->getKind() == OR_TempRegister;
}
void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIR_AddTempRegister")
<< MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
<< MatchTable::Comment("TempRegID") << MatchTable::IntValue(TempRegID)
<< MatchTable::Comment("TempRegFlags");
if (IsDef)
Table << MatchTable::NamedValue("RegState::Define");
else
Table << MatchTable::IntValue(0);
Table << MatchTable::LineBreak;
}
};
/// Adds a specific immediate to the instruction being built.
class ImmRenderer : public OperandRenderer {
protected:
unsigned InsnID;
int64_t Imm;
public:
ImmRenderer(unsigned InsnID, int64_t Imm)
: OperandRenderer(OR_Imm), InsnID(InsnID), Imm(Imm) {}
static bool classof(const OperandRenderer *R) {
return R->getKind() == OR_Imm;
}
void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIR_AddImm") << MatchTable::Comment("InsnID")
<< MatchTable::IntValue(InsnID) << MatchTable::Comment("Imm")
<< MatchTable::IntValue(Imm) << MatchTable::LineBreak;
}
};
/// Adds operands by calling a renderer function supplied by the ComplexPattern
/// matcher function.
class RenderComplexPatternOperand : public OperandRenderer {
private:
unsigned InsnID;
const Record &TheDef;
/// The name of the operand.
const StringRef SymbolicName;
/// The renderer number. This must be unique within a rule since it's used to
/// identify a temporary variable to hold the renderer function.
unsigned RendererID;
/// When provided, this is the suboperand of the ComplexPattern operand to
/// render. Otherwise all the suboperands will be rendered.
Optional<unsigned> SubOperand;
unsigned getNumOperands() const {
return TheDef.getValueAsDag("Operands")->getNumArgs();
}
public:
RenderComplexPatternOperand(unsigned InsnID, const Record &TheDef,
StringRef SymbolicName, unsigned RendererID,
Optional<unsigned> SubOperand = None)
: OperandRenderer(OR_ComplexPattern), InsnID(InsnID), TheDef(TheDef),
SymbolicName(SymbolicName), RendererID(RendererID),
SubOperand(SubOperand) {}
static bool classof(const OperandRenderer *R) {
return R->getKind() == OR_ComplexPattern;
}
void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
Table << MatchTable::Opcode(SubOperand.hasValue() ? "GIR_ComplexSubOperandRenderer"
: "GIR_ComplexRenderer")
<< MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
<< MatchTable::Comment("RendererID")
<< MatchTable::IntValue(RendererID);
if (SubOperand.hasValue())
Table << MatchTable::Comment("SubOperand")
<< MatchTable::IntValue(SubOperand.getValue());
Table << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
}
};
class CustomRenderer : public OperandRenderer {
protected:
unsigned InsnID;
const Record &Renderer;
/// The name of the operand.
const std::string SymbolicName;
public:
CustomRenderer(unsigned InsnID, const Record &Renderer,
StringRef SymbolicName)
: OperandRenderer(OR_Custom), InsnID(InsnID), Renderer(Renderer),
SymbolicName(SymbolicName) {}
static bool classof(const OperandRenderer *R) {
return R->getKind() == OR_Custom;
}
void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
InstructionMatcher &InsnMatcher = Rule.getInstructionMatcher(SymbolicName);
unsigned OldInsnVarID = Rule.getInsnVarID(InsnMatcher);
Table << MatchTable::Opcode("GIR_CustomRenderer")
<< MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
<< MatchTable::Comment("OldInsnID")
<< MatchTable::IntValue(OldInsnVarID)
<< MatchTable::Comment("Renderer")
<< MatchTable::NamedValue(
"GICR_" + Renderer.getValueAsString("RendererFn").str())
<< MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
}
};
/// An action taken when all Matcher predicates succeeded for a parent rule.
///
/// Typical actions include:
/// * Changing the opcode of an instruction.
/// * Adding an operand to an instruction.
class MatchAction {
public:
virtual ~MatchAction() {}
/// Emit the MatchTable opcodes to implement the action.
virtual void emitActionOpcodes(MatchTable &Table,
RuleMatcher &Rule) const = 0;
};
/// Generates a comment describing the matched rule being acted upon.
class DebugCommentAction : public MatchAction {
private:
std::string S;
public:
DebugCommentAction(StringRef S) : S(S) {}
void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
Table << MatchTable::Comment(S) << MatchTable::LineBreak;
}
};
/// Generates code to build an instruction or mutate an existing instruction
/// into the desired instruction when this is possible.
class BuildMIAction : public MatchAction {
private:
unsigned InsnID;
const CodeGenInstruction *I;
InstructionMatcher *Matched;
std::vector<std::unique_ptr<OperandRenderer>> OperandRenderers;
/// True if the instruction can be built solely by mutating the opcode.
bool canMutate(RuleMatcher &Rule, const InstructionMatcher *Insn) const {
if (!Insn)
return false;
if (OperandRenderers.size() != Insn->getNumOperands())
return false;
for (const auto &Renderer : enumerate(OperandRenderers)) {
if (const auto *Copy = dyn_cast<CopyRenderer>(&*Renderer.value())) {
const OperandMatcher &OM = Rule.getOperandMatcher(Copy->getSymbolicName());
if (Insn != &OM.getInstructionMatcher() ||
OM.getOpIdx() != Renderer.index())
return false;
} else
return false;
}
return true;
}
public:
BuildMIAction(unsigned InsnID, const CodeGenInstruction *I)
: InsnID(InsnID), I(I), Matched(nullptr) {}
unsigned getInsnID() const { return InsnID; }
const CodeGenInstruction *getCGI() const { return I; }
void chooseInsnToMutate(RuleMatcher &Rule) {
for (auto *MutateCandidate : Rule.mutatable_insns()) {
if (canMutate(Rule, MutateCandidate)) {
// Take the first one we're offered that we're able to mutate.
Rule.reserveInsnMatcherForMutation(MutateCandidate);
Matched = MutateCandidate;
return;
}
}
}
template <class Kind, class... Args>
Kind &addRenderer(Args&&... args) {
OperandRenderers.emplace_back(
llvm::make_unique<Kind>(InsnID, std::forward<Args>(args)...));
return *static_cast<Kind *>(OperandRenderers.back().get());
}
void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
if (Matched) {
assert(canMutate(Rule, Matched) &&
"Arranged to mutate an insn that isn't mutatable");
unsigned RecycleInsnID = Rule.getInsnVarID(*Matched);
Table << MatchTable::Opcode("GIR_MutateOpcode")
<< MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
<< MatchTable::Comment("RecycleInsnID")
<< MatchTable::IntValue(RecycleInsnID)
<< MatchTable::Comment("Opcode")
<< MatchTable::NamedValue(I->Namespace, I->TheDef->getName())
<< MatchTable::LineBreak;
if (!I->ImplicitDefs.empty() || !I->ImplicitUses.empty()) {
for (auto Def : I->ImplicitDefs) {
auto Namespace = Def->getValue("Namespace")
? Def->getValueAsString("Namespace")
: "";
Table << MatchTable::Opcode("GIR_AddImplicitDef")
<< MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
<< MatchTable::NamedValue(Namespace, Def->getName())
<< MatchTable::LineBreak;
}
for (auto Use : I->ImplicitUses) {
auto Namespace = Use->getValue("Namespace")
? Use->getValueAsString("Namespace")
: "";
Table << MatchTable::Opcode("GIR_AddImplicitUse")
<< MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
<< MatchTable::NamedValue(Namespace, Use->getName())
<< MatchTable::LineBreak;
}
}
return;
}
// TODO: Simple permutation looks like it could be almost as common as
// mutation due to commutative operations.
Table << MatchTable::Opcode("GIR_BuildMI") << MatchTable::Comment("InsnID")
<< MatchTable::IntValue(InsnID) << MatchTable::Comment("Opcode")
<< MatchTable::NamedValue(I->Namespace, I->TheDef->getName())
<< MatchTable::LineBreak;
for (const auto &Renderer : OperandRenderers)
Renderer->emitRenderOpcodes(Table, Rule);
if (I->mayLoad || I->mayStore) {
Table << MatchTable::Opcode("GIR_MergeMemOperands")
<< MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
<< MatchTable::Comment("MergeInsnID's");
// Emit the ID's for all the instructions that are matched by this rule.
// TODO: Limit this to matched instructions that mayLoad/mayStore or have
// some other means of having a memoperand. Also limit this to
// emitted instructions that expect to have a memoperand too. For
// example, (G_SEXT (G_LOAD x)) that results in separate load and
// sign-extend instructions shouldn't put the memoperand on the
// sign-extend since it has no effect there.
std::vector<unsigned> MergeInsnIDs;
for (const auto &IDMatcherPair : Rule.defined_insn_vars())
MergeInsnIDs.push_back(IDMatcherPair.second);
llvm::sort(MergeInsnIDs.begin(), MergeInsnIDs.end());
for (const auto &MergeInsnID : MergeInsnIDs)
Table << MatchTable::IntValue(MergeInsnID);
Table << MatchTable::NamedValue("GIU_MergeMemOperands_EndOfList")
<< MatchTable::LineBreak;
}
// FIXME: This is a hack but it's sufficient for ISel. We'll need to do
// better for combines. Particularly when there are multiple match
// roots.
if (InsnID == 0)
Table << MatchTable::Opcode("GIR_EraseFromParent")
<< MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
<< MatchTable::LineBreak;
}
};
/// Generates code to constrain the operands of an output instruction to the
/// register classes specified by the definition of that instruction.
class ConstrainOperandsToDefinitionAction : public MatchAction {
unsigned InsnID;
public:
ConstrainOperandsToDefinitionAction(unsigned InsnID) : InsnID(InsnID) {}
void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIR_ConstrainSelectedInstOperands")
<< MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
<< MatchTable::LineBreak;
}
};
/// Generates code to constrain the specified operand of an output instruction
/// to the specified register class.
class ConstrainOperandToRegClassAction : public MatchAction {
unsigned InsnID;
unsigned OpIdx;
const CodeGenRegisterClass &RC;
public:
ConstrainOperandToRegClassAction(unsigned InsnID, unsigned OpIdx,
const CodeGenRegisterClass &RC)
: InsnID(InsnID), OpIdx(OpIdx), RC(RC) {}
void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIR_ConstrainOperandRC")
<< MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
<< MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
<< MatchTable::Comment("RC " + RC.getName())
<< MatchTable::IntValue(RC.EnumValue) << MatchTable::LineBreak;
}
};
/// Generates code to create a temporary register which can be used to chain
/// instructions together.
class MakeTempRegisterAction : public MatchAction {
private:
LLTCodeGen Ty;
unsigned TempRegID;
public:
MakeTempRegisterAction(const LLTCodeGen &Ty, unsigned TempRegID)
: Ty(Ty), TempRegID(TempRegID) {}
void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
Table << MatchTable::Opcode("GIR_MakeTempReg")
<< MatchTable::Comment("TempRegID") << MatchTable::IntValue(TempRegID)
<< MatchTable::Comment("TypeID")
<< MatchTable::NamedValue(Ty.getCxxEnumValue())
<< MatchTable::LineBreak;
}
};
InstructionMatcher &RuleMatcher::addInstructionMatcher(StringRef SymbolicName) {
Matchers.emplace_back(new InstructionMatcher(*this, SymbolicName));
MutatableInsns.insert(Matchers.back().get());
return *Matchers.back();
}
void RuleMatcher::addRequiredFeature(Record *Feature) {
RequiredFeatures.push_back(Feature);
}
const std::vector<Record *> &RuleMatcher::getRequiredFeatures() const {
return RequiredFeatures;
}
// Emplaces an action of the specified Kind at the end of the action list.
//
// Returns a reference to the newly created action.
//
// Like std::vector::emplace_back(), may invalidate all iterators if the new
// size exceeds the capacity. Otherwise, only invalidates the past-the-end
// iterator.
template <class Kind, class... Args>
Kind &RuleMatcher::addAction(Args &&... args) {
Actions.emplace_back(llvm::make_unique<Kind>(std::forward<Args>(args)...));
return *static_cast<Kind *>(Actions.back().get());
}
// Emplaces an action of the specified Kind before the given insertion point.
//
// Returns an iterator pointing at the newly created instruction.
//
// Like std::vector::insert(), may invalidate all iterators if the new size
// exceeds the capacity. Otherwise, only invalidates the iterators from the
// insertion point onwards.
template <class Kind, class... Args>
action_iterator RuleMatcher::insertAction(action_iterator InsertPt,
Args &&... args) {
return Actions.emplace(InsertPt,
llvm::make_unique<Kind>(std::forward<Args>(args)...));
}
unsigned RuleMatcher::implicitlyDefineInsnVar(InstructionMatcher &Matcher) {
unsigned NewInsnVarID = NextInsnVarID++;
InsnVariableIDs[&Matcher] = NewInsnVarID;
return NewInsnVarID;
}
unsigned RuleMatcher::getInsnVarID(InstructionMatcher &InsnMatcher) const {
const auto &I = InsnVariableIDs.find(&InsnMatcher);
if (I != InsnVariableIDs.end())
return I->second;
llvm_unreachable("Matched Insn was not captured in a local variable");
}
void RuleMatcher::defineOperand(StringRef SymbolicName, OperandMatcher &OM) {
if (DefinedOperands.find(SymbolicName) == DefinedOperands.end()) {
DefinedOperands[SymbolicName] = &OM;
return;
}
// If the operand is already defined, then we must ensure both references in
// the matcher have the exact same node.
OM.addPredicate<SameOperandMatcher>(OM.getSymbolicName());
}
InstructionMatcher &
RuleMatcher::getInstructionMatcher(StringRef SymbolicName) const {
for (const auto &I : InsnVariableIDs)
if (I.first->getSymbolicName() == SymbolicName)
return *I.first;
llvm_unreachable(
("Failed to lookup instruction " + SymbolicName).str().c_str());
}
const OperandMatcher &
RuleMatcher::getOperandMatcher(StringRef Name) const {
const auto &I = DefinedOperands.find(Name);
if (I == DefinedOperands.end())
PrintFatalError(SrcLoc, "Operand " + Name + " was not declared in matcher");
return *I->second;
}
void RuleMatcher::emit(MatchTable &Table) {
if (Matchers.empty())
llvm_unreachable("Unexpected empty matcher!");
// The representation supports rules that require multiple roots such as:
// %ptr(p0) = ...
// %elt0(s32) = G_LOAD %ptr
// %1(p0) = G_ADD %ptr, 4
// %elt1(s32) = G_LOAD p0 %1
// which could be usefully folded into:
// %ptr(p0) = ...
// %elt0(s32), %elt1(s32) = TGT_LOAD_PAIR %ptr
// on some targets but we don't need to make use of that yet.
assert(Matchers.size() == 1 && "Cannot handle multi-root matchers yet");
unsigned LabelID = Table.allocateLabelID();
Table << MatchTable::Opcode("GIM_Try", +1)
<< MatchTable::Comment("On fail goto")
<< MatchTable::JumpTarget(LabelID)
<< MatchTable::Comment(("Rule ID " + Twine(RuleID) + " //").str())
<< MatchTable::LineBreak;
if (!RequiredFeatures.empty()) {
Table << MatchTable::Opcode("GIM_CheckFeatures")
<< MatchTable::NamedValue(getNameForFeatureBitset(RequiredFeatures))
<< MatchTable::LineBreak;
}
Matchers.front()->emitPredicateOpcodes(Table, *this);
// We must also check if it's safe to fold the matched instructions.
if (InsnVariableIDs.size() >= 2) {
// Invert the map to create stable ordering (by var names)
SmallVector<unsigned, 2> InsnIDs;
for (const auto &Pair : InsnVariableIDs) {
// Skip the root node since it isn't moving anywhere. Everything else is
// sinking to meet it.
if (Pair.first == Matchers.front().get())
continue;
InsnIDs.push_back(Pair.second);
}
llvm::sort(InsnIDs.begin(), InsnIDs.end());
for (const auto &InsnID : InsnIDs) {
// Reject the difficult cases until we have a more accurate check.
Table << MatchTable::Opcode("GIM_CheckIsSafeToFold")
<< MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
<< MatchTable::LineBreak;
// FIXME: Emit checks to determine it's _actually_ safe to fold and/or
// account for unsafe cases.
//
// Example:
// MI1--> %0 = ...
// %1 = ... %0
// MI0--> %2 = ... %0
// It's not safe to erase MI1. We currently handle this by not
// erasing %0 (even when it's dead).
//
// Example:
// MI1--> %0 = load volatile @a
// %1 = load volatile @a
// MI0--> %2 = ... %0
// It's not safe to sink %0's def past %1. We currently handle
// this by rejecting all loads.
//
// Example:
// MI1--> %0 = load @a
// %1 = store @a
// MI0--> %2 = ... %0
// It's not safe to sink %0's def past %1. We currently handle
// this by rejecting all loads.
//
// Example:
// G_CONDBR %cond, @BB1
// BB0:
// MI1--> %0 = load @a
// G_BR @BB1
// BB1:
// MI0--> %2 = ... %0
// It's not always safe to sink %0 across control flow. In this
// case it may introduce a memory fault. We currentl handle this
// by rejecting all loads.
}
}
for (const auto &PM : EpilogueMatchers)
PM->emitPredicateOpcodes(Table, *this);
for (const auto &MA : Actions)
MA->emitActionOpcodes(Table, *this);
if (Table.isWithCoverage())
Table << MatchTable::Opcode("GIR_Coverage") << MatchTable::IntValue(RuleID)
<< MatchTable::LineBreak;
else
Table << MatchTable::Comment(("GIR_Coverage, " + Twine(RuleID) + ",").str())
<< MatchTable::LineBreak;
Table << MatchTable::Opcode("GIR_Done", -1) << MatchTable::LineBreak
<< MatchTable::Label(LabelID);
++NumPatternEmitted;
}
bool RuleMatcher::isHigherPriorityThan(const RuleMatcher &B) const {
// Rules involving more match roots have higher priority.
if (Matchers.size() > B.Matchers.size())
return true;
if (Matchers.size() < B.Matchers.size())
return false;
for (const auto &Matcher : zip(Matchers, B.Matchers)) {
if (std::get<0>(Matcher)->isHigherPriorityThan(*std::get<1>(Matcher)))
return true;
if (std::get<1>(Matcher)->isHigherPriorityThan(*std::get<0>(Matcher)))
return false;
}
return false;
}
unsigned RuleMatcher::countRendererFns() const {
return std::accumulate(
Matchers.begin(), Matchers.end(), 0,
[](unsigned A, const std::unique_ptr<InstructionMatcher> &Matcher) {
return A + Matcher->countRendererFns();
});
}
bool OperandPredicateMatcher::isHigherPriorityThan(
const OperandPredicateMatcher &B) const {
// Generally speaking, an instruction is more important than an Int or a
// LiteralInt because it can cover more nodes but theres an exception to
// this. G_CONSTANT's are less important than either of those two because they
// are more permissive.
const InstructionOperandMatcher *AOM =
dyn_cast<InstructionOperandMatcher>(this);
const InstructionOperandMatcher *BOM =
dyn_cast<InstructionOperandMatcher>(&B);
bool AIsConstantInsn = AOM && AOM->getInsnMatcher().isConstantInstruction();
bool BIsConstantInsn = BOM && BOM->getInsnMatcher().isConstantInstruction();
if (AOM && BOM) {
// The relative priorities between a G_CONSTANT and any other instruction
// don't actually matter but this code is needed to ensure a strict weak
// ordering. This is particularly important on Windows where the rules will
// be incorrectly sorted without it.
if (AIsConstantInsn != BIsConstantInsn)
return AIsConstantInsn < BIsConstantInsn;
return false;
}
if (AOM && AIsConstantInsn && (B.Kind == OPM_Int || B.Kind == OPM_LiteralInt))
return false;
if (BOM && BIsConstantInsn && (Kind == OPM_Int || Kind == OPM_LiteralInt))
return true;
return Kind < B.Kind;
}
void SameOperandMatcher::emitPredicateOpcodes(MatchTable &Table,
RuleMatcher &Rule) const {
const OperandMatcher &OtherOM = Rule.getOperandMatcher(MatchingName);
unsigned OtherInsnVarID = Rule.getInsnVarID(OtherOM.getInstructionMatcher());
assert(OtherInsnVarID == OtherOM.getInstructionMatcher().getInsnVarID());
Table << MatchTable::Opcode("GIM_CheckIsSameOperand")
<< MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
<< MatchTable::Comment("OpIdx") << MatchTable::IntValue(OpIdx)
<< MatchTable::Comment("OtherMI")
<< MatchTable::IntValue(OtherInsnVarID)
<< MatchTable::Comment("OtherOpIdx")
<< MatchTable::IntValue(OtherOM.getOpIdx())
<< MatchTable::LineBreak;
}
//===- GlobalISelEmitter class --------------------------------------------===//
class GlobalISelEmitter {
public:
explicit GlobalISelEmitter(RecordKeeper &RK);
void run(raw_ostream &OS);
private:
const RecordKeeper &RK;
const CodeGenDAGPatterns CGP;
const CodeGenTarget &Target;
CodeGenRegBank CGRegs;
/// Keep track of the equivalence between SDNodes and Instruction by mapping
/// SDNodes to the GINodeEquiv mapping. We need to map to the GINodeEquiv to
/// check for attributes on the relation such as CheckMMOIsNonAtomic.
/// This is defined using 'GINodeEquiv' in the target description.
DenseMap<Record *, Record *> NodeEquivs;
/// Keep track of the equivalence between ComplexPattern's and
/// GIComplexOperandMatcher. Map entries are specified by subclassing
/// GIComplexPatternEquiv.
DenseMap<const Record *, const Record *> ComplexPatternEquivs;
/// Keep track of the equivalence between SDNodeXForm's and
/// GICustomOperandRenderer. Map entries are specified by subclassing
/// GISDNodeXFormEquiv.
DenseMap<const Record *, const Record *> SDNodeXFormEquivs;
/// Keep track of Scores of PatternsToMatch similar to how the DAG does.
/// This adds compatibility for RuleMatchers to use this for ordering rules.
DenseMap<uint64_t, int> RuleMatcherScores;
// Map of predicates to their subtarget features.
SubtargetFeatureInfoMap SubtargetFeatures;
// Rule coverage information.
Optional<CodeGenCoverage> RuleCoverage;
void gatherOpcodeValues();
void gatherTypeIDValues();
void gatherNodeEquivs();
Record *findNodeEquiv(Record *N) const;
const CodeGenInstruction *getEquivNode(Record &Equiv,
const TreePatternNode *N) const;
Error importRulePredicates(RuleMatcher &M, ArrayRef<Predicate> Predicates);
Expected<InstructionMatcher &> createAndImportSelDAGMatcher(
RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
const TreePatternNode *Src, unsigned &TempOpIdx) const;
Error importComplexPatternOperandMatcher(OperandMatcher &OM, Record *R,
unsigned &TempOpIdx) const;
Error importChildMatcher(RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
const TreePatternNode *SrcChild,
bool OperandIsAPointer, unsigned OpIdx,
unsigned &TempOpIdx) const;
Expected<BuildMIAction &>
createAndImportInstructionRenderer(RuleMatcher &M,
const TreePatternNode *Dst);
Expected<action_iterator> createAndImportSubInstructionRenderer(
action_iterator InsertPt, RuleMatcher &M, const TreePatternNode *Dst,
unsigned TempReg);
Expected<action_iterator>
createInstructionRenderer(action_iterator InsertPt, RuleMatcher &M,
const TreePatternNode *Dst);
void importExplicitDefRenderers(BuildMIAction &DstMIBuilder);
Expected<action_iterator>
importExplicitUseRenderers(action_iterator InsertPt, RuleMatcher &M,
BuildMIAction &DstMIBuilder,
const llvm::TreePatternNode *Dst);
Expected<action_iterator>
importExplicitUseRenderer(action_iterator InsertPt, RuleMatcher &Rule,
BuildMIAction &DstMIBuilder,
TreePatternNode *DstChild);
Error importDefaultOperandRenderers(BuildMIAction &DstMIBuilder,
DagInit *DefaultOps) const;
Error
importImplicitDefRenderers(BuildMIAction &DstMIBuilder,
const std::vector<Record *> &ImplicitDefs) const;
void emitImmPredicates(raw_ostream &OS, StringRef TypeIdentifier,
StringRef Type,
std::function<bool(const Record *R)> Filter);
/// Analyze pattern \p P, returning a matcher for it if possible.
/// Otherwise, return an Error explaining why we don't support it.
Expected<RuleMatcher> runOnPattern(const PatternToMatch &P);
void declareSubtargetFeature(Record *Predicate);
MatchTable buildMatchTable(MutableArrayRef<RuleMatcher> Rules, bool Optimize,
bool WithCoverage);
public:
/// Takes a sequence of \p Rules and group them based on the predicates
/// they share. \p MatcherStorage is used as a memory container
/// for the group that are created as part of this process.
///
/// What this optimization does looks like if GroupT = GroupMatcher:
/// Output without optimization:
/// \verbatim
/// # R1
/// # predicate A
/// # predicate B
/// ...
/// # R2
/// # predicate A // <-- effectively this is going to be checked twice.
/// // Once in R1 and once in R2.
/// # predicate C
/// \endverbatim
/// Output with optimization:
/// \verbatim
/// # Group1_2
/// # predicate A // <-- Check is now shared.
/// # R1
/// # predicate B
/// # R2
/// # predicate C
/// \endverbatim
template <class GroupT>
static std::vector<Matcher *> optimizeRules(
ArrayRef<Matcher *> Rules,
std::vector<std::unique_ptr<Matcher>> &MatcherStorage);
};
void GlobalISelEmitter::gatherOpcodeValues() {
InstructionOpcodeMatcher::initOpcodeValuesMap(Target);
}
void GlobalISelEmitter::gatherTypeIDValues() {
LLTOperandMatcher::initTypeIDValuesMap();
}
void GlobalISelEmitter::gatherNodeEquivs() {
assert(NodeEquivs.empty());
for (Record *Equiv : RK.getAllDerivedDefinitions("GINodeEquiv"))
NodeEquivs[Equiv->getValueAsDef("Node")] = Equiv;
assert(ComplexPatternEquivs.empty());
for (Record *Equiv : RK.getAllDerivedDefinitions("GIComplexPatternEquiv")) {
Record *SelDAGEquiv = Equiv->getValueAsDef("SelDAGEquivalent");
if (!SelDAGEquiv)
continue;
ComplexPatternEquivs[SelDAGEquiv] = Equiv;
}
assert(SDNodeXFormEquivs.empty());
for (Record *Equiv : RK.getAllDerivedDefinitions("GISDNodeXFormEquiv")) {
Record *SelDAGEquiv = Equiv->getValueAsDef("SelDAGEquivalent");
if (!SelDAGEquiv)
continue;
SDNodeXFormEquivs[SelDAGEquiv] = Equiv;
}
}
Record *GlobalISelEmitter::findNodeEquiv(Record *N) const {
return NodeEquivs.lookup(N);
}
const CodeGenInstruction *
GlobalISelEmitter::getEquivNode(Record &Equiv, const TreePatternNode *N) const {
for (const auto &Predicate : N->getPredicateFns()) {
if (!Equiv.isValueUnset("IfSignExtend") && Predicate.isLoad() &&
Predicate.isSignExtLoad())
return &Target.getInstruction(Equiv.getValueAsDef("IfSignExtend"));
if (!Equiv.isValueUnset("IfZeroExtend") && Predicate.isLoad() &&
Predicate.isZeroExtLoad())
return &Target.getInstruction(Equiv.getValueAsDef("IfZeroExtend"));
}
return &Target.getInstruction(Equiv.getValueAsDef("I"));
}
GlobalISelEmitter::GlobalISelEmitter(RecordKeeper &RK)
: RK(RK), CGP(RK), Target(CGP.getTargetInfo()),
CGRegs(RK, Target.getHwModes()) {}
//===- Emitter ------------------------------------------------------------===//
Error
GlobalISelEmitter::importRulePredicates(RuleMatcher &M,
ArrayRef<Predicate> Predicates) {
for (const Predicate &P : Predicates) {
if (!P.Def)
continue;
declareSubtargetFeature(P.Def);
M.addRequiredFeature(P.Def);
}
return Error::success();
}
Expected<InstructionMatcher &> GlobalISelEmitter::createAndImportSelDAGMatcher(
RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
const TreePatternNode *Src, unsigned &TempOpIdx) const {
Record *SrcGIEquivOrNull = nullptr;
const CodeGenInstruction *SrcGIOrNull = nullptr;
// Start with the defined operands (i.e., the results of the root operator).
if (Src->getExtTypes().size() > 1)
return failedImport("Src pattern has multiple results");
if (Src->isLeaf()) {
Init *SrcInit = Src->getLeafValue();
if (isa<IntInit>(SrcInit)) {
InsnMatcher.addPredicate<InstructionOpcodeMatcher>(
&Target.getInstruction(RK.getDef("G_CONSTANT")));
} else
return failedImport(
"Unable to deduce gMIR opcode to handle Src (which is a leaf)");
} else {
SrcGIEquivOrNull = findNodeEquiv(Src->getOperator());
if (!SrcGIEquivOrNull)
return failedImport("Pattern operator lacks an equivalent Instruction" +
explainOperator(Src->getOperator()));
SrcGIOrNull = getEquivNode(*SrcGIEquivOrNull, Src);
// The operators look good: match the opcode
InsnMatcher.addPredicate<InstructionOpcodeMatcher>(SrcGIOrNull);
}
unsigned OpIdx = 0;
for (const TypeSetByHwMode &VTy : Src->getExtTypes()) {
// Results don't have a name unless they are the root node. The caller will
// set the name if appropriate.
OperandMatcher &OM = InsnMatcher.addOperand(OpIdx++, "", TempOpIdx);
if (auto Error = OM.addTypeCheckPredicate(VTy, false /* OperandIsAPointer */))
return failedImport(toString(std::move(Error)) +
" for result of Src pattern operator");
}
for (const auto &Predicate : Src->getPredicateFns()) {
if (Predicate.isAlwaysTrue())
continue;
if (Predicate.isImmediatePattern()) {
InsnMatcher.addPredicate<InstructionImmPredicateMatcher>(Predicate);
continue;
}
// G_LOAD is used for both non-extending and any-extending loads.
if (Predicate.isLoad() && Predicate.isNonExtLoad()) {
InsnMatcher.addPredicate<MemoryVsLLTSizePredicateMatcher>(
0, MemoryVsLLTSizePredicateMatcher::EqualTo, 0);
continue;
}
if (Predicate.isLoad() && Predicate.isAnyExtLoad()) {
InsnMatcher.addPredicate<MemoryVsLLTSizePredicateMatcher>(
0, MemoryVsLLTSizePredicateMatcher::LessThan, 0);
continue;
}
// No check required. We already did it by swapping the opcode.
if (!SrcGIEquivOrNull->isValueUnset("IfSignExtend") &&
Predicate.isSignExtLoad())
continue;
// No check required. We already did it by swapping the opcode.
if (!SrcGIEquivOrNull->isValueUnset("IfZeroExtend") &&
Predicate.isZeroExtLoad())
continue;
// No check required. G_STORE by itself is a non-extending store.
if (Predicate.isNonTruncStore())
continue;
if (Predicate.isLoad() || Predicate.isStore() || Predicate.isAtomic()) {
if (Predicate.getMemoryVT() != nullptr) {
Optional<LLTCodeGen> MemTyOrNone =
MVTToLLT(getValueType(Predicate.getMemoryVT()));
if (!MemTyOrNone)
return failedImport("MemVT could not be converted to LLT");
// MMO's work in bytes so we must take care of unusual types like i1
// don't round down.
unsigned MemSizeInBits =
llvm::alignTo(MemTyOrNone->get().getSizeInBits(), 8);
InsnMatcher.addPredicate<MemorySizePredicateMatcher>(
0, MemSizeInBits / 8);
continue;
}
}
if (Predicate.isLoad() || Predicate.isStore()) {
// No check required. A G_LOAD/G_STORE is an unindexed load.
if (Predicate.isUnindexed())
continue;
}
if (Predicate.isAtomic()) {
if (Predicate.isAtomicOrderingMonotonic()) {
InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
"Monotonic");
continue;
}
if (Predicate.isAtomicOrderingAcquire()) {
InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>("Acquire");
continue;
}
if (Predicate.isAtomicOrderingRelease()) {
InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>("Release");
continue;
}
if (Predicate.isAtomicOrderingAcquireRelease()) {
InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
"AcquireRelease");
continue;
}
if (Predicate.isAtomicOrderingSequentiallyConsistent()) {
InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
"SequentiallyConsistent");
continue;
}
if (Predicate.isAtomicOrderingAcquireOrStronger()) {
InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
"Acquire", AtomicOrderingMMOPredicateMatcher::AO_OrStronger);
continue;
}
if (Predicate.isAtomicOrderingWeakerThanAcquire()) {
InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
"Acquire", AtomicOrderingMMOPredicateMatcher::AO_WeakerThan);
continue;
}
if (Predicate.isAtomicOrderingReleaseOrStronger()) {
InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
"Release", AtomicOrderingMMOPredicateMatcher::AO_OrStronger);
continue;
}
if (Predicate.isAtomicOrderingWeakerThanRelease()) {
InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
"Release", AtomicOrderingMMOPredicateMatcher::AO_WeakerThan);
continue;
}
}
return failedImport("Src pattern child has predicate (" +
explainPredicates(Src) + ")");
}
if (SrcGIEquivOrNull && SrcGIEquivOrNull->getValueAsBit("CheckMMOIsNonAtomic"))
InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>("NotAtomic");
if (Src->isLeaf()) {
Init *SrcInit = Src->getLeafValue();
if (IntInit *SrcIntInit = dyn_cast<IntInit>(SrcInit)) {
OperandMatcher &OM =
InsnMatcher.addOperand(OpIdx++, Src->getName(), TempOpIdx);
OM.addPredicate<LiteralIntOperandMatcher>(SrcIntInit->getValue());
} else
return failedImport(
"Unable to deduce gMIR opcode to handle Src (which is a leaf)");
} else {
assert(SrcGIOrNull &&
"Expected to have already found an equivalent Instruction");
if (SrcGIOrNull->TheDef->getName() == "G_CONSTANT" ||
SrcGIOrNull->TheDef->getName() == "G_FCONSTANT") {
// imm/fpimm still have operands but we don't need to do anything with it
// here since we don't support ImmLeaf predicates yet. However, we still
// need to note the hidden operand to get GIM_CheckNumOperands correct.
InsnMatcher.addOperand(OpIdx++, "", TempOpIdx);
return InsnMatcher;
}
// Match the used operands (i.e. the children of the operator).
for (unsigned i = 0, e = Src->getNumChildren(); i != e; ++i) {
TreePatternNode *SrcChild = Src->getChild(i);
// SelectionDAG allows pointers to be represented with iN since it doesn't
// distinguish between pointers and integers but they are different types in GlobalISel.
// Coerce integers to pointers to address space 0 if the context indicates a pointer.
bool OperandIsAPointer = SrcGIOrNull->isOperandAPointer(i);
// For G_INTRINSIC/G_INTRINSIC_W_SIDE_EFFECTS, the operand immediately
// following the defs is an intrinsic ID.
if ((SrcGIOrNull->TheDef->getName() == "G_INTRINSIC" ||
SrcGIOrNull->TheDef->getName() == "G_INTRINSIC_W_SIDE_EFFECTS") &&
i == 0) {
if (const CodeGenIntrinsic *II = Src->getIntrinsicInfo(CGP)) {
OperandMatcher &OM =
InsnMatcher.addOperand(OpIdx++, SrcChild->getName(), TempOpIdx);
OM.addPredicate<IntrinsicIDOperandMatcher>(II);
continue;
}
return failedImport("Expected IntInit containing instrinsic ID)");
}
if (auto Error =
importChildMatcher(Rule, InsnMatcher, SrcChild, OperandIsAPointer,
OpIdx++, TempOpIdx))
return std::move(Error);
}
}
return InsnMatcher;
}
Error GlobalISelEmitter::importComplexPatternOperandMatcher(
OperandMatcher &OM, Record *R, unsigned &TempOpIdx) const {
const auto &ComplexPattern = ComplexPatternEquivs.find(R);
if (ComplexPattern == ComplexPatternEquivs.end())
return failedImport("SelectionDAG ComplexPattern (" + R->getName() +
") not mapped to GlobalISel");
OM.addPredicate<ComplexPatternOperandMatcher>(OM, *ComplexPattern->second);
TempOpIdx++;
return Error::success();
}
Error GlobalISelEmitter::importChildMatcher(RuleMatcher &Rule,
InstructionMatcher &InsnMatcher,
const TreePatternNode *SrcChild,
bool OperandIsAPointer,
unsigned OpIdx,
unsigned &TempOpIdx) const {
OperandMatcher &OM =
InsnMatcher.addOperand(OpIdx, SrcChild->getName(), TempOpIdx);
if (OM.isSameAsAnotherOperand())
return Error::success();
ArrayRef<TypeSetByHwMode> ChildTypes = SrcChild->getExtTypes();
if (ChildTypes.size() != 1)
return failedImport("Src pattern child has multiple results");
// Check MBB's before the type check since they are not a known type.
if (!SrcChild->isLeaf()) {
if (SrcChild->getOperator()->isSubClassOf("SDNode")) {
auto &ChildSDNI = CGP.getSDNodeInfo(SrcChild->getOperator());
if (ChildSDNI.getSDClassName() == "BasicBlockSDNode") {
OM.addPredicate<MBBOperandMatcher>();
return Error::success();
}
}
}
if (auto Error =
OM.addTypeCheckPredicate(ChildTypes.front(), OperandIsAPointer))
return failedImport(toString(std::move(Error)) + " for Src operand (" +
to_string(*SrcChild) + ")");
// Check for nested instructions.
if (!SrcChild->isLeaf()) {
if (SrcChild->getOperator()->isSubClassOf("ComplexPattern")) {
// When a ComplexPattern is used as an operator, it should do the same
// thing as when used as a leaf. However, the children of the operator
// name the sub-operands that make up the complex operand and we must
// prepare to reference them in the renderer too.
unsigned RendererID = TempOpIdx;
if (auto Error = importComplexPatternOperandMatcher(
OM, SrcChild->getOperator(), TempOpIdx))
return Error;
for (unsigned i = 0, e = SrcChild->getNumChildren(); i != e; ++i) {
auto *SubOperand = SrcChild->getChild(i);
if (!SubOperand->getName().empty())
Rule.defineComplexSubOperand(SubOperand->getName(),
SrcChild->getOperator(), RendererID, i);
}
return Error::success();
}
auto MaybeInsnOperand = OM.addPredicate<InstructionOperandMatcher>(
InsnMatcher.getRuleMatcher(), SrcChild->getName());
if (!MaybeInsnOperand.hasValue()) {
// This isn't strictly true. If the user were to provide exactly the same
// matchers as the original operand then we could allow it. However, it's
// simpler to not permit the redundant specification.
return failedImport("Nested instruction cannot be the same as another operand");
}
// Map the node to a gMIR instruction.
InstructionOperandMatcher &InsnOperand = **MaybeInsnOperand;
auto InsnMatcherOrError = createAndImportSelDAGMatcher(
Rule, InsnOperand.getInsnMatcher(), SrcChild, TempOpIdx);
if (auto Error = InsnMatcherOrError.takeError())
return Error;
return Error::success();
}
if (SrcChild->hasAnyPredicate())
return failedImport("Src pattern child has unsupported predicate");
// Check for constant immediates.
if (auto *ChildInt = dyn_cast<IntInit>(SrcChild->getLeafValue())) {
OM.addPredicate<ConstantIntOperandMatcher>(ChildInt->getValue());
return Error::success();
}
// Check for def's like register classes or ComplexPattern's.
if (auto *ChildDefInit = dyn_cast<DefInit>(SrcChild->getLeafValue())) {
auto *ChildRec = ChildDefInit->getDef();
// Check for register classes.
if (ChildRec->isSubClassOf("RegisterClass") ||
ChildRec->isSubClassOf("RegisterOperand")) {
OM.addPredicate<RegisterBankOperandMatcher>(
Target.getRegisterClass(getInitValueAsRegClass(ChildDefInit)));
return Error::success();
}
// Check for ValueType.
if (ChildRec->isSubClassOf("ValueType")) {
// We already added a type check as standard practice so this doesn't need
// to do anything.
return Error::success();
}
// Check for ComplexPattern's.
if (ChildRec->isSubClassOf("ComplexPattern"))
return importComplexPatternOperandMatcher(OM, ChildRec, TempOpIdx);
if (ChildRec->isSubClassOf("ImmLeaf")) {
return failedImport(
"Src pattern child def is an unsupported tablegen class (ImmLeaf)");
}
return failedImport(
"Src pattern child def is an unsupported tablegen class");
}
return failedImport("Src pattern child is an unsupported kind");
}
Expected<action_iterator> GlobalISelEmitter::importExplicitUseRenderer(
action_iterator InsertPt, RuleMatcher &Rule, BuildMIAction &DstMIBuilder,
TreePatternNode *DstChild) {
const auto &SubOperand = Rule.getComplexSubOperand(DstChild->getName());
if (SubOperand.hasValue()) {
DstMIBuilder.addRenderer<RenderComplexPatternOperand>(
*std::get<0>(*SubOperand), DstChild->getName(),
std::get<1>(*SubOperand), std::get<2>(*SubOperand));
return InsertPt;
}
if (!DstChild->isLeaf()) {
if (DstChild->getOperator()->isSubClassOf("SDNodeXForm")) {
auto Child = DstChild->getChild(0);
auto I = SDNodeXFormEquivs.find(DstChild->getOperator());
if (I != SDNodeXFormEquivs.end()) {
DstMIBuilder.addRenderer<CustomRenderer>(*I->second, Child->getName());
return InsertPt;
}
return failedImport("SDNodeXForm " + Child->getName() +
" has no custom renderer");
}
// We accept 'bb' here. It's an operator because BasicBlockSDNode isn't
// inline, but in MI it's just another operand.
if (DstChild->getOperator()->isSubClassOf("SDNode")) {
auto &ChildSDNI = CGP.getSDNodeInfo(DstChild->getOperator());
if (ChildSDNI.getSDClassName() == "BasicBlockSDNode") {
DstMIBuilder.addRenderer<CopyRenderer>(DstChild->getName());
return InsertPt;
}
}
// Similarly, imm is an operator in TreePatternNode's view but must be
// rendered as operands.
// FIXME: The target should be able to choose sign-extended when appropriate
// (e.g. on Mips).
if (DstChild->getOperator()->getName() == "imm") {
DstMIBuilder.addRenderer<CopyConstantAsImmRenderer>(DstChild->getName());
return InsertPt;
} else if (DstChild->getOperator()->getName() == "fpimm") {
DstMIBuilder.addRenderer<CopyFConstantAsFPImmRenderer>(
DstChild->getName());
return InsertPt;
}
if (DstChild->getOperator()->isSubClassOf("Instruction")) {
ArrayRef<TypeSetByHwMode> ChildTypes = DstChild->getExtTypes();
if (ChildTypes.size() != 1)
return failedImport("Dst pattern child has multiple results");
Optional<LLTCodeGen> OpTyOrNone = None;
if (ChildTypes.front().isMachineValueType())
OpTyOrNone =
MVTToLLT(ChildTypes.front().getMachineValueType().SimpleTy);
if (!OpTyOrNone)
return failedImport("Dst operand has an unsupported type");
unsigned TempRegID = Rule.allocateTempRegID();
InsertPt = Rule.insertAction<MakeTempRegisterAction>(
InsertPt, OpTyOrNone.getValue(), TempRegID);
DstMIBuilder.addRenderer<TempRegRenderer>(TempRegID);
auto InsertPtOrError = createAndImportSubInstructionRenderer(
++InsertPt, Rule, DstChild, TempRegID);
if (auto Error = InsertPtOrError.takeError())
return std::move(Error);
return InsertPtOrError.get();
}
return failedImport("Dst pattern child isn't a leaf node or an MBB" + llvm::to_string(*DstChild));
}
// It could be a specific immediate in which case we should just check for
// that immediate.
if (const IntInit *ChildIntInit =
dyn_cast<IntInit>(DstChild->getLeafValue())) {
DstMIBuilder.addRenderer<ImmRenderer>(ChildIntInit->getValue());
return InsertPt;
}
// Otherwise, we're looking for a bog-standard RegisterClass operand.
if (auto *ChildDefInit = dyn_cast<DefInit>(DstChild->getLeafValue())) {
auto *ChildRec = ChildDefInit->getDef();
ArrayRef<TypeSetByHwMode> ChildTypes = DstChild->getExtTypes();
if (ChildTypes.size() != 1)
return failedImport("Dst pattern child has multiple results");
Optional<LLTCodeGen> OpTyOrNone = None;
if (ChildTypes.front().isMachineValueType())
OpTyOrNone = MVTToLLT(ChildTypes.front().getMachineValueType().SimpleTy);
if (!OpTyOrNone)
return failedImport("Dst operand has an unsupported type");
if (ChildRec->isSubClassOf("Register")) {
DstMIBuilder.addRenderer<AddRegisterRenderer>(ChildRec);
return InsertPt;
}
if (ChildRec->isSubClassOf("RegisterClass") ||
ChildRec->isSubClassOf("RegisterOperand") ||
ChildRec->isSubClassOf("ValueType")) {
if (ChildRec->isSubClassOf("RegisterOperand") &&
!ChildRec->isValueUnset("GIZeroRegister")) {
DstMIBuilder.addRenderer<CopyOrAddZeroRegRenderer>(
DstChild->getName(), ChildRec->getValueAsDef("GIZeroRegister"));
return InsertPt;
}
DstMIBuilder.addRenderer<CopyRenderer>(DstChild->getName());
return InsertPt;
}
if (ChildRec->isSubClassOf("ComplexPattern")) {
const auto &ComplexPattern = ComplexPatternEquivs.find(ChildRec);
if (ComplexPattern == ComplexPatternEquivs.end())
return failedImport(
"SelectionDAG ComplexPattern not mapped to GlobalISel");
const OperandMatcher &OM = Rule.getOperandMatcher(DstChild->getName());
DstMIBuilder.addRenderer<RenderComplexPatternOperand>(
*ComplexPattern->second, DstChild->getName(),
OM.getAllocatedTemporariesBaseID());
return InsertPt;
}
return failedImport(
"Dst pattern child def is an unsupported tablegen class");
}
return failedImport("Dst pattern child is an unsupported kind");
}
Expected<BuildMIAction &> GlobalISelEmitter::createAndImportInstructionRenderer(
RuleMatcher &M, const TreePatternNode *Dst) {
auto InsertPtOrError = createInstructionRenderer(M.actions_end(), M, Dst);
if (auto Error = InsertPtOrError.takeError())
return std::move(Error);
action_iterator InsertPt = InsertPtOrError.get();
BuildMIAction &DstMIBuilder = *static_cast<BuildMIAction *>(InsertPt->get());
importExplicitDefRenderers(DstMIBuilder);
if (auto Error = importExplicitUseRenderers(InsertPt, M, DstMIBuilder, Dst)
.takeError())
return std::move(Error);
return DstMIBuilder;
}
Expected<action_iterator>
GlobalISelEmitter::createAndImportSubInstructionRenderer(
const action_iterator InsertPt, RuleMatcher &M, const TreePatternNode *Dst,
unsigned TempRegID) {
auto InsertPtOrError = createInstructionRenderer(InsertPt, M, Dst);
// TODO: Assert there's exactly one result.
if (auto Error = InsertPtOrError.takeError())
return std::move(Error);
BuildMIAction &DstMIBuilder =
*static_cast<BuildMIAction *>(InsertPtOrError.get()->get());
// Assign the result to TempReg.
DstMIBuilder.addRenderer<TempRegRenderer>(TempRegID, true);
InsertPtOrError =
importExplicitUseRenderers(InsertPtOrError.get(), M, DstMIBuilder, Dst);
if (auto Error = InsertPtOrError.takeError())
return std::move(Error);
M.insertAction<ConstrainOperandsToDefinitionAction>(InsertPt,
DstMIBuilder.getInsnID());
return InsertPtOrError.get();
}
Expected<action_iterator> GlobalISelEmitter::createInstructionRenderer(
action_iterator InsertPt, RuleMatcher &M, const TreePatternNode *Dst) {
Record *DstOp = Dst->getOperator();
if (!DstOp->isSubClassOf("Instruction")) {
if (DstOp->isSubClassOf("ValueType"))
return failedImport(
"Pattern operator isn't an instruction (it's a ValueType)");
return failedImport("Pattern operator isn't an instruction");
}
CodeGenInstruction *DstI = &Target.getInstruction(DstOp);
// COPY_TO_REGCLASS is just a copy with a ConstrainOperandToRegClassAction
// attached. Similarly for EXTRACT_SUBREG except that's a subregister copy.
if (DstI->TheDef->getName() == "COPY_TO_REGCLASS")
DstI = &Target.getInstruction(RK.getDef("COPY"));
else if (DstI->TheDef->getName() == "EXTRACT_SUBREG")
DstI = &Target.getInstruction(RK.getDef("COPY"));
else if (DstI->TheDef->getName() == "REG_SEQUENCE")
return failedImport("Unable to emit REG_SEQUENCE");
return M.insertAction<BuildMIAction>(InsertPt, M.allocateOutputInsnID(),
DstI);
}
void GlobalISelEmitter::importExplicitDefRenderers(
BuildMIAction &DstMIBuilder) {
const CodeGenInstruction *DstI = DstMIBuilder.getCGI();
for (unsigned I = 0; I < DstI->Operands.NumDefs; ++I) {
const CGIOperandList::OperandInfo &DstIOperand = DstI->Operands[I];
DstMIBuilder.addRenderer<CopyRenderer>(DstIOperand.Name);
}
}
Expected<action_iterator> GlobalISelEmitter::importExplicitUseRenderers(
action_iterator InsertPt, RuleMatcher &M, BuildMIAction &DstMIBuilder,
const llvm::TreePatternNode *Dst) {
const CodeGenInstruction *DstI = DstMIBuilder.getCGI();
CodeGenInstruction *OrigDstI = &Target.getInstruction(Dst->getOperator());
// EXTRACT_SUBREG needs to use a subregister COPY.
if (OrigDstI->TheDef->getName() == "EXTRACT_SUBREG") {
if (!Dst->getChild(0)->isLeaf())
return failedImport("EXTRACT_SUBREG child #1 is not a leaf");
if (DefInit *SubRegInit =
dyn_cast<DefInit>(Dst->getChild(1)->getLeafValue())) {
Record *RCDef = getInitValueAsRegClass(Dst->getChild(0)->getLeafValue());
if (!RCDef)
return failedImport("EXTRACT_SUBREG child #0 could not "
"be coerced to a register class");
CodeGenRegisterClass *RC = CGRegs.getRegClass(RCDef);
CodeGenSubRegIndex *SubIdx = CGRegs.getSubRegIdx(SubRegInit->getDef());
const auto &SrcRCDstRCPair =
RC->getMatchingSubClassWithSubRegs(CGRegs, SubIdx);
if (SrcRCDstRCPair.hasValue()) {
assert(SrcRCDstRCPair->second && "Couldn't find a matching subclass");
if (SrcRCDstRCPair->first != RC)
return failedImport("EXTRACT_SUBREG requires an additional COPY");
}
DstMIBuilder.addRenderer<CopySubRegRenderer>(Dst->getChild(0)->getName(),
SubIdx);
return InsertPt;
}
return failedImport("EXTRACT_SUBREG child #1 is not a subreg index");
}
// Render the explicit uses.
unsigned DstINumUses = OrigDstI->Operands.size() - OrigDstI->Operands.NumDefs;
unsigned ExpectedDstINumUses = Dst->getNumChildren();
if (OrigDstI->TheDef->getName() == "COPY_TO_REGCLASS") {
DstINumUses--; // Ignore the class constraint.
ExpectedDstINumUses--;
}
unsigned Child = 0;
unsigned NumDefaultOps = 0;
for (unsigned I = 0; I != DstINumUses; ++I) {
const CGIOperandList::OperandInfo &DstIOperand =
DstI->Operands[DstI->Operands.NumDefs + I];
// If the operand has default values, introduce them now.
// FIXME: Until we have a decent test case that dictates we should do
// otherwise, we're going to assume that operands with default values cannot
// be specified in the patterns. Therefore, adding them will not cause us to
// end up with too many rendered operands.
if (DstIOperand.Rec->isSubClassOf("OperandWithDefaultOps")) {
DagInit *DefaultOps = DstIOperand.Rec->getValueAsDag("DefaultOps");
if (auto Error = importDefaultOperandRenderers(DstMIBuilder, DefaultOps))
return std::move(Error);
++NumDefaultOps;
continue;
}
auto InsertPtOrError = importExplicitUseRenderer(InsertPt, M, DstMIBuilder,
Dst->getChild(Child));
if (auto Error = InsertPtOrError.takeError())
return std::move(Error);
InsertPt = InsertPtOrError.get();
++Child;
}
if (NumDefaultOps + ExpectedDstINumUses != DstINumUses)
return failedImport("Expected " + llvm::to_string(DstINumUses) +
" used operands but found " +
llvm::to_string(ExpectedDstINumUses) +
" explicit ones and " + llvm::to_string(NumDefaultOps) +
" default ones");
return InsertPt;
}
Error GlobalISelEmitter::importDefaultOperandRenderers(
BuildMIAction &DstMIBuilder, DagInit *DefaultOps) const {
for (const auto *DefaultOp : DefaultOps->getArgs()) {
// Look through ValueType operators.
if (const DagInit *DefaultDagOp = dyn_cast<DagInit>(DefaultOp)) {
if (const DefInit *DefaultDagOperator =
dyn_cast<DefInit>(DefaultDagOp->getOperator())) {
if (DefaultDagOperator->getDef()->isSubClassOf("ValueType"))
DefaultOp = DefaultDagOp->getArg(0);
}
}
if (const DefInit *DefaultDefOp = dyn_cast<DefInit>(DefaultOp)) {
DstMIBuilder.addRenderer<AddRegisterRenderer>(DefaultDefOp->getDef());
continue;
}
if (const IntInit *DefaultIntOp = dyn_cast<IntInit>(DefaultOp)) {
DstMIBuilder.addRenderer<ImmRenderer>(DefaultIntOp->getValue());
continue;
}
return failedImport("Could not add default op");
}
return Error::success();
}
Error GlobalISelEmitter::importImplicitDefRenderers(
BuildMIAction &DstMIBuilder,
const std::vector<Record *> &ImplicitDefs) const {
if (!ImplicitDefs.empty())
return failedImport("Pattern defines a physical register");
return Error::success();
}
Expected<RuleMatcher> GlobalISelEmitter::runOnPattern(const PatternToMatch &P) {
// Keep track of the matchers and actions to emit.
int Score = P.getPatternComplexity(CGP);
RuleMatcher M(P.getSrcRecord()->getLoc());
RuleMatcherScores[M.getRuleID()] = Score;
M.addAction<DebugCommentAction>(llvm::to_string(*P.getSrcPattern()) +
" => " +
llvm::to_string(*P.getDstPattern()));
if (auto Error = importRulePredicates(M, P.getPredicates()))
return std::move(Error);
// Next, analyze the pattern operators.
TreePatternNode *Src = P.getSrcPattern();
TreePatternNode *Dst = P.getDstPattern();
// If the root of either pattern isn't a simple operator, ignore it.
if (auto Err = isTrivialOperatorNode(Dst))
return failedImport("Dst pattern root isn't a trivial operator (" +
toString(std::move(Err)) + ")");
if (auto Err = isTrivialOperatorNode(Src))
return failedImport("Src pattern root isn't a trivial operator (" +
toString(std::move(Err)) + ")");
// The different predicates and matchers created during
// addInstructionMatcher use the RuleMatcher M to set up their
// instruction ID (InsnVarID) that are going to be used when
// M is going to be emitted.
// However, the code doing the emission still relies on the IDs
// returned during that process by the RuleMatcher when issuing
// the recordInsn opcodes.
// Because of that:
// 1. The order in which we created the predicates
// and such must be the same as the order in which we emit them,
// and
// 2. We need to reset the generation of the IDs in M somewhere between
// addInstructionMatcher and emit
//
// FIXME: Long term, we don't want to have to rely on this implicit
// naming being the same. One possible solution would be to have
// explicit operator for operation capture and reference those.
// The plus side is that it would expose opportunities to share
// the capture accross rules. The downside is that it would
// introduce a dependency between predicates (captures must happen
// before their first use.)
InstructionMatcher &InsnMatcherTemp = M.addInstructionMatcher(Src->getName());
unsigned TempOpIdx = 0;
auto InsnMatcherOrError =
createAndImportSelDAGMatcher(M, InsnMatcherTemp, Src, TempOpIdx);
if (auto Error = InsnMatcherOrError.takeError())
return std::move(Error);
InstructionMatcher &InsnMatcher = InsnMatcherOrError.get();
if (Dst->isLeaf()) {
Record *RCDef = getInitValueAsRegClass(Dst->getLeafValue());
const CodeGenRegisterClass &RC = Target.getRegisterClass(RCDef);
if (RCDef) {
// We need to replace the def and all its uses with the specified
// operand. However, we must also insert COPY's wherever needed.
// For now, emit a copy and let the register allocator clean up.
auto &DstI = Target.getInstruction(RK.getDef("COPY"));
const auto &DstIOperand = DstI.Operands[0];
OperandMatcher &OM0 = InsnMatcher.getOperand(0);
OM0.setSymbolicName(DstIOperand.Name);
M.defineOperand(OM0.getSymbolicName(), OM0);
OM0.addPredicate<RegisterBankOperandMatcher>(RC);
auto &DstMIBuilder =
M.addAction<BuildMIAction>(M.allocateOutputInsnID(), &DstI);
DstMIBuilder.addRenderer<CopyRenderer>(DstIOperand.Name);
DstMIBuilder.addRenderer<CopyRenderer>(Dst->getName());
M.addAction<ConstrainOperandToRegClassAction>(0, 0, RC);
// We're done with this pattern! It's eligible for GISel emission; return
// it.
++NumPatternImported;
return std::move(M);
}
return failedImport("Dst pattern root isn't a known leaf");
}
// Start with the defined operands (i.e., the results of the root operator).
Record *DstOp = Dst->getOperator();
if (!DstOp->isSubClassOf("Instruction"))
return failedImport("Pattern operator isn't an instruction");
auto &DstI = Target.getInstruction(DstOp);
if (DstI.Operands.NumDefs != Src->getExtTypes().size())
return failedImport("Src pattern results and dst MI defs are different (" +
to_string(Src->getExtTypes().size()) + " def(s) vs " +
to_string(DstI.Operands.NumDefs) + " def(s))");
// The root of the match also has constraints on the register bank so that it
// matches the result instruction.
unsigned OpIdx = 0;
for (const TypeSetByHwMode &VTy : Src->getExtTypes()) {
(void)VTy;
const auto &DstIOperand = DstI.Operands[OpIdx];
Record *DstIOpRec = DstIOperand.Rec;
if (DstI.TheDef->getName() == "COPY_TO_REGCLASS") {
DstIOpRec = getInitValueAsRegClass(Dst->getChild(1)->getLeafValue());
if (DstIOpRec == nullptr)
return failedImport(
"COPY_TO_REGCLASS operand #1 isn't a register class");
} else if (DstI.TheDef->getName() == "EXTRACT_SUBREG") {
if (!Dst->getChild(0)->isLeaf())
return failedImport("EXTRACT_SUBREG operand #0 isn't a leaf");
// We can assume that a subregister is in the same bank as it's super
// register.
DstIOpRec = getInitValueAsRegClass(Dst->getChild(0)->getLeafValue());
if (DstIOpRec == nullptr)
return failedImport(
"EXTRACT_SUBREG operand #0 isn't a register class");
} else if (DstIOpRec->isSubClassOf("RegisterOperand"))
DstIOpRec = DstIOpRec->getValueAsDef("RegClass");
else if (!DstIOpRec->isSubClassOf("RegisterClass"))
return failedImport("Dst MI def isn't a register class" +
to_string(*Dst));
OperandMatcher &OM = InsnMatcher.getOperand(OpIdx);
OM.setSymbolicName(DstIOperand.Name);
M.defineOperand(OM.getSymbolicName(), OM);
OM.addPredicate<RegisterBankOperandMatcher>(
Target.getRegisterClass(DstIOpRec));
++OpIdx;
}
auto DstMIBuilderOrError = createAndImportInstructionRenderer(M, Dst);
if (auto Error = DstMIBuilderOrError.takeError())
return std::move(Error);
BuildMIAction &DstMIBuilder = DstMIBuilderOrError.get();
// Render the implicit defs.
// These are only added to the root of the result.
if (auto Error = importImplicitDefRenderers(DstMIBuilder, P.getDstRegs()))
return std::move(Error);
DstMIBuilder.chooseInsnToMutate(M);
// Constrain the registers to classes. This is normally derived from the
// emitted instruction but a few instructions require special handling.
if (DstI.TheDef->getName() == "COPY_TO_REGCLASS") {
// COPY_TO_REGCLASS does not provide operand constraints itself but the
// result is constrained to the class given by the second child.
Record *DstIOpRec =
getInitValueAsRegClass(Dst->getChild(1)->getLeafValue());
if (DstIOpRec == nullptr)
return failedImport("COPY_TO_REGCLASS operand #1 isn't a register class");
M.addAction<ConstrainOperandToRegClassAction>(
0, 0, Target.getRegisterClass(DstIOpRec));
// We're done with this pattern! It's eligible for GISel emission; return
// it.
++NumPatternImported;
return std::move(M);
}
if (DstI.TheDef->getName() == "EXTRACT_SUBREG") {
// EXTRACT_SUBREG selects into a subregister COPY but unlike most
// instructions, the result register class is controlled by the
// subregisters of the operand. As a result, we must constrain the result
// class rather than check that it's already the right one.
if (!Dst->getChild(0)->isLeaf())
return failedImport("EXTRACT_SUBREG child #1 is not a leaf");
DefInit *SubRegInit = dyn_cast<DefInit>(Dst->getChild(1)->getLeafValue());
if (!SubRegInit)
return failedImport("EXTRACT_SUBREG child #1 is not a subreg index");
// Constrain the result to the same register bank as the operand.
Record *DstIOpRec =
getInitValueAsRegClass(Dst->getChild(0)->getLeafValue());
if (DstIOpRec == nullptr)
return failedImport("EXTRACT_SUBREG operand #1 isn't a register class");
CodeGenSubRegIndex *SubIdx = CGRegs.getSubRegIdx(SubRegInit->getDef());
CodeGenRegisterClass *SrcRC = CGRegs.getRegClass(DstIOpRec);
// It would be nice to leave this constraint implicit but we're required
// to pick a register class so constrain the result to a register class
// that can hold the correct MVT.
//
// FIXME: This may introduce an extra copy if the chosen class doesn't
// actually contain the subregisters.
assert(Src->getExtTypes().size() == 1 &&
"Expected Src of EXTRACT_SUBREG to have one result type");
const auto &SrcRCDstRCPair =
SrcRC->getMatchingSubClassWithSubRegs(CGRegs, SubIdx);
assert(SrcRCDstRCPair->second && "Couldn't find a matching subclass");
M.addAction<ConstrainOperandToRegClassAction>(0, 0, *SrcRCDstRCPair->second);
M.addAction<ConstrainOperandToRegClassAction>(0, 1, *SrcRCDstRCPair->first);
// We're done with this pattern! It's eligible for GISel emission; return
// it.
++NumPatternImported;
return std::move(M);
}
M.addAction<ConstrainOperandsToDefinitionAction>(0);
// We're done with this pattern! It's eligible for GISel emission; return it.
++NumPatternImported;
return std::move(M);
}
// Emit imm predicate table and an enum to reference them with.
// The 'Predicate_' part of the name is redundant but eliminating it is more
// trouble than it's worth.
void GlobalISelEmitter::emitImmPredicates(
raw_ostream &OS, StringRef TypeIdentifier, StringRef Type,
std::function<bool(const Record *R)> Filter) {
std::vector<const Record *> MatchedRecords;
const auto &Defs = RK.getAllDerivedDefinitions("PatFrag");
std::copy_if(Defs.begin(), Defs.end(), std::back_inserter(MatchedRecords),
[&](Record *Record) {
return !Record->getValueAsString("ImmediateCode").empty() &&
Filter(Record);
});
if (!MatchedRecords.empty()) {
OS << "// PatFrag predicates.\n"
<< "enum {\n";
std::string EnumeratorSeparator =
(" = GIPFP_" + TypeIdentifier + "_Invalid + 1,\n").str();
for (const auto *Record : MatchedRecords) {
OS << " GIPFP_" << TypeIdentifier << "_Predicate_" << Record->getName()
<< EnumeratorSeparator;
EnumeratorSeparator = ",\n";
}
OS << "};\n";
}
OS << "bool " << Target.getName() << "InstructionSelector::testImmPredicate_"
<< TypeIdentifier << "(unsigned PredicateID, " << Type
<< " Imm) const {\n";
if (!MatchedRecords.empty())
OS << " switch (PredicateID) {\n";
for (const auto *Record : MatchedRecords) {
OS << " case GIPFP_" << TypeIdentifier << "_Predicate_"
<< Record->getName() << ": {\n"
<< " " << Record->getValueAsString("ImmediateCode") << "\n"
<< " llvm_unreachable(\"ImmediateCode should have returned\");\n"
<< " return false;\n"
<< " }\n";
}
if (!MatchedRecords.empty())
OS << " }\n";
OS << " llvm_unreachable(\"Unknown predicate\");\n"
<< " return false;\n"
<< "}\n";
}
template <class GroupT>
std::vector<Matcher *> GlobalISelEmitter::optimizeRules(
ArrayRef<Matcher *> Rules,
std::vector<std::unique_ptr<Matcher>> &MatcherStorage) {
std::vector<Matcher *> OptRules;
std::unique_ptr<GroupT> CurrentGroup = make_unique<GroupT>();
assert(CurrentGroup->empty() && "Newly created group isn't empty!");
unsigned NumGroups = 0;
auto ProcessCurrentGroup = [&]() {
if (CurrentGroup->empty())
// An empty group is good to be reused:
return;
// If the group isn't large enough to provide any benefit, move all the
// added rules out of it and make sure to re-create the group to properly
// re-initialize it:
if (CurrentGroup->size() < 2)
for (Matcher *M : CurrentGroup->matchers())
OptRules.push_back(M);
else {
CurrentGroup->finalize();
OptRules.push_back(CurrentGroup.get());
MatcherStorage.emplace_back(std::move(CurrentGroup));
++NumGroups;
}
CurrentGroup = make_unique<GroupT>();
};
for (Matcher *Rule : Rules) {
// Greedily add as many matchers as possible to the current group:
if (CurrentGroup->addMatcher(*Rule))
continue;
ProcessCurrentGroup();
assert(CurrentGroup->empty() && "A group wasn't properly re-initialized");
// Try to add the pending matcher to a newly created empty group:
if (!CurrentGroup->addMatcher(*Rule))
// If we couldn't add the matcher to an empty group, that group type
// doesn't support that kind of matchers at all, so just skip it:
OptRules.push_back(Rule);
}
ProcessCurrentGroup();
DEBUG(dbgs() << "NumGroups: " << NumGroups << "\n");
assert(CurrentGroup->empty() && "The last group wasn't properly processed");
return OptRules;
}
MatchTable
GlobalISelEmitter::buildMatchTable(MutableArrayRef<RuleMatcher> Rules,
bool Optimize, bool WithCoverage) {
std::vector<Matcher *> InputRules;
for (Matcher &Rule : Rules)
InputRules.push_back(&Rule);
if (!Optimize)
return MatchTable::buildTable(InputRules, WithCoverage);
+ unsigned CurrentOrdering = 0;
+ StringMap<unsigned> OpcodeOrder;
+ for (RuleMatcher &Rule : Rules) {
+ const StringRef Opcode = Rule.getOpcode();
+ assert(!Opcode.empty() && "Didn't expect an undefined opcode");
+ if (OpcodeOrder.count(Opcode) == 0)
+ OpcodeOrder[Opcode] = CurrentOrdering++;
+ }
+
+ std::stable_sort(InputRules.begin(), InputRules.end(),
+ [&OpcodeOrder](const Matcher *A, const Matcher *B) {
+ auto *L = static_cast<const RuleMatcher *>(A);
+ auto *R = static_cast<const RuleMatcher *>(B);
+ return std::make_tuple(OpcodeOrder[L->getOpcode()],
+ L->getNumOperands()) <
+ std::make_tuple(OpcodeOrder[R->getOpcode()],
+ R->getNumOperands());
+ });
+
for (Matcher *Rule : InputRules)
Rule->optimize();
std::vector<std::unique_ptr<Matcher>> MatcherStorage;
std::vector<Matcher *> OptRules =
optimizeRules<GroupMatcher>(InputRules, MatcherStorage);
for (Matcher *Rule : OptRules)
Rule->optimize();
return MatchTable::buildTable(OptRules, WithCoverage);
}
void GlobalISelEmitter::run(raw_ostream &OS) {
if (!UseCoverageFile.empty()) {
RuleCoverage = CodeGenCoverage();
auto RuleCoverageBufOrErr = MemoryBuffer::getFile(UseCoverageFile);
if (!RuleCoverageBufOrErr) {
PrintWarning(SMLoc(), "Missing rule coverage data");
RuleCoverage = None;
} else {
if (!RuleCoverage->parse(*RuleCoverageBufOrErr.get(), Target.getName())) {
PrintWarning(SMLoc(), "Ignoring invalid or missing rule coverage data");
RuleCoverage = None;
}
}
}
// Track the run-time opcode values
gatherOpcodeValues();
// Track the run-time LLT ID values
gatherTypeIDValues();
// Track the GINodeEquiv definitions.
gatherNodeEquivs();
emitSourceFileHeader(("Global Instruction Selector for the " +
Target.getName() + " target").str(), OS);
std::vector<RuleMatcher> Rules;
// Look through the SelectionDAG patterns we found, possibly emitting some.
for (const PatternToMatch &Pat : CGP.ptms()) {
++NumPatternTotal;
auto MatcherOrErr = runOnPattern(Pat);
// The pattern analysis can fail, indicating an unsupported pattern.
// Report that if we've been asked to do so.
if (auto Err = MatcherOrErr.takeError()) {
if (WarnOnSkippedPatterns) {
PrintWarning(Pat.getSrcRecord()->getLoc(),
"Skipped pattern: " + toString(std::move(Err)));
} else {
consumeError(std::move(Err));
}
++NumPatternImportsSkipped;
continue;
}
if (RuleCoverage) {
if (RuleCoverage->isCovered(MatcherOrErr->getRuleID()))
++NumPatternsTested;
else
PrintWarning(Pat.getSrcRecord()->getLoc(),
"Pattern is not covered by a test");
}
Rules.push_back(std::move(MatcherOrErr.get()));
}
// Comparison function to order records by name.
auto orderByName = [](const Record *A, const Record *B) {
return A->getName() < B->getName();
};
std::vector<Record *> ComplexPredicates =
RK.getAllDerivedDefinitions("GIComplexOperandMatcher");
llvm::sort(ComplexPredicates.begin(), ComplexPredicates.end(), orderByName);
std::vector<Record *> CustomRendererFns =
RK.getAllDerivedDefinitions("GICustomOperandRenderer");
llvm::sort(CustomRendererFns.begin(), CustomRendererFns.end(), orderByName);
unsigned MaxTemporaries = 0;
for (const auto &Rule : Rules)
MaxTemporaries = std::max(MaxTemporaries, Rule.countRendererFns());
OS << "#ifdef GET_GLOBALISEL_PREDICATE_BITSET\n"
<< "const unsigned MAX_SUBTARGET_PREDICATES = " << SubtargetFeatures.size()
<< ";\n"
<< "using PredicateBitset = "
"llvm::PredicateBitsetImpl<MAX_SUBTARGET_PREDICATES>;\n"
<< "#endif // ifdef GET_GLOBALISEL_PREDICATE_BITSET\n\n";
OS << "#ifdef GET_GLOBALISEL_TEMPORARIES_DECL\n"
<< " mutable MatcherState State;\n"
<< " typedef "
"ComplexRendererFns("
<< Target.getName()
<< "InstructionSelector::*ComplexMatcherMemFn)(MachineOperand &) const;\n"
<< " typedef void(" << Target.getName()
<< "InstructionSelector::*CustomRendererFn)(MachineInstrBuilder &, const "
"MachineInstr&) "
"const;\n"
<< " const ISelInfoTy<PredicateBitset, ComplexMatcherMemFn, "
"CustomRendererFn> "
"ISelInfo;\n";
OS << " static " << Target.getName()
<< "InstructionSelector::ComplexMatcherMemFn ComplexPredicateFns[];\n"
<< " static " << Target.getName()
<< "InstructionSelector::CustomRendererFn CustomRenderers[];\n"
<< " bool testImmPredicate_I64(unsigned PredicateID, int64_t Imm) const "
"override;\n"
<< " bool testImmPredicate_APInt(unsigned PredicateID, const APInt &Imm) "
"const override;\n"
<< " bool testImmPredicate_APFloat(unsigned PredicateID, const APFloat "
"&Imm) const override;\n"
<< " const int64_t *getMatchTable() const override;\n"
<< "#endif // ifdef GET_GLOBALISEL_TEMPORARIES_DECL\n\n";
OS << "#ifdef GET_GLOBALISEL_TEMPORARIES_INIT\n"
<< ", State(" << MaxTemporaries << "),\n"
<< "ISelInfo(TypeObjects, NumTypeObjects, FeatureBitsets"
<< ", ComplexPredicateFns, CustomRenderers)\n"
<< "#endif // ifdef GET_GLOBALISEL_TEMPORARIES_INIT\n\n";
OS << "#ifdef GET_GLOBALISEL_IMPL\n";
SubtargetFeatureInfo::emitSubtargetFeatureBitEnumeration(SubtargetFeatures,
OS);
// Separate subtarget features by how often they must be recomputed.
SubtargetFeatureInfoMap ModuleFeatures;
std::copy_if(SubtargetFeatures.begin(), SubtargetFeatures.end(),
std::inserter(ModuleFeatures, ModuleFeatures.end()),
[](const SubtargetFeatureInfoMap::value_type &X) {
return !X.second.mustRecomputePerFunction();
});
SubtargetFeatureInfoMap FunctionFeatures;
std::copy_if(SubtargetFeatures.begin(), SubtargetFeatures.end(),
std::inserter(FunctionFeatures, FunctionFeatures.end()),
[](const SubtargetFeatureInfoMap::value_type &X) {
return X.second.mustRecomputePerFunction();
});
SubtargetFeatureInfo::emitComputeAvailableFeatures(
Target.getName(), "InstructionSelector", "computeAvailableModuleFeatures",
ModuleFeatures, OS);
SubtargetFeatureInfo::emitComputeAvailableFeatures(
Target.getName(), "InstructionSelector",
"computeAvailableFunctionFeatures", FunctionFeatures, OS,
"const MachineFunction *MF");
// Emit a table containing the LLT objects needed by the matcher and an enum
// for the matcher to reference them with.
std::vector<LLTCodeGen> TypeObjects;
for (const auto &Ty : KnownTypes)
TypeObjects.push_back(Ty);
llvm::sort(TypeObjects.begin(), TypeObjects.end());
OS << "// LLT Objects.\n"
<< "enum {\n";
for (const auto &TypeObject : TypeObjects) {
OS << " ";
TypeObject.emitCxxEnumValue(OS);
OS << ",\n";
}
OS << "};\n";
OS << "const static size_t NumTypeObjects = " << TypeObjects.size() << ";\n"
<< "const static LLT TypeObjects[] = {\n";
for (const auto &TypeObject : TypeObjects) {
OS << " ";
TypeObject.emitCxxConstructorCall(OS);
OS << ",\n";
}
OS << "};\n\n";
// Emit a table containing the PredicateBitsets objects needed by the matcher
// and an enum for the matcher to reference them with.
std::vector<std::vector<Record *>> FeatureBitsets;
for (auto &Rule : Rules)
FeatureBitsets.push_back(Rule.getRequiredFeatures());
llvm::sort(
FeatureBitsets.begin(), FeatureBitsets.end(),
[&](const std::vector<Record *> &A, const std::vector<Record *> &B) {
if (A.size() < B.size())
return true;
if (A.size() > B.size())
return false;
for (const auto &Pair : zip(A, B)) {
if (std::get<0>(Pair)->getName() < std::get<1>(Pair)->getName())
return true;
if (std::get<0>(Pair)->getName() > std::get<1>(Pair)->getName())
return false;
}
return false;
});
FeatureBitsets.erase(
std::unique(FeatureBitsets.begin(), FeatureBitsets.end()),
FeatureBitsets.end());
OS << "// Feature bitsets.\n"
<< "enum {\n"
<< " GIFBS_Invalid,\n";
for (const auto &FeatureBitset : FeatureBitsets) {
if (FeatureBitset.empty())
continue;
OS << " " << getNameForFeatureBitset(FeatureBitset) << ",\n";
}
OS << "};\n"
<< "const static PredicateBitset FeatureBitsets[] {\n"
<< " {}, // GIFBS_Invalid\n";
for (const auto &FeatureBitset : FeatureBitsets) {
if (FeatureBitset.empty())
continue;
OS << " {";
for (const auto &Feature : FeatureBitset) {
const auto &I = SubtargetFeatures.find(Feature);
assert(I != SubtargetFeatures.end() && "Didn't import predicate?");
OS << I->second.getEnumBitName() << ", ";
}
OS << "},\n";
}
OS << "};\n\n";
// Emit complex predicate table and an enum to reference them with.
OS << "// ComplexPattern predicates.\n"
<< "enum {\n"
<< " GICP_Invalid,\n";
for (const auto &Record : ComplexPredicates)
OS << " GICP_" << Record->getName() << ",\n";
OS << "};\n"
<< "// See constructor for table contents\n\n";
emitImmPredicates(OS, "I64", "int64_t", [](const Record *R) {
bool Unset;
return !R->getValueAsBitOrUnset("IsAPFloat", Unset) &&
!R->getValueAsBit("IsAPInt");
});
emitImmPredicates(OS, "APFloat", "const APFloat &", [](const Record *R) {
bool Unset;
return R->getValueAsBitOrUnset("IsAPFloat", Unset);
});
emitImmPredicates(OS, "APInt", "const APInt &", [](const Record *R) {
return R->getValueAsBit("IsAPInt");
});
OS << "\n";
OS << Target.getName() << "InstructionSelector::ComplexMatcherMemFn\n"
<< Target.getName() << "InstructionSelector::ComplexPredicateFns[] = {\n"
<< " nullptr, // GICP_Invalid\n";
for (const auto &Record : ComplexPredicates)
OS << " &" << Target.getName()
<< "InstructionSelector::" << Record->getValueAsString("MatcherFn")
<< ", // " << Record->getName() << "\n";
OS << "};\n\n";
OS << "// Custom renderers.\n"
<< "enum {\n"
<< " GICR_Invalid,\n";
for (const auto &Record : CustomRendererFns)
OS << " GICR_" << Record->getValueAsString("RendererFn") << ", \n";
OS << "};\n";
OS << Target.getName() << "InstructionSelector::CustomRendererFn\n"
<< Target.getName() << "InstructionSelector::CustomRenderers[] = {\n"
<< " nullptr, // GICP_Invalid\n";
for (const auto &Record : CustomRendererFns)
OS << " &" << Target.getName()
<< "InstructionSelector::" << Record->getValueAsString("RendererFn")
<< ", // " << Record->getName() << "\n";
OS << "};\n\n";
std::stable_sort(Rules.begin(), Rules.end(), [&](const RuleMatcher &A,
const RuleMatcher &B) {
int ScoreA = RuleMatcherScores[A.getRuleID()];
int ScoreB = RuleMatcherScores[B.getRuleID()];
if (ScoreA > ScoreB)
return true;
if (ScoreB > ScoreA)
return false;
if (A.isHigherPriorityThan(B)) {
assert(!B.isHigherPriorityThan(A) && "Cannot be more important "
"and less important at "
"the same time");
return true;
}
return false;
});
OS << "bool " << Target.getName()
<< "InstructionSelector::selectImpl(MachineInstr &I, CodeGenCoverage "
"&CoverageInfo) const {\n"
<< " MachineFunction &MF = *I.getParent()->getParent();\n"
<< " MachineRegisterInfo &MRI = MF.getRegInfo();\n"
<< " // FIXME: This should be computed on a per-function basis rather "
"than per-insn.\n"
<< " AvailableFunctionFeatures = computeAvailableFunctionFeatures(&STI, "
"&MF);\n"
<< " const PredicateBitset AvailableFeatures = getAvailableFeatures();\n"
<< " NewMIVector OutMIs;\n"
<< " State.MIs.clear();\n"
<< " State.MIs.push_back(&I);\n\n"
<< " if (executeMatchTable(*this, OutMIs, State, ISelInfo"
<< ", getMatchTable(), TII, MRI, TRI, RBI, AvailableFeatures"
<< ", CoverageInfo)) {\n"
<< " return true;\n"
<< " }\n\n"
<< " return false;\n"
<< "}\n\n";
const MatchTable Table =
buildMatchTable(Rules, OptimizeMatchTable, GenerateCoverage);
OS << "const int64_t *" << Target.getName()
<< "InstructionSelector::getMatchTable() const {\n";
Table.emitDeclaration(OS);
OS << " return ";
Table.emitUse(OS);
OS << ";\n}\n";
OS << "#endif // ifdef GET_GLOBALISEL_IMPL\n";
OS << "#ifdef GET_GLOBALISEL_PREDICATES_DECL\n"
<< "PredicateBitset AvailableModuleFeatures;\n"
<< "mutable PredicateBitset AvailableFunctionFeatures;\n"
<< "PredicateBitset getAvailableFeatures() const {\n"
<< " return AvailableModuleFeatures | AvailableFunctionFeatures;\n"
<< "}\n"
<< "PredicateBitset\n"
<< "computeAvailableModuleFeatures(const " << Target.getName()
<< "Subtarget *Subtarget) const;\n"
<< "PredicateBitset\n"
<< "computeAvailableFunctionFeatures(const " << Target.getName()
<< "Subtarget *Subtarget,\n"
<< " const MachineFunction *MF) const;\n"
<< "#endif // ifdef GET_GLOBALISEL_PREDICATES_DECL\n";
OS << "#ifdef GET_GLOBALISEL_PREDICATES_INIT\n"
<< "AvailableModuleFeatures(computeAvailableModuleFeatures(&STI)),\n"
<< "AvailableFunctionFeatures()\n"
<< "#endif // ifdef GET_GLOBALISEL_PREDICATES_INIT\n";
}
void GlobalISelEmitter::declareSubtargetFeature(Record *Predicate) {
if (SubtargetFeatures.count(Predicate) == 0)
SubtargetFeatures.emplace(
Predicate, SubtargetFeatureInfo(Predicate, SubtargetFeatures.size()));
}
void RuleMatcher::optimize() {
for (auto &Item : InsnVariableIDs) {
InstructionMatcher &InsnMatcher = *Item.first;
for (auto &OM : InsnMatcher.operands()) {
// Register Banks checks rarely fail, but often crash as targets usually
// provide only partially defined RegisterBankInfo::getRegBankFromRegClass
// method. Often the problem is hidden as non-optimized MatchTable checks
// banks rather late, most notably after checking target / function /
// module features and a few opcodes. That makes these checks a)
// beneficial to delay until the very end (we don't want to perform a lot
// of checks that all pass and then fail at the very end) b) not safe to
// have as early checks.
for (auto &OP : OM->predicates())
if (isa<RegisterBankOperandMatcher>(OP) ||
isa<ComplexPatternOperandMatcher>(OP))
EpilogueMatchers.emplace_back(std::move(OP));
OM->eraseNullPredicates();
}
InsnMatcher.optimize();
}
llvm::sort(
EpilogueMatchers.begin(), EpilogueMatchers.end(),
[](const std::unique_ptr<PredicateMatcher> &L,
const std::unique_ptr<PredicateMatcher> &R) {
return std::make_tuple(L->getKind(), L->getInsnVarID(), L->getOpIdx()) <
std::make_tuple(R->getKind(), R->getInsnVarID(), R->getOpIdx());
});
}
bool RuleMatcher::hasFirstCondition() const {
if (insnmatchers_empty())
return false;
InstructionMatcher &Matcher = insnmatchers_front();
if (!Matcher.predicates_empty())
return true;
for (auto &OM : Matcher.operands())
for (auto &OP : OM->predicates())
if (!isa<InstructionOperandMatcher>(OP))
return true;
return false;
}
const PredicateMatcher &RuleMatcher::getFirstCondition() const {
assert(!insnmatchers_empty() &&
"Trying to get a condition from an empty RuleMatcher");
InstructionMatcher &Matcher = insnmatchers_front();
if (!Matcher.predicates_empty())
return **Matcher.predicates_begin();
// If there is no more predicate on the instruction itself, look at its
// operands.
for (auto &OM : Matcher.operands())
for (auto &OP : OM->predicates())
if (!isa<InstructionOperandMatcher>(OP))
return *OP;
llvm_unreachable("Trying to get a condition from an InstructionMatcher with "
"no conditions");
}
std::unique_ptr<PredicateMatcher> RuleMatcher::popFirstCondition() {
assert(!insnmatchers_empty() &&
"Trying to pop a condition from an empty RuleMatcher");
InstructionMatcher &Matcher = insnmatchers_front();
if (!Matcher.predicates_empty())
return Matcher.predicates_pop_front();
// If there is no more predicate on the instruction itself, look at its
// operands.
for (auto &OM : Matcher.operands())
for (auto &OP : OM->predicates())
if (!isa<InstructionOperandMatcher>(OP)) {
std::unique_ptr<PredicateMatcher> Result = std::move(OP);
OM->eraseNullPredicates();
return Result;
}
llvm_unreachable("Trying to pop a condition from an InstructionMatcher with "
"no conditions");
}
bool GroupMatcher::candidateConditionMatches(
const PredicateMatcher &Predicate) const {
if (empty()) {
// Sharing predicates for nested instructions is not supported yet as we
// currently don't hoist the GIM_RecordInsn's properly, therefore we can
// only work on the original root instruction (InsnVarID == 0):
if (Predicate.getInsnVarID() != 0)
return false;
// ... otherwise an empty group can handle any predicate with no specific
// requirements:
return true;
}
const Matcher &Representative = **Matchers.begin();
const auto &RepresentativeCondition = Representative.getFirstCondition();
// ... if not empty, the group can only accomodate matchers with the exact
// same first condition:
return Predicate.isIdentical(RepresentativeCondition);
}
bool GroupMatcher::addMatcher(Matcher &Candidate) {
if (!Candidate.hasFirstCondition())
return false;
const PredicateMatcher &Predicate = Candidate.getFirstCondition();
if (!candidateConditionMatches(Predicate))
return false;
Matchers.push_back(&Candidate);
return true;
}
void GroupMatcher::finalize() {
assert(Conditions.empty() && "Already finalized?");
if (empty())
return;
Matcher &FirstRule = **Matchers.begin();
Conditions.push_back(FirstRule.popFirstCondition());
for (unsigned I = 1, E = Matchers.size(); I < E; ++I)
Matchers[I]->popFirstCondition();
}
void GroupMatcher::emit(MatchTable &Table) {
unsigned LabelID = ~0U;
if (!Conditions.empty()) {
LabelID = Table.allocateLabelID();
Table << MatchTable::Opcode("GIM_Try", +1)
<< MatchTable::Comment("On fail goto")
<< MatchTable::JumpTarget(LabelID) << MatchTable::LineBreak;
}
for (auto &Condition : Conditions)
Condition->emitPredicateOpcodes(
Table, *static_cast<RuleMatcher *>(*Matchers.begin()));
for (const auto &M : Matchers)
M->emit(Table);
// Exit the group
if (!Conditions.empty())
Table << MatchTable::Opcode("GIM_Reject", -1) << MatchTable::LineBreak
<< MatchTable::Label(LabelID);
}
unsigned OperandMatcher::getInsnVarID() const { return Insn.getInsnVarID(); }
} // end anonymous namespace
//===----------------------------------------------------------------------===//
namespace llvm {
void EmitGlobalISel(RecordKeeper &RK, raw_ostream &OS) {
GlobalISelEmitter(RK).run(OS);
}
} // End llvm namespace
llvm-svn: 332999
|
|
|
Lang Hames
|
5261aa9f91 |
[ORC] Move symbol-scanning and discard from BasicIRLayerMaterializationUnit in
to a base class (IRMaterializationUnit). The new class, IRMaterializationUnit, provides a convenient base for any client that wants to write a materializer for LLVM IR. llvm-svn: 332993 |
|
|
Sanjay Patel
|
dd5fb8f03f |
[InstCombine] fix broken test
Looks like the last line got chopped off from rL332990. llvm-svn: 332992 |
|
|
David Bolvansky
|
41f4b64ee1 |
[InstCombine] Calloc-ed strings optimizations
Summary: Example cases: strlen(calloc(...)) -> 0 Reviewers: efriedma, bkramer Reviewed By: bkramer Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D47059 llvm-svn: 332990 |
|
|
Dan Liew
|
6dfcc78364 |
[lit] Try to make `shtest-timeout.py` test more reliable by using a
larger timeout value. This really isn't very good because it will still be susceptible to machine performance. While we are here also fix a bug in validation of `maxIndividualTestTime` where previously it wasn't checked if the type was an int. rdar://problem/40221572 llvm-svn: 332987 |
|
|
Dan Liew
|
9f7786855c |
[lit] Don't run `slow.py` in `shtest-timeout.py` test.
The program used to be used in `quick_then_slow.py` but that was removed in r328702. The tests always run `slow.py` on its own but this doesn't really test additional code so we'll just drop running `slow.py` so the tests run faster. rdar://problem/40221572 llvm-svn: 332986 |
|
|
Dan Liew
|
8b4d36a530 |
[lit] Don't check output of commands used in `shtest-timeout.py` test.
If the system is under heavy load 1 second might not be long enough for it to produce output which could lead to spurious test failures. What matters is that the right test cases reach a timeout. rdar://problem/40221572 llvm-svn: 332985 |
|
|
Simon Dardis
|
13de555737 |
[FastISel] Permit instructions to be skipped for FastISel generation.
Some ISA's such as microMIPS32(R6) have instructions which are near identical for code generation purposes, e.g. xor and xor16. These instructions take the same value types for operands and return values, have the same instruction predicates and map to the same ISD opcode. (These instructions do differ by register classes.) In such cases, the FastISel generator rejects the instruction definition. This patch borrows the 'FastIselShouldIgnore' bit from rL129692 and enables applying it to an instruction definition. Reviewers: mcrosier Differential Revision: https://reviews.llvm.org/D46953 llvm-svn: 332983 |
|
|
Clement Courbet
|
488ebfb732 |
[llvm-exegesis] Update doc to mention that the output is in html.
llvm-svn: 332980 |
|
|
Clement Courbet
|
17d3c257b9 |
[llvm-exegesis] Analysis output uses HTML.
Summary: This makes the report much more readable. Reviewers: gchatelet Subscribers: tschuett, mgrang, craig.topper, RKSimon, llvm-commits Differential Revision: https://reviews.llvm.org/D47189 llvm-svn: 332979 |
|
|
Aleksandar Beserminji
|
a5f755186a |
[mips] Merge MipsLongBranch and MipsHazardSchedule passes
MipsLongBranchPass and MipsHazardSchedule passes are joined to one pass because of mutual conflict. When MipsHazardSchedule inserts 'nop's, it potentially breaks some jumps, so they have to be expanded to long branches. When some branch is expanded to long branch, it potentially creates a hazard situation, which should be fixed by adding nops. New pass is called MipsBranchExpansion, it combines these two passes, and runs them alternately until one of them reports no changes were made. Differential Revision: https://reviews.llvm.org/D46641 llvm-svn: 332977 |
|
|
Simon Dardis
|
437153bb80 |
[mips] Correct the predicates of the cache and pref instructions
Reviewers: atanasyan, abeserminji, smaksimovic Differential Revision: https://reviews.llvm.org/D46949 llvm-svn: 332970 |
|
|
Simon Pilgrim
|
4162d77744 |
[TTI] Add uniform/non-uniform constant Pow2 detection to TargetTransformInfo::getInstructionThroughput
This enables us to detect more fast path sdiv cases under cost analysis. This patch also enables us to handle non-uniform-constant pow2 cases for X86 SDIV costs. Found while working on D46276 Future patches can then extend the vectorizers to more fully support non-uniform pow2 cases. Differential Revision: https://reviews.llvm.org/D46637 llvm-svn: 332969 |
|
|
Hans Wennborg
|
2cfcc01b22 |
LangRef.rst: the "\01" prefix applies not just to variables
llvm-svn: 332967 |
|
|
Gabor Buella
|
e96c488aa7 |
[x86] NFC Add some more shuffle-vs-trunc tests
These are related to: https://reviews.llvm.org/D46957 llvm-svn: 332962 |
|
|
Karl-Johan Karlsson
|
11d68a619e |
[LowerSwitch] Fixed faulty PHI node update
Summary: When lowerswitch merge several cases into a new default block it's not updating the PHI nodes accordingly. The code that update the PHI nodes for the default edge only update the first entry and do not remove the remaining ones, to make sure the number of entries match the number of predecessors. This is easily fixed by replacing the code that update the PHI node with the already existing utility function for updating PHI nodes. Reviewers: hans, reames, arsenm Reviewed By: arsenm Subscribers: wdng, llvm-commits Differential Revision: https://reviews.llvm.org/D47055 llvm-svn: 332960 |
|
|
Bjorn Pettersson
|
fecef6be9e |
[LoopVersioning] Don't modify the list that we iterate over in addPHINodes
Summary: In LoopVersioning::addPHINodes we need to iterate over all users for a value "Inst", and if the user is outside of the VersionedLoop we should replace the use of "Inst" by using the value "PN" instead. Replacing the use of "Inst" for a user of "Inst" also means that Inst->users() is modified. So it is not safe to do the replace while iterating over Inst->users() as we used to do. This patch splits the task into two steps. First we iterate over Inst->users() to find all users that should be updated. Those users are saved into a local data structure on the stack. And then, in the second step, we do the actual updates. This time iterating over the local data structure. Reviewers: mzolotukhin, anemet Reviewed By: mzolotukhin Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D47134 llvm-svn: 332958 |
|
|
Stanislav Mekhanoshin
|
0e132dca53 |
[AMDGPU] Optimze old value of v_mov_b32_dpp
We can eliminate old value if bound_ctrl = 1 and row_mask = bank_mask = 0xf. This is alternative implementation working with the intrinsic in InstCombine. Original review for past-ISel optimization: D46570. Differential Revision: https://reviews.llvm.org/D46596 llvm-svn: 332956 |
|
|
Matt Arsenault
|
1349a04ef5 |
AMDGPU: Make v2i16/v2f16 legal on VI
This usually results in better code. Fixes using inline asm with short2, and also fixes having a different ABI for function parameters between VI and gfx9. Partially cleans up the mess used for lowering of the d16 operations. Making v4f16 legal will help clean this up more, but this requires additional work. llvm-svn: 332953 |
|
|
Dan Gohman
|
b81848272d |
[WebAssembly] Fix fast-isel lowering illegal argument and return types.
For both argument and return types, promote illegal types like i24 to i32, and if a type can't be easily promoted, clear out the signature before bailing out, so avoid leaving it in a partially complete state. Fixes PR37546. llvm-svn: 332947 |
|
|
Roman Tereshin
|
19da667599 |
[GlobalISel][InstructionSelect] Removing redundant num operands and nested def operands checks, perf patch 2
This patch continues a series of patches that decrease time spent by GlobalISel in its InstructionSelect pass by roughly 60% for -O0 builds for large inputs as measured on sqlite3-amalgamation (http://sqlite.org/download.html) targeting AArch64. This commit specifically removes number of operands checks that are redundant if the instruction's opcode already guarantees that number of operands (or more), and also avoids any kind of checks on a def operand of a nested instruction as everything about it was already checked at its use. The expected performance implication is about 3% off InstructionSelect comparing to the baseline (before the series of patches) This patch also contains a bit of NFC changes required for further patches in the series. Every commit planned shares the same Phabricator Review. Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar Reviewed By: qcolombet Subscribers: rovka, llvm-commits, kristof.beyls Differential Revision: https://reviews.llvm.org/D44700 llvm-svn: 332945 |
|
|
Tom Stellard
|
44b30b4537 |
AMDGPU: Remove #include "MCTargetDesc/AMDGPUMCTargetDesc.h" from common headers
Summary: MCTargetDesc/AMDGPUMCTargetDesc.h contains enums for all the instuction and register defintions, which are huge so we only want to include them where needed. This will also make it easier if we want to split the R600 and GCN definitions into separate tablegenerated files. I was unable to remove AMDGPUMCTargetDesc.h from SIMachineFunctionInfo.h because it uses some enums from the header to initialize default values for the SIMachineFunction class, so I ended up having to remove includes of SIMachineFunctionInfo.h from headers too. Reviewers: arsenm, nhaehnle Reviewed By: nhaehnle Subscribers: MatzeB, kzhuravl, wdng, yaxunl, dstuttard, tpr, t-tye, javed.absar, llvm-commits Differential Revision: https://reviews.llvm.org/D46272 llvm-svn: 332930 |
|
|
Peter Collingbourne
|
2d3eaeb2d4 |
MC: Remove dead code. NFCI.
This code appears to have been copied from the mach-o streamer. It has no effect in ELF because indirect symbols are specific to mach-o. llvm-svn: 332926 |
|
|
Paul Semel
|
31a212d694 |
Revert "[llvm-objcopy] Add --strip-unneeded option"
There is a use after free I didn't see. Need to investigate. This reverts commit f7624abeb1f0d012309baf2e78cf2499fbfe5e5f. llvm-svn: 332925 |
|
|
Petr Hosek
|
0681d9e6d9 |
[CMake] Pass Clang defaults to runtimes builds
This enables the use of Clang default options from runtimes CMake files. Differential Revision: https://reviews.llvm.org/D47168 llvm-svn: 332923 |
|
|
Sanjay Patel
|
17a870f07c |
[DAG] fold FP binops with undef operands to NaN
This is the FP sibling of D43141 with the corresponding IR change in rL327212. We can't propagate undef here because if a variable operand is a NaN, these binops must propagate NaN. Neither global nor node-level fast-math makes a difference. If we have 'nnan', I think later folds can turn the NaN into undef. The tests in X86/fp-undef.ll are meant to be the definitive verification for these folds - everything reduces identically now. The other test changes are collateral damage. They may need to be altered to preserve their intent. Differential Revision: https://reviews.llvm.org/D47026 llvm-svn: 332920 |
|
|
Lang Hames
|
5cb2e30c98 |
[ORC] Make some more operations on VSO private. These should be done via a
MaterializationResponsibility instance. llvm-svn: 332919 |
|
|
Lang Hames
|
373f4628a5 |
[LKH] Add a replacement RTDyldLayer.
llvm-svn: 332918 |
|
|
Roman Tereshin
|
f1aa348b31 |
Reapply r332907 "[GlobalISel] Improving InstructionSelect's performance by reducing MatchTable..."
Apparently the compile time problem was caused by the fact that not all compilers / STL implementations can automatically convert std::unique_ptr<Derived> to std::unique_ptr<Base>. Fixed (hopefully) by making sure it's std::unique_ptr<Derived>&& (rvalue ref) to std::unique_ptr<Base> conversion instead. llvm-svn: 332917 |
|
|
Craig Topper
|
358b094971 |
[X86] Remove 128/256-bit cvtdq2ps, cvtudq2ps, cvtqq2pd, cvtuqq2pd intrinsics.
These can all be implemented with sitofp/uitofp instructions. llvm-svn: 332916 |
|
|
Paul Semel
|
040df77ed6 |
[llvm-objcopy] Add --strip-unneeded option
This option removes symbols that are not needed by relocations. Differential Revision: https://reviews.llvm.org/D46896 llvm-svn: 332915 |
|
|
Roman Tereshin
|
8bdf7be5bb |
Revert r332907 "[GlobalISel] Improving InstructionSelect's performance by reducing MatchTable..."
There is a compile time error I didn't see locally, investigating now. llvm-svn: 332912 |
|
|
Diego Caballero
|
1bd5f2261d |
Fix warning from r332654 with LLVM_ATTRIBUTE_USED
r332654 tried to fix an unused function warning with a void cast. This approach worked for clang and gcc but not for MSVC. This commit replaces the void cast with the LLVM_ATTRIBUTE_USED approach. llvm-svn: 332910 |
|
|
Peter Collingbourne
|
7de496f460 |
Unbreak kaleidoscope example.
llvm-svn: 332908 |
|
|
Roman Tereshin
|
f0dc9fa934 |
[GlobalISel] Improving InstructionSelect's performance by reducing MatchTable, mostly NFC, perf patch 1
This patch starts a series of patches that decrease time spent by GlobalISel in its InstructionSelect pass by roughly 60% for -O0 builds for large inputs as measured on sqlite3-amalgamation (http://sqlite.org/download.html) targeting AArch64. The performance improvements are achieved solely by reducing the number of matching GIM_* opcodes executed by the MatchTable's interpreter during the selection by approx. a factor of 30, which also brings contribution of this particular part of the selection process to the overall runtime of InstructionSelect pass down from approx. 60-70% to 5-7%, thus making further improvements in this particular direction not very profitable. The improvements described above are expected for any target that doesn't have many complex patterns. The targets that do should strictly benefit from the changes, but by how much exactly is hard to estimate beforehand. It's also likely that such target WILL benefit from further improvements to MatchTable, most likely the ones that bring it closer to a perfect decision tree. This commit specifically is rather large mostly NFC commit that does necessary preparation work and refactoring, there will be a following series of small patches introducing a specific optimization each shortly after. This commit specifically is expected to cause a small compile time regression (around 2.5% of InstructionSelect pass time), which should be fixed by the next commit of the series. Every commit planned shares the same Phabricator Review. Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar Reviewed By: qcolombet Subscribers: rovka, llvm-commits, kristof.beyls Differential Revision: https://reviews.llvm.org/D44700 llvm-svn: 332907 |
|
|
Roman Lebedev
|
9f65d16d5d |
[DAGCombiner] isAllOnesConstantOrAllOnesSplatConstant(): look through bitcasts
Summary: As pointed out in D46528, we errneously transform cases like `xor X, -1`, even though we use said function. It's because the `-1` is actually a bitcast there. So i think we can just look through it in the function. Differential Revision: https://reviews.llvm.org/D47156 llvm-svn: 332905 |
|
|
Roman Lebedev
|
7772de25d0 |
[DAGCombine][X86][AArch64] Masked merge unfolding: vector edition.
Summary: This **appears** to be the last missing piece for the masked merge pattern handling in the backend. This is [[ https://bugs.llvm.org/show_bug.cgi?id=37104 | PR37104 ]]. [[ https://bugs.llvm.org/show_bug.cgi?id=6773 | PR6773 ]] will introduce an IR canonicalization that is likely bad for the end assembly. Previously, `andps`+`andnps` / `bsl` would be generated. (see `@out`) Now, they would no longer be generated (see `@in`), and we need to make sure that they are generated. Differential Revision: https://reviews.llvm.org/D46528 llvm-svn: 332904 |
|
|
Roman Lebedev
|
fd79bc3aa2 |
[X86][AArch64][NFC] Add tests for vector masked merge unfolding
Summary: This is [[ https://bugs.llvm.org/show_bug.cgi?id=37104 | PR37104 ]]. [[ https://bugs.llvm.org/show_bug.cgi?id=6773 | PR6773 ]] will introduce an IR canonicalization that is likely bad for the end assembly. Previously, `andps`+`andnps` / `bsl` would be generated. (see `@out`) Now, they would no longer be generated (see `@in`). Differential Revision: https://reviews.llvm.org/D46008 llvm-svn: 332903 |
|
|
Lang Hames
|
502f81e37e |
[ORC] Preserve Materializing symbol flag during resolution.
llvm-svn: 332899 |
|
|
Lang Hames
|
0b0b41fcce |
[ORC] Lookup now returns an error if any symbols are not found.
Also tightens the behavior of ExecutionSession::failQuery. Queries can usually only be failed by marking a symbol as failed-to-materialize, but ExecutionSession::failQuery provides a second route, and both routes may be executed from different threads. In the case that a query has already been failed due to a materialization error, ExecutionSession::failQuery will direct the error to ExecutionSession::reportError instead. llvm-svn: 332898 |
|
|
Lang Hames
|
add9b6805c |
[ORC] Remove the optional MaterializationResponsibility argument from lookup.
The lookup function provides blocking symbol resolution for JIT clients (not layers themselves) so it does not need to track symbol dependencies via a MaterializationResponsibility. llvm-svn: 332897 |
|
|
Lang Hames
|
1cf9987f6e |
[ORC] Add IRLayer and ObjectLayer interfaces and related MaterializationUnits.
llvm-svn: 332896 |
|
|
Craig Topper
|
25444c852a |
[DAGCombiner] Use computeKnownBits to match rotate patterns that have had their amount masking modified by simplifyDemandedBits
SimplifyDemandedBits can remove bits from the masks for the shift amounts we need to see to detect rotates. This patch uses zeroes from computeKnownBits to fill in some of these mask bits to make the match work. As currently written this calls computeKnownBits even when the mask hasn't been simplified because it made the code simpler. If we're worried about compile time performance we can improve this. I know we're talking about making a rotate intrinsic, but hopefully we can go ahead and do this change and just make sure the rotate intrinsic also handles it. Differential Revision: https://reviews.llvm.org/D47116 llvm-svn: 332895 |
|
|
Sanjay Patel
|
ec50effbd6 |
[InstCombine] regenerate checks; NFC
llvm-svn: 332894 |
|
|
Reid Kleckner
|
537917d13c |
[X86] Simplify some X86 address mode folding code, NFCI
This code should really do exactly the same thing for 32-bit x86 and 64-bit small code models, with the exception that RIP-relative addressing can't use base and index registers. llvm-svn: 332893 |
|
|
Craig Topper
|
dc3bf90447 |
[X86] Remove some unneeded check lines that I copy and pasted when I made vector tests from some scalar test cases.
llvm-svn: 332892 |
|
|
Craig Topper
|
aad3aefaeb |
[X86] Remove masking from vpternlog intrinsics. Use a select in IR instead.
This removes 6 intrinsics since we no longer need separate mask and maskz intrinsics. Differential Revision: https://reviews.llvm.org/D47124 llvm-svn: 332890 |
|
|
Peter Collingbourne
|
274c4f7ab4 |
Fix a make_unique ambiguity.
llvm-svn: 332889 |
|
|
Sanjay Patel
|
b8346e3f07 |
[InstCombine] remove fptrunc (select) code; NFCI
This pattern is handled within commonCastTransforms(), so the code here is dead AFAICT. llvm-svn: 332887 |
|
|
Peter Collingbourne
|
c5a9765cea |
LTO: Replace split dwarf implementation that uses objcopy with one that uses direct emission.
Part of PR37466. Differential Revision: https://reviews.llvm.org/D47091 llvm-svn: 332884 |
|
|
Sanjay Patel
|
94b1f846b2 |
[InstCombine] add tests for cast-of-select; NFC
In all cases, we're pulling the cast above the select. That's not a good canonicalization if we're creating a select that then mismatches the operand size of its condition. llvm-svn: 332883 |
|
|
Peter Collingbourne
|
9a45114b3c |
CodeGen: Add a dwo output file argument to addPassesToEmitFile and hook it up to dwo output.
Part of PR37466. Differential Revision: https://reviews.llvm.org/D47089 llvm-svn: 332881 |
|
|
Reid Kleckner
|
9b2df56c59 |
Remove CMake workaround for LLD PR24476 which is no longer needed
llvm-svn: 332880 |
|
|
Brian Gesiak
|
a398590f56 |
[DebugInfo] Fix typo "DWARG" in test comment (NFC)
Summary: The correct spelling is "DWARF", the debugging format, not "DWARG". The typo is in a (not executed by lit) comment in a test file, so fixing it does not result in any functional change. Test Plan: check-llvm, just in case llvm-svn: 332878 |
|
|
Peter Collingbourne
|
63062d9d0f |
MC: Introduce an ELF dwo object writer and teach llvm-mc about it.
Part of PR37466. Differential Revision: https://reviews.llvm.org/D47051 llvm-svn: 332875 |
|
|
Matt Arsenault
|
16fcc5b6db |
AMDGPU: Update GCCBuiltin names for DS FP atomic intrinsics
llvm-svn: 332874 |
|
|
Jonas Devlieghere
|
c111382aa8 |
[DebugInfo] Use absolute addresses in location lists
Rather than relying on the user to do the address calculating in DW_AT_location we should just dump the absolute address. rdar://problem/38513870 Differential revision: https://reviews.llvm.org/D47152 llvm-svn: 332873 |
|
|
Craig Topper
|
a010b3c9dc |
[X86] Add test cases for D47012.
Patch by Thomasz Krupa. llvm-svn: 332872 |
|
|
Peter Collingbourne
|
f0226e62a8 |
MC: Extract a derived class from ELFObjectWriter. NFCI.
This class will be used to create regular, non-split ELF files. Part of PR37466. Differential Revision: https://reviews.llvm.org/D47049 llvm-svn: 332870 |
|
|
Craig Topper
|
ef313905f0 |
[X86] Add test cases for missed vector rotate matching due to SimplifyDemandedBits interfering with the AND masks
As requested in D47116 llvm-svn: 332869 |
|
|
Peter Collingbourne
|
dcd7d6c331 |
MC: Separate creating a generic object writer from creating a target object writer. NFCI.
With this we gain a little flexibility in how the generic object writer is created. Part of PR37466. Differential Revision: https://reviews.llvm.org/D47045 llvm-svn: 332868 |
|
|
Peter Collingbourne
|
a29fe579f4 |
MC: Extract ELFObjectWriter's ELF writing functionality into an ELFWriter class. NFCI.
The idea is that we will be able to use this class to create multiple files. Differential Revision: https://reviews.llvm.org/D47048 llvm-svn: 332867 |
|
|
Peter Collingbourne
|
2602a0d40c |
Fix ubsan bounds check failure.
llvm-svn: 332866 |
|
|
Craig Topper
|
f14e62c9a5 |
[EarlyCSE] Improve EarlyCSE of some absolute value cases.
Change matchSelectPattern to return X and -X for ABS/NABS in a well defined order. Adjust EarlyCSE to account for this. Ensure the SPF result is some kind of min/max and not abs/nabs in one place in InstCombine that made me nervous. Prevously we returned the two operands of the compare part of the abs pattern. The RHS is always going to be a 0i, 1 or -1 constant. This isn't a very meaningful thing to return for any one. There's also some freedom in the abs pattern as to what happens when the value is equal to 0. This freedom led to early cse failing to match when different constants were used in otherwise equivalent operations. By returning the input and its negation in a defined order we can ensure an exact match. This also makes sure both patterns use the exact same subtract instruction for the negation. I believe CSE should evebntually make this happen and properly merge the nsw/nuw flags. But I'm not familiar with CSE and what order it does things in so it seemed like it might be good to really enforce that they were the same. Differential Revision: https://reviews.llvm.org/D47037 llvm-svn: 332865 |
|
|
Peter Collingbourne
|
59a6fc469f |
MC: Remove stream and output functions from MCObjectWriter. NFCI.
Part of PR37466. Differential Revision: https://reviews.llvm.org/D47043 llvm-svn: 332864 |
|
|
Peter Collingbourne
|
438390fae1 |
MC: Have the object writers return the number of bytes written. NFCI.
This removes the last external use of the stream. Part of PR37466. Differential Revision: https://reviews.llvm.org/D47042 llvm-svn: 332863 |
|
|
Stanislav Mekhanoshin
|
9badad2051 |
[AMDGPU] Add divergence analysis as a dependency for ISel
AMDGPUDAGToDAGISel adds DivergenceAnalysis in getAnalysisUsage but does not list it in pass dependencies which may lead to crash. Differential Revision: https://reviews.llvm.org/D47151 llvm-svn: 332862 |
|
|
Peter Collingbourne
|
f17b149d8c |
MC: Change object writers to use endian::Writer. NFCI.
Part of PR37466. Differential Revision: https://reviews.llvm.org/D47040 llvm-svn: 332861 |
|
|
Diego Caballero
|
168d04d544 |
[VPlan] Reland r332654 and silence unused func warning
r332654 was reverted due to an unused function warning in release build. This commit includes the same code with the warning silenced. Differential Revision: https://reviews.llvm.org/D44338 llvm-svn: 332860 |
|
|
Peter Collingbourne
|
147db3e628 |
MC: Change MCAssembler::writeSectionData and writeFragmentPadding to take a raw_ostream. NFCI.
Also clean up a couple of hacks where we were writing the section contents to another stream by setting the object writer's stream, writing and setting it back. Part of PR37466. Differential Revision: https://reviews.llvm.org/D47038 llvm-svn: 332858 |
|
|
Peter Collingbourne
|
571a3301ae |
MC: Change MCAsmBackend::writeNopData() to take a raw_ostream instead of an MCObjectWriter. NFCI.
To make this work I needed to add an endianness field to MCAsmBackend so that writeNopData() implementations know which endianness to use. Part of PR37466. Differential Revision: https://reviews.llvm.org/D47035 llvm-svn: 332857 |
|
|
Tom Stellard
|
a91ce17b5f |
AMDGPU/GlobalISel: Address post-commit review comments for r332379
MCRegisterInfo::getPhysRegSize() will be deprecated. llvm-svn: 332856 |
|
|
Alexey Bataev
|
7c9ad0db3d |
[InstCombine] Fix PR37526: MinMax patterns produce an infinite loop.
Summary: This patch fixes PR37526 by simplifying the newly generated LoadInst instructions. If the pointer address is a bitcast from the pointer to the NewType, we can just remove this extra bitcast instead of creating the new one. This fixes the PR37526 + may speed up the whole compilation process. Reviewers: spatel, RKSimon, hfinkel Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D47144 llvm-svn: 332855 |
|
|
Andrea Di Biagio
|
cb1ed400a4 |
[llvm-mca] Removed an empty line generated by the timeline view. NFC.
Also, regenerate all tests. llvm-svn: 332853 |
|
|
Andrea Di Biagio
|
b5757abefb |
[X86][BtVer2] Add a 'J' prefix to the PRF/RCU defs. NFC
This is to keep the Jaguar model's naming convention. Processor resources all have a 'J' prefix in the BtVer2 scheduling model. llvm-svn: 332851 |
|
|
Robert Widmann
|
38fa750b7a |
[LLVM-C] Add DIBuilder Bindings For ObjC Classes
Summary: Add LLVMDIBuilderCreateObjCIVar, LLVMDIBuilderCreateObjCProperty, and LLVMDIBuilderCreateInheritance to allow declaring metadata for Objective-C class hierarchies and their associated properties and instance variables. Reviewers: whitequark, deadalnix Reviewed By: whitequark Subscribers: harlanhaskins, llvm-commits Differential Revision: https://reviews.llvm.org/D47123 llvm-svn: 332850 |
|
|
Lama Saba
|
9417f7ff2e |
[X86] - Avoid SFB pass - fix bug in updating the offsets for newly created copies
Change-Id: I169ab6fe7e187727c0298c2a1e2868a683f3e688 llvm-svn: 332849 |
|
|
James Henderson
|
004b729ed1 |
[DWARF] Refactor callback usage for .debug_line error handling
Change the "recoverable" error callback to take an Error instaed of a string. Reviewed by: JDevlieghere Differential Revision: https://reviews.llvm.org/D46831 llvm-svn: 332845 |
|
|
Simon Pilgrim
|
a8869e68a9 |
[X86][SSE] Add an assert to ensure that rotation amount is converted to a scale
Missed in rL332832 where we added SSE v4i32 rotations for PR37426. llvm-svn: 332844 |
|
|
Tim Northover
|
4e3eec39fa |
ARM: be conservative when asked load/store alignment of weird type.
Chances are we'll be asked again after type legalization, but before that point it's better to claim misaligned accesses aren't allowed than to assert. llvm-svn: 332840 |
|
|
Nico Weber
|
e4a12cfa2f |
revert r332610, it breaks cfi, see D46326
llvm-svn: 332838 |
|
|
Aleksandar Beserminji
|
4977705727 |
[mips] Revert Merge MipsLongBranch and MipsHazardSchedule passes
Revert this patch due buildbot failure. Differential Revision: https://reviews.llvm.org/D46641 llvm-svn: 332837 |
|
|
David Green
|
8ceab61c75 |
[CVP] Require DomTree for new Pass Manager
We were previously using a DT in CVP through SimplifyQuery, but not requiring it in the new pass manager. Hence it would crash if DT was not already available. This now gets DT directly and plumbs it through to where it is used (instead of using it through SQ). llvm-svn: 332836 |
|
|
Eric Christopher
|
563d0b9cb9 |
Fix up a few grammar issues.
llvm-svn: 332835 |
|
|
Aleksandar Beserminji
|
de7be5e46f |
[mips] Merge MipsLongBranch and MipsHazardSchedule passes
MipsLongBranchPass and MipsHazardSchedule passes are joined to one pass because of mutual conflict. When MipsHazardSchedule inserts 'nop's, it potentially breaks some jumps, so they have to be expanded to long branches. When some branch is expanded to long branch, it potentially creates a hazard situation, which should be fixed by adding nops. New pass is called MipsBranchExpansion, it combines these two passes, and runs them alternately until one of them reports no changes were made. Differential Revision: https://reviews.llvm.org/D46641 llvm-svn: 332834 |
|
|
Simon Pilgrim
|
5aa7cdfd70 |
[X86][SSE] Support v4i32 rotations (PR37426)
As suggested by Fabian on PR37426, we can use PMULUDQ to perform v4i32 vector rotations as the upper 32bits of the multiply will contain the 'wrapped' bits of the rotation. v8i16/v16i8 rotations would be straightforward to add to lowerRotate in the future - ideally we'd mostly share code with the vector shifts lowering. Differential Revision: https://reviews.llvm.org/D46954 llvm-svn: 332832 |
|
|
Nico Weber
|
d418776e04 |
win: try more to fix dia tests with newer msvc versions
llvm-svn: 332828 |
|
|
Nico Weber
|
da5513b9c4 |
win: try to fix dia tests with newer msvc versions
llvm-svn: 332827 |
|
|
Robert Widmann
|
360d6e35e6 |
[LLVM-C] Improve Bindings For Aliases
Summary: Add wrappers for a module's alias iterators and a getter and setter for the aliasee value. Reviewers: whitequark, deadalnix Reviewed By: whitequark Subscribers: llvm-commits, harlanhaskins Differential Revision: https://reviews.llvm.org/D46808 llvm-svn: 332826 |
|
|
Craig Topper
|
e4c045b7df |
[X86] Remove mask arguments from permvar builtins/intrinsics. Use a select in IR instead.
Someday maybe we'll use selects for all intrinsics. llvm-svn: 332824 |
|
|
Nico Weber
|
41597b92b1 |
Revert 332750, llvm part (see comment on D46910).
llvm-svn: 332823 |
|
|
Simon Dardis
|
777afc7fbd |
[mips] Add microMIPSR6 ll/sc instructions.
Previously the compiler was using the microMIPSR3 variants, incorrectly. Reviewers: atanasyan, abeserminji, smaksimovic Differential Revision: https://reviews.llvm.org/D46948 llvm-svn: 332820 |
|
|
Sanjay Patel
|
a003c728a5 |
[InstCombine] choose 1 form of abs and nabs as canonical
We already do this for min/max (see the blob above the diff), so we should do the same for abs/nabs. A sign-bit check (<s 0) is used as a predicate for other IR transforms and it's likely the best for codegen. This might solve the motivating cases for D47037 and D47041, but I think those patches still make sense. We can't guarantee this canonicalization if the icmp has more than one use. Differential Revision: https://reviews.llvm.org/D47076 llvm-svn: 332819 |
|
|
Vassil Vassilev
|
ce73760755 |
[cmake] Add a switch to enable/disable bindings.
Differential Revision: https://reviews.llvm.org/D42026 llvm-svn: 332816 |
|
|
Craig Topper
|
9ed890b0db |
[X86] Add test cases to show missed rotate opportunities due to SimplifyDemandedBits.
llvm-svn: 332815 |
|
|
Haicheng Wu
|
69ba0613f2 |
[GlobalMerge] Exit early if only one global is to be merged
To save some compilation time and prevent some unnecessary changes. Differential Revision: https://reviews.llvm.org/D46640 llvm-svn: 332813 |
|
|
Brian Gesiak
|
9968e0dd49 |
Re-revert "[Option] Fix PR37006 prefix choice in findNearest"
Summary: Reverting due to a test failure in an llvm-mt test on some buildbots, namely http://green.lab.llvm.org/green/job/clang-stage2-configure-Rlto/26020/. llvm-svn: 332812 |
|
|
Robert Widmann
|
025c78f5d7 |
[LLVM-C] Use Length-Providing Value Name Getters and Setters
Summary: - Provide LLVMGetValueName2 and LLVMSetValueName2 that return and take the length of the provided C string respectively - Deprecate LLVMGetValueName and LLVMSetValueName Reviewers: whitequark, deadalnix Reviewed By: whitequark Subscribers: llvm-commits, harlanhaskins Differential Revision: https://reviews.llvm.org/D46890 llvm-svn: 332810 |
|
|
Max Kazantsev
|
c0b268f90c |
[IRCE] Fix miscompile with range checks against negative values
In the patch rL329547, we have lifted the over-restrictive limitation on collected range checks, allowing to work with range checks with the end of their range not being provably non-negative. However it appeared that the non-negativity of this value was assumed in the utility function `ClampedSubtract`. In particular, its reasoning is based on the fact that `0 <= SINT_MAX - X`, which is not true if `X` is negative. The function `ClampedSubtract` is only called twice, once with `X = 0` (which is OK) and the second time with `X = IRC.getEnd()`, where we may now see the problem if the end is actually a negative value. In this case, we may sometimes miscompile. This patch is the conservative fix of the miscompile problem. Rather than rejecting non-provably non-negative `getEnd()` values, we will check it for non-negativity in runtime. For this, we use function `smax(smin(X, 0), -1) + 1` that is equal to `1` if `X` is non-negative and is equal to 0 if `X` is negative. If we multiply `Begin, End` of safe iteration space by this function calculated for `X = IRC.getEnd()`, we will get the original `[Begin, End)` if `IRC.getEnd()` was non-negative (and, thus, `ClampedSubtract` worked correctly) and the empty range `[0, 0)` in case if ` IRC.getEnd()` was negative. So we in fact prohibit execution of the main loop if at least one of range checks was made against a negative value (and we figured it out in runtime). It is still better than what we have before (non-negativity had to be proved in compile time) and prevents us from miscompile, however it is sometiles too restrictive for unsigned range checks against a negative value (which in fact can be eliminated). Once we re-implement `ClampedSubtract` in a way that it handles negative `X` correctly, this limitation can be lifted, too. Differential Revision: https://reviews.llvm.org/D46860 Reviewed By: samparker llvm-svn: 332809 |
|
|
Benjamin Kramer
|
a76b64ff80 |
[MergeICmps] Don't crash when memcmp is not available
Fixes clang crashing with -fno-builtin, PR37527. llvm-svn: 332808 |
|
|
Simon Pilgrim
|
ede0e4073e |
Fix MSVC unused variable warning. NFCI.
AMDGPURegisterInfo::getSubRegFromChannel is a static method - we don't need to get the AMDGPURegisterInfo instance. llvm-svn: 332807 |
|
|
Brian Gesiak
|
8cfb4b6d41 |
Un-revert "[Option] Fix PR37006 prefix choice in findNearest"
Summary: In https://reviews.llvm.org/rL332804 I loosed the assertion in the Clang driver test that forced me to revert https://reviews.llvm.org/rL332299. Once this lands I should be able to narrow down what caused PS4 buildbots to fail, and reinstate the check in that test. Test Plan: check-llvm & check-clang llvm-svn: 332805 |
|
|
Nico Weber
|
5533357c38 |
Fix build warning compiling TestPlugin on Windows and disable Passes plugin stuff on Windows since it fundamentally can't work
Aaron Ballman reported that TestPlugin warned about it using exception handling without /EHsc flag, and that llvmGetPassInfo() had conflicting export attributes (dllimport in the header, dllexport in the source file). /EHsc is because TestPlugin didn't use the llvm_ cmake functions, so llvm_update_compile_flags didn't get called for the target (llvm_update_compile_flags explicitly passes /Ehs-c-, which fixes the warning). Use add_llvm_loadable_module instead of add_library(... MODULE) to fix this. This also has the side effect of not building the plugin on Windows. That's not a big problem, since before the plugin was built on Windows, but the test didn't attempt to load it, due to -DLLVM_ENABLE_PLUGIN not being passed to PluginsTests.cpp during compilation on Windows. This makes the plugin behavior consistent with e.g. lib/Transforms/Hello/CMakeLists.txt. (This also automatically sets LTDL_SHLIB_EXT correctly.) The dllimport/dllexport warning is more serious: Since LLVM doesn't generally use export annotations for its code, the only way the plugin could link was by linking in some LLVM libraries both into the test and the dll, so the plugin would call the llvm code in the dll instead of the copy in the main executable. This means globals weren't shared, and things generally can't work. (I think there's a build config where you can build a LLVM.dll which might work, but that wasn't how the test was configured. If that config is used, the dll should still be built, but I haven't checked). Now that add_llvm_loadable_module is used, LLVM_LINK_COMPONENTS got linked into both executable and plugin on posix too, so unset it after the executable so that the plugin doesn't end up with a 2nd copy of things on posix. https://reviews.llvm.org/D47082 llvm-svn: 332796 |
|
|
Yaxun Liu
|
ea988f1fd9 |
Fix evaluator for non-zero alloca addr space
The evaluator goes through BB and creates global vars as temporary values to evaluate results of LLVM instructions. It creates undef for alloca, however it assumes alloca in addr space 0. If the next instruction is addrspace cast to 0, then we get an invalid cast instruction. This patch let the temp global var have an address space matching alloca addr space, so that the valuation can be done. Differential Revision: https://reviews.llvm.org/D47081 llvm-svn: 332794 |
|
|
Nico Weber
|
429e06e76b |
Enable colored diagnostics in ninja builds when building with gcc 4.9+.
GCC has supported -fdiagnostics-color since 4.9. https://reviews.llvm.org/D47083 llvm-svn: 332793 |
|
|
Piotr Padlewski
|
5642a42442 |
Propagate nonnull and dereferenceable throught launder
Summary: invariant.group.launder should not stop propagation of nonnull and dereferenceable, because e.g. we would not be able to hoist loads speculatively. Reviewers: rsmith, amharc, kuhar, xbolva00, hfinkel Subscribers: hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D46972 llvm-svn: 332788 |
|
|
Piotr Padlewski
|
ce358262eb |
Dissallow non-empty metadata for invariant.group
Summary: This feature is not needed, but it might be usefull in the future to use metadata to mark what which function should support it (and strip it when not). Reviewers: rsmith, sanjoy, amharc, kuhar Subscribers: hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D45419 llvm-svn: 332787 |
|
|
Piotr Padlewski
|
a26a08cb52 |
Constant fold launder of null and undef
Summary: This might be useful because clang will add some barriers for pointer comparisons. Reviewers: majnemer, dberlin, hfinkel, nlewycky, davide, rsmith, amharc, kuhar Subscribers: davide, amharc, llvm-commits Differential Revision: https://reviews.llvm.org/D32423 llvm-svn: 332786 |
|
|
Piotr Padlewski
|
153fe60079 |
[MemDep] Fixed handling of invariant.group
Summary: Memdep had funny bug related to invariant.groups - because it did not invalidated cache, in some very rare cases it was possible to show memory dependence of the instruction that was deleted, but because other instruction took it's place it resulted in call to vtable! Thanks @amharc for repro!. Reviewers: dberlin, kuhar, amharc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D45320 Co-authored-by: Krzysztof Pszeniczny <krzysztof.pszeniczny@gmail.com> llvm-svn: 332781 |
|
|
Sanjay Patel
|
b41a5affea |
[x86] add more FP with FMF simplification tests; NFC
llvm-svn: 332780 |
|
|
Matt Arsenault
|
9fc8593a77 |
DAG: Fix crash on shift with large shift amounts
Fixes bug 37521. llvm-svn: 332774 |
|
|
Wolfgang Pieb
|
20e1546655 |
Fixing buildbot error introduced with r332759.
llvm-svn: 332772 |
|
|
Matt Arsenault
|
372d796ab1 |
AMDGPU: Add pass to optimize reqd_work_group_size
Eliminate loads from the dispatch packet when they will have a known value. Also pattern match the code used by the library to handle partial workgroup dispatches, which isn't necessary if reqd_work_group_size is used. llvm-svn: 332771 |
|
|
Craig Topper
|
0198b73769 |
[InstCombine] Qualify a select pattern based transform to restrct to only min/max and ignore abs/nabs.
llvm-svn: 332770 |
|
|
Sam Clegg
|
4bbc6b55e7 |
[WebAssembly] Object: Add more error checking for object file reading
This should address some the assert failures the fuzzer has been finding such as: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=6719 Differential Revision: https://reviews.llvm.org/D47046 llvm-svn: 332769 |
|
|
Wolfgang Pieb
|
401b5ecfea |
Addressing a couple of compiler warnings introduced with r332759.
llvm-svn: 332766 |
|
|
Wolfgang Pieb
|
da71639cdb |
Fixing build error introduced with r332759.
llvm-svn: 332762 |
|
|
Evgeniy Stepanov
|
28f330fd6f |
[msan] Don't check divisor shadow in fdiv.
Summary: Floating point division by zero or even undef does not have undefined behavior and may occur due to optimizations. Fixes https://bugs.llvm.org/show_bug.cgi?id=37523. Reviewers: kcc Subscribers: hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D47085 llvm-svn: 332761 |
|
|
Wolfgang Pieb
|
ad60559be7 |
[DWARF v5] Improved support for .debug_rnglists (consumer). Enables any consumer to
extract DWARF v5 encoded rangelists. Reviewer: JDevlieghere Differential Revision: https://reviews.llvm.org/D45549 llvm-svn: 332759 |
|
|
Jessica Paquette
|
c604817493 |
[NFC] Change cast from r332739 to a static cast
The casts in the delta computation for size remarks should have been static casts. This fixes that. Thanks to Dávid Bolvanský for pointing that out. llvm-svn: 332758 |
|
|
Peter Collingbourne
|
e3f652973e |
Support: Simplify endian stream interface. NFCI.
Provide some free functions to reduce verbosity of endian-writing a single value, and replace the endianness template parameter with a field. Part of PR37466. Differential Revision: https://reviews.llvm.org/D47032 llvm-svn: 332757 |
|
|
Michael Berg
|
1fa76cc3ea |
adding baseline fp fold tests for unsafe on and off
llvm-svn: 332756 |
|
|
Amara Emerson
|
08099c7edd |
Delete a test that was missed in the revert r332747.
r332747 originally reverted r332654 which added this test. llvm-svn: 332755 |
|
|
Konstantin Zhuravlyov
|
caa8251971 |
AMDGPU/NFC: Set symbol's type that is coming from an argument in
EmitAMDGPUSymbolType, instead of hard-coding it to STT_AMDGPU_HSA_KERNEL. llvm-svn: 332753 |
|
|
Petr Hosek
|
24b61ac832 |
[Support] Avoid normalization in sys::getDefaultTargetTriple
The return value of sys::getDefaultTargetTriple, which is derived from -DLLVM_DEFAULT_TRIPLE, is used to construct tool names, default target, and in the future also to control the search path directly; as such it should be used textually, without interpretation by LLVM. Normalization of this value may lead to unexpected results, for example if we configure LLVM with -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-linux-gnu, normalization will transform that value to x86_64--linux-gnu. Driver will use that value to search for tools prefixed with x86_64--linux-gnu- which may be confusing. This is also inconsistent with the behavior of the --target flag which is taken as-is without any normalization and overrides the value of LLVM_DEFAULT_TARGET_TRIPLE. Users of sys::getDefaultTargetTriple already perform their own normalization as needed, so this change shouldn't impact existing logic. Differential Revision: https://reviews.llvm.org/D46910 llvm-svn: 332750 |