Commit Graph

4533 Commits

Author SHA1 Message Date
Nicolai Haehnle ab390f0c41 TableGen/DAGPatterns: Allow bit constants in addition to int constants
Summary:
Implicit casting is a simple quality of life improvement.

Change-Id: I3d2b31b8b8f12cbb1e84f691e359fa713a9c4b42

Reviewers: tra, simon_tatham, craig.topper, MartinO, arsenm

Subscribers: wdng, llvm-commits

Differential Revision: https://reviews.llvm.org/D47432

llvm-svn: 333904
2018-06-04 14:45:12 +00:00
Andrea Di Biagio be8616f5f2 [MCSchedule] Add the ability to compute the latency and throughput information for MCInst.
This patch extends the MCSchedModel API with new methods that can be used to
obtain the latency and reciprocal througput information for an MCInst.

Scheduling models have recently gained the ability to resolve variant scheduling
classes associated with MCInst objects. Before, models were only able to resolve
a variant scheduling class from a MachineInstr object.

This patch is mainly required by D47374 to avoid regressing a pair of x86
specific -print-schedule tests for btver2. Patch D47374 introduces a new variant
class to teach the btver scheduling model (x86 target) how to correctly compute
the latency profile for some zero-idioms using the new scheduling predicates.

The new methods added by this patch would be mainly used by llc when flag
-print-schedule is specified. In particular, tests that contain inline assembly
require that code is parsed at code emission stage into a sequence of MCInst.
That forces the print-schedule functionality to query the latency/rthroughput
information for MCInst instructions too. If we don't expose this new API, then
we lose "-print-schedule" test coverage as soon as variant scheduling classes
are added to the x86 models.

The tablegen SubtargetEmitter changes teaches how to query latency profile
information using a object that derives from TargetSubtargetInfo. Note that this
should really have been part of r333286. To avoid code duplication, the logic
that "resolves" variant scheduling classes for MCInst, has been moved to a
common place in MC. That logic is used by the "resolveVariantSchedClass" methods
redefined in override by the tablegen'd GenSubtargetInfo classes.

Differential Revision: https://reviews.llvm.org/D47536

llvm-svn: 333650
2018-05-31 13:30:42 +00:00
Florian Hahn 75e87c3f2a [TableGen] Avoid leaking TreePatternNodes by using shared_ptr.
By using std::shared_ptr for TreePatternNode, we can avoid leaking them.

Reviewers: craig.topper, dsanders, stoklund, tstellar, zturner

Reviewed By: dsanders

Differential Revision: https://reviews.llvm.org/D47463

llvm-svn: 333591
2018-05-30 21:00:18 +00:00
Florian Hahn 33b6f9acc4 [TableGen] Use explicit constructor for InstMemo
This should fix a few buildbot failures with old
GCC versions.

llvm-svn: 333448
2018-05-29 18:34:42 +00:00
Florian Hahn 7d3f9a88b9 [TableGen] Fix leaking of PhysRegInputs.
Instead of dynamically allocating the vector for PhysRegs, we can
allocate it on the stack and move it into InstructionMemo.

Reviewers: mcrosier, craig.topper, RKSimon, dsanders

Reviewed By: dsanders

Differential Revision: https://reviews.llvm.org/D47461

llvm-svn: 333438
2018-05-29 17:40:03 +00:00
Florian Hahn 6c21b3b595 [TableGen] Fix leaking synthesized registers.
By keeping track of unique_ptrs to the synthesized definitions in
CodeGenRegBank we avoid leaking them.

Reviewers: dsanders, kparzysz, stoklund

Reviewed By: dsanders

Differential Revision: https://reviews.llvm.org/D47462

llvm-svn: 333434
2018-05-29 16:55:06 +00:00
Andrea Di Biagio df8e919957 [Tablegen] Avoid generating empty switch statements. NFC
This fixes an MSVC warning (warning C4065: switch statement contains 'default'
but no 'case' labels) introduced with revision 333293.

llvm-svn: 333363
2018-05-27 19:08:12 +00:00
Andrea Di Biagio 6c40057bd0 [Tablegen][SubtargetEmitter] Add a default case to the auto-generated switch in MCSubtargetInfo::resolveVariantSchedClass(). NFC
llvm-svn: 333293
2018-05-25 18:25:54 +00:00
Andrea Di Biagio 8f66adecd0 [RFC][Patch 2/3] Add a MCSubtargetInfo hook to resolve variant scheduling classes.
This patch is the second of a sequence of three patches related to LLVM-dev RFC
"MC support for varinat scheduling classes".
https://lists.llvm.org/pipermail/llvm-dev/2018-May/123181.html

The goal of this patch is to enable the resolution of variant classes in MC with
the help of a new method named `MCSubtargetInfo::resolveVariantSchedClass()`.

This patch also teaches the SubtargetEmitter how to automatically generate the
definition of method resolveVariantSchedClass().  That definition is emitted
within a sub-class of MCSubtargetInfo named XXXGenMCSubtargetInfo (where XXX is
the name of the Target).

Differential Revision: https://reviews.llvm.org/D47077

llvm-svn: 333286
2018-05-25 16:02:43 +00:00
Andrea Di Biagio 95140023b5 [RFC][Patch 1/3] Add a new class of predicates for variant scheduling classes.
This patch is the first of a sequence of three patches described by the LLVM-dev
RFC "MC support for variant scheduling classes".
http://lists.llvm.org/pipermail/llvm-dev/2018-May/123181.html

The goal of this patch is to introduce a new class of scheduling predicates for
SchedReadVariant and SchedWriteVariant.

An MCSchedPredicate can be used instead of a normal SchedPredicate to model
checks on the instruction (either a MachineInstr or a MCInst).
Internally, an MCSchedPredicate encapsulates an MCInstPredicate definition.
MCInstPredicate allows the definition of expressions with a well-known semantic,
that can be used to generate code for both MachineInstr and MCInst.

This is the first step toward teaching to tools like lllvm-mca how to resolve
variant scheduling classes.

Differential Revision: https://reviews.llvm.org/D46695

llvm-svn: 333282
2018-05-25 15:55:37 +00:00
Roman Tereshin a4c410d50d [GlobalISel][InstructionSelect] Switching over root LLTs, perf patch 10
This patch continues a series of patches started by r332907 (reapplied
as r332917).

In this commit we introduce new matching opcode for the MatchTable:
GIM_SwitchType, similar to GIM_SwitchOpcode, and use it to switch over
LLTs of def operands of root instructions on the 2nd level of the
MatchTable within GIM_SwitchOpcode's cases.

This is expected to decrease time GlobalISel spends in its
InstructionSelect pass by about 6.5% for an -O0 build as measured on
sqlite3-amalgamation (http://sqlite.org/download.html) targeting
AArch64 (cross-compile on x86).

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333146
2018-05-24 00:24:15 +00:00
Roman Tereshin 5f5e55008f [GlobalISel][InstructionSelect] Moving Reg Bank Checks forward, perf patch 9
This patch continues a series of patches started by r332907 (reapplied
as r332917).

In this commit we move register bank checks back from epilogue of
every rule matcher to a position locally close to the rest of the
checks for a particular (nested) instruction.

This increases the number of common conditions within 2nd level
groups.

This is expected to decrease time GlobalISel spends in its
InstructionSelect pass by about 2% for an -O0 build as measured on
sqlite3-amalgamation (http://sqlite.org/download.html) targeting
AArch64 (cross-compile on x86).

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333144
2018-05-23 23:58:10 +00:00
Roman Tereshin 152fc1605e [GlobalISel][InstructionSelect] Maximizing # of Group's common conditions, perf patch 8
This patch continues a series of patches started by r332907 (reapplied
as r332917).

In this commit we greedily stuff 2nd level GroupMatcher's common
conditions with as many predicates as possible. This is purely
post-processing and it doesn't change which rules are put into the
groups in the first place: that decision is made by looking at the
first common predicate only.

The compile time improvements are minor and well within error margin,
however, it's highly improbable that this transformation could
pessimize performance, thus I'm still committing it for potential
gains for targets not implementing GlobalISel yet and out of tree
targets.

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333139
2018-05-23 22:50:53 +00:00
Roman Tereshin 9e4931834b [GlobalISel][Tablegen] Assign small opcodes to pseudos
Sort pseudo instructions first while emitting enum's for target
instructions info. That puts them close to each other and to generic
G_* opcodes for GlobalISel. This makes it easier to build small jump
tables over opcodes that could be directly embedded into MatchTable's
Tablegen'erated for GlobalISel's InstructionSelect.

Reviewed By: bogner

Differential Revision: https://reviews.llvm.org/D47240

llvm-svn: 333135
2018-05-23 22:10:21 +00:00
Roman Tereshin 9a9fa49cd2 [GlobalISel][InstructionSelect] Sorting MatchTable's 2nd level by root LLT, perf patch 7
This patch continues a series of patches started by r332907 (reapplied
as r332917).

In this commit we sort rules within their 2nd level by the type check
on def operand of the root instruction, which allows for better
nesting grouping on the level.

This is expected to decrease time GlobalISel spends in its
InstructionSelect pass by roughly 22% for an -O0 build as measured on
sqlite3-amalgamation (http://sqlite.org/download.html) targeting
AArch64 (cross-compile on x86).

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333131
2018-05-23 21:30:16 +00:00
Roman Tereshin d760c20c94 [Tablegen] Tidying up InstRegexOp a little, NFC
Differential Review: https://reviews.llvm.org/D47240

llvm-svn: 333121
2018-05-23 20:45:43 +00:00
Roman Tereshin b1ba127aa8 [GlobalISel][InstructionSelect] Moving type checks forward, perf patch 6
This patch continues a series of patches started by r332907 (reapplied
as r332917)

In this commit we sort type checks towards the beginning of every rule
within the MatchTable as they fail often and it's best to fail early.

This is expected to decrease time GlobalISel spends in its
InstructionSelect pass by roughly 7% for an -O0 build as measured on
sqlite3-amalgamation (http://sqlite.org/download.html) targeting
AArch64. The amalgamation is a large single-file C-source that makes
compiler backend performance improvements to stand out from frontend.
It's also a part of CTMark.

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333114
2018-05-23 19:16:59 +00:00
Petar Jovanovic c051000b83 [X86][MIPS][ARM] New machine instruction property 'isMoveReg'
This property is needed in order to follow values movement between
registers. This property is used in TII to implement method that
returns true if simple copy like instruction is recognized, along
with source and destination machine operands.

Patch by Nikola Prica.

Differential Revision: https://reviews.llvm.org/D45204

llvm-svn: 333093
2018-05-23 15:28:28 +00:00
Nicola Zaghen 03d0b91f43 Remove DEBUG macro.
Now that the LLVM_DEBUG() macro landed on the various sub-projects
the DEBUG macro can be removed.
Also change the new uses of DEBUG to LLVM_DEBUG.

Differential Revision: https://reviews.llvm.org/D46952

llvm-svn: 333091
2018-05-23 15:09:29 +00:00
Roman Tereshin fedae33efa [GlobalISel][InstructionSelect] MatchTable second level grouping, perf patch 5
This patch continues a series of patches started by r332907 (reapplied
as r332917)

In this commit we start grouping rules with common first condition on
the second level of the table.

This is expected to decrease time GlobalISel spends in its
InstructionSelect pass by roughly 13% for an -O0 build as measured on
sqlite3-amalgamation (http://sqlite.org/download.html) targeting
AArch64.

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333053
2018-05-23 02:04:19 +00:00
Roman Tereshin 0ee082f3b9 [GlobalISel][InstructionSelect] Switching MatchTable over opcodes, perf patch 4
This patch continues a series of patches started by r332907 (reapplied
as r332917)

In this commit we introduce a new matching opcode GIM_SwitchOpcode
that implements a jump table over opcodes and start emitting them for
root instructions.

This is expected to decrease time GlobalISel spends in its
InstructionSelect pass by roughly 20% for an -O0 build as measured on
sqlite3-amalgamation (http://sqlite.org/download.html) targeting
AArch64.

To some degree, we assume here that the opcodes form a dense set,
which is true at the moment for all upstream targets given the
limitations of our rule importing mechanism.

It might not be true for out of tree targets, specifically due to
pseudo's. If so, we might noticeably increase the size of the
MatchTable with this patch due to padding zeros. This will be
addressed later.

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333017
2018-05-22 19:37:59 +00:00
Roman Tereshin 7701360308 [GlobalISel][InstructionSelect] Sorting MatchTable's first level by opcodes and num operands, perf patch 3
This patch continues a series of patches started by r332907 (reapplied as r332917)

In this commit we start sorting the rules by the opcode first, and if
the same, by the number of operands of the root instructions. This
allows better grouping and safe as patterns with different opcodes are
mutually exclusive.

This is expected to decrease time GlobalISel spends in its
InstructionSelect pass by roughly 18% for an -O0 build as measured on
sqlite3-amalgamation (http://sqlite.org/download.html) targeting
AArch64.

I'm also removing RuleMatcher::getFirstConditionAsRootType() function
here and moving it to a later patch within the series as it's not used
yet and was causing a warning on sanitizer-ppc64le-linux bot.

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 333001
2018-05-22 16:54:27 +00:00
Roman Tereshin dad5d6f347 Reverting 332999 to get it a proper commit message
llvm-svn: 333000
2018-05-22 16:53:42 +00:00
Roman Tereshin a7b5d45f9b diff --git a/utils/TableGen/GlobalISelEmitter.cpp b/utils/TableGen/GlobalISelEmitter.cpp
index cdc9df7bf6b..be08165a200 100644
--- a/utils/TableGen/GlobalISelEmitter.cpp
+++ b/utils/TableGen/GlobalISelEmitter.cpp
@@ -1,4531 +1,4539 @@
 //===- GlobalISelEmitter.cpp - Generate an instruction selector -----------===//
 //
 //                     The LLVM Compiler Infrastructure
 //
 // This file is distributed under the University of Illinois Open Source
 // License. See LICENSE.TXT for details.
 //
 //===----------------------------------------------------------------------===//
 //
 /// \file
 /// This tablegen backend emits code for use by the GlobalISel instruction
 /// selector. See include/llvm/CodeGen/TargetGlobalISel.td.
 ///
 /// This file analyzes the patterns recognized by the SelectionDAGISel tablegen
 /// backend, filters out the ones that are unsupported, maps
 /// SelectionDAG-specific constructs to their GlobalISel counterpart
 /// (when applicable: MVT to LLT;  SDNode to generic Instruction).
 ///
 /// Not all patterns are supported: pass the tablegen invocation
 /// "-warn-on-skipped-patterns" to emit a warning when a pattern is skipped,
 /// as well as why.
 ///
 /// The generated file defines a single method:
 ///     bool <Target>InstructionSelector::selectImpl(MachineInstr &I) const;
 /// intended to be used in InstructionSelector::select as the first-step
 /// selector for the patterns that don't require complex C++.
 ///
 /// FIXME: We'll probably want to eventually define a base
 /// "TargetGenInstructionSelector" class.
 ///
 //===----------------------------------------------------------------------===//
 
 #include "CodeGenDAGPatterns.h"
 #include "SubtargetFeatureInfo.h"
 #include "llvm/ADT/Optional.h"
 #include "llvm/ADT/SmallSet.h"
 #include "llvm/ADT/Statistic.h"
 #include "llvm/Support/CodeGenCoverage.h"
 #include "llvm/Support/CommandLine.h"
 #include "llvm/Support/Error.h"
 #include "llvm/Support/LowLevelTypeImpl.h"
 #include "llvm/Support/MachineValueType.h"
 #include "llvm/Support/ScopedPrinter.h"
 #include "llvm/TableGen/Error.h"
 #include "llvm/TableGen/Record.h"
 #include "llvm/TableGen/TableGenBackend.h"
 #include <numeric>
 #include <string>
 using namespace llvm;
 
 #define DEBUG_TYPE "gisel-emitter"
 
 STATISTIC(NumPatternTotal, "Total number of patterns");
 STATISTIC(NumPatternImported, "Number of patterns imported from SelectionDAG");
 STATISTIC(NumPatternImportsSkipped, "Number of SelectionDAG imports skipped");
 STATISTIC(NumPatternsTested, "Number of patterns executed according to coverage information");
 STATISTIC(NumPatternEmitted, "Number of patterns emitted");
 
 cl::OptionCategory GlobalISelEmitterCat("Options for -gen-global-isel");
 
 static cl::opt<bool> WarnOnSkippedPatterns(
     "warn-on-skipped-patterns",
     cl::desc("Explain why a pattern was skipped for inclusion "
              "in the GlobalISel selector"),
     cl::init(false), cl::cat(GlobalISelEmitterCat));
 
 static cl::opt<bool> GenerateCoverage(
     "instrument-gisel-coverage",
     cl::desc("Generate coverage instrumentation for GlobalISel"),
     cl::init(false), cl::cat(GlobalISelEmitterCat));
 
 static cl::opt<std::string> UseCoverageFile(
     "gisel-coverage-file", cl::init(""),
     cl::desc("Specify file to retrieve coverage information from"),
     cl::cat(GlobalISelEmitterCat));
 
 static cl::opt<bool> OptimizeMatchTable(
     "optimize-match-table",
     cl::desc("Generate an optimized version of the match table"),
     cl::init(true), cl::cat(GlobalISelEmitterCat));
 
 namespace {
 //===- Helper functions ---------------------------------------------------===//
 
 /// Get the name of the enum value used to number the predicate function.
 std::string getEnumNameForPredicate(const TreePredicateFn &Predicate) {
   return "GIPFP_" + Predicate.getImmTypeIdentifier().str() + "_" +
          Predicate.getFnName();
 }
 
 /// Get the opcode used to check this predicate.
 std::string getMatchOpcodeForPredicate(const TreePredicateFn &Predicate) {
   return "GIM_Check" + Predicate.getImmTypeIdentifier().str() + "ImmPredicate";
 }
 
 /// This class stands in for LLT wherever we want to tablegen-erate an
 /// equivalent at compiler run-time.
 class LLTCodeGen {
 private:
   LLT Ty;
 
 public:
   LLTCodeGen() = default;
   LLTCodeGen(const LLT &Ty) : Ty(Ty) {}
 
   std::string getCxxEnumValue() const {
     std::string Str;
     raw_string_ostream OS(Str);
 
     emitCxxEnumValue(OS);
     return OS.str();
   }
 
   void emitCxxEnumValue(raw_ostream &OS) const {
     if (Ty.isScalar()) {
       OS << "GILLT_s" << Ty.getSizeInBits();
       return;
     }
     if (Ty.isVector()) {
       OS << "GILLT_v" << Ty.getNumElements() << "s" << Ty.getScalarSizeInBits();
       return;
     }
     if (Ty.isPointer()) {
       OS << "GILLT_p" << Ty.getAddressSpace();
       if (Ty.getSizeInBits() > 0)
         OS << "s" << Ty.getSizeInBits();
       return;
     }
     llvm_unreachable("Unhandled LLT");
   }
 
   void emitCxxConstructorCall(raw_ostream &OS) const {
     if (Ty.isScalar()) {
       OS << "LLT::scalar(" << Ty.getSizeInBits() << ")";
       return;
     }
     if (Ty.isVector()) {
       OS << "LLT::vector(" << Ty.getNumElements() << ", "
          << Ty.getScalarSizeInBits() << ")";
       return;
     }
     if (Ty.isPointer() && Ty.getSizeInBits() > 0) {
       OS << "LLT::pointer(" << Ty.getAddressSpace() << ", "
          << Ty.getSizeInBits() << ")";
       return;
     }
     llvm_unreachable("Unhandled LLT");
   }
 
   const LLT &get() const { return Ty; }
 
   /// This ordering is used for std::unique() and llvm::sort(). There's no
   /// particular logic behind the order but either A < B or B < A must be
   /// true if A != B.
   bool operator<(const LLTCodeGen &Other) const {
     if (Ty.isValid() != Other.Ty.isValid())
       return Ty.isValid() < Other.Ty.isValid();
     if (!Ty.isValid())
       return false;
 
     if (Ty.isVector() != Other.Ty.isVector())
       return Ty.isVector() < Other.Ty.isVector();
     if (Ty.isScalar() != Other.Ty.isScalar())
       return Ty.isScalar() < Other.Ty.isScalar();
     if (Ty.isPointer() != Other.Ty.isPointer())
       return Ty.isPointer() < Other.Ty.isPointer();
 
     if (Ty.isPointer() && Ty.getAddressSpace() != Other.Ty.getAddressSpace())
       return Ty.getAddressSpace() < Other.Ty.getAddressSpace();
 
     if (Ty.isVector() && Ty.getNumElements() != Other.Ty.getNumElements())
       return Ty.getNumElements() < Other.Ty.getNumElements();
 
     return Ty.getSizeInBits() < Other.Ty.getSizeInBits();
   }
 
   bool operator==(const LLTCodeGen &B) const { return Ty == B.Ty; }
 };
 
 // Track all types that are used so we can emit the corresponding enum.
 std::set<LLTCodeGen> KnownTypes;
 
 class InstructionMatcher;
 /// Convert an MVT to an equivalent LLT if possible, or the invalid LLT() for
 /// MVTs that don't map cleanly to an LLT (e.g., iPTR, *any, ...).
 static Optional<LLTCodeGen> MVTToLLT(MVT::SimpleValueType SVT) {
   MVT VT(SVT);
 
   if (VT.isVector() && VT.getVectorNumElements() != 1)
     return LLTCodeGen(
         LLT::vector(VT.getVectorNumElements(), VT.getScalarSizeInBits()));
 
   if (VT.isInteger() || VT.isFloatingPoint())
     return LLTCodeGen(LLT::scalar(VT.getSizeInBits()));
   return None;
 }
 
 static std::string explainPredicates(const TreePatternNode *N) {
   std::string Explanation = "";
   StringRef Separator = "";
   for (const auto &P : N->getPredicateFns()) {
     Explanation +=
         (Separator + P.getOrigPatFragRecord()->getRecord()->getName()).str();
     Separator = ", ";
 
     if (P.isAlwaysTrue())
       Explanation += " always-true";
     if (P.isImmediatePattern())
       Explanation += " immediate";
 
     if (P.isUnindexed())
       Explanation += " unindexed";
 
     if (P.isNonExtLoad())
       Explanation += " non-extload";
     if (P.isAnyExtLoad())
       Explanation += " extload";
     if (P.isSignExtLoad())
       Explanation += " sextload";
     if (P.isZeroExtLoad())
       Explanation += " zextload";
 
     if (P.isNonTruncStore())
       Explanation += " non-truncstore";
     if (P.isTruncStore())
       Explanation += " truncstore";
 
     if (Record *VT = P.getMemoryVT())
       Explanation += (" MemVT=" + VT->getName()).str();
     if (Record *VT = P.getScalarMemoryVT())
       Explanation += (" ScalarVT(MemVT)=" + VT->getName()).str();
 
     if (P.isAtomicOrderingMonotonic())
       Explanation += " monotonic";
     if (P.isAtomicOrderingAcquire())
       Explanation += " acquire";
     if (P.isAtomicOrderingRelease())
       Explanation += " release";
     if (P.isAtomicOrderingAcquireRelease())
       Explanation += " acq_rel";
     if (P.isAtomicOrderingSequentiallyConsistent())
       Explanation += " seq_cst";
     if (P.isAtomicOrderingAcquireOrStronger())
       Explanation += " >=acquire";
     if (P.isAtomicOrderingWeakerThanAcquire())
       Explanation += " <acquire";
     if (P.isAtomicOrderingReleaseOrStronger())
       Explanation += " >=release";
     if (P.isAtomicOrderingWeakerThanRelease())
       Explanation += " <release";
   }
   return Explanation;
 }
 
 std::string explainOperator(Record *Operator) {
   if (Operator->isSubClassOf("SDNode"))
     return (" (" + Operator->getValueAsString("Opcode") + ")").str();
 
   if (Operator->isSubClassOf("Intrinsic"))
     return (" (Operator is an Intrinsic, " + Operator->getName() + ")").str();
 
   if (Operator->isSubClassOf("ComplexPattern"))
     return (" (Operator is an unmapped ComplexPattern, " + Operator->getName() +
             ")")
         .str();
 
   if (Operator->isSubClassOf("SDNodeXForm"))
     return (" (Operator is an unmapped SDNodeXForm, " + Operator->getName() +
             ")")
         .str();
 
   return (" (Operator " + Operator->getName() + " not understood)").str();
 }
 
 /// Helper function to let the emitter report skip reason error messages.
 static Error failedImport(const Twine &Reason) {
   return make_error<StringError>(Reason, inconvertibleErrorCode());
 }
 
 static Error isTrivialOperatorNode(const TreePatternNode *N) {
   std::string Explanation = "";
   std::string Separator = "";
 
   bool HasUnsupportedPredicate = false;
   for (const auto &Predicate : N->getPredicateFns()) {
     if (Predicate.isAlwaysTrue())
       continue;
 
     if (Predicate.isImmediatePattern())
       continue;
 
     if (Predicate.isNonExtLoad() || Predicate.isAnyExtLoad() ||
         Predicate.isSignExtLoad() || Predicate.isZeroExtLoad())
       continue;
 
     if (Predicate.isNonTruncStore())
       continue;
 
     if (Predicate.isLoad() && Predicate.getMemoryVT())
       continue;
 
     if (Predicate.isLoad() || Predicate.isStore()) {
       if (Predicate.isUnindexed())
         continue;
     }
 
     if (Predicate.isAtomic() && Predicate.getMemoryVT())
       continue;
 
     if (Predicate.isAtomic() &&
         (Predicate.isAtomicOrderingMonotonic() ||
          Predicate.isAtomicOrderingAcquire() ||
          Predicate.isAtomicOrderingRelease() ||
          Predicate.isAtomicOrderingAcquireRelease() ||
          Predicate.isAtomicOrderingSequentiallyConsistent() ||
          Predicate.isAtomicOrderingAcquireOrStronger() ||
          Predicate.isAtomicOrderingWeakerThanAcquire() ||
          Predicate.isAtomicOrderingReleaseOrStronger() ||
          Predicate.isAtomicOrderingWeakerThanRelease()))
       continue;
 
     HasUnsupportedPredicate = true;
     Explanation = Separator + "Has a predicate (" + explainPredicates(N) + ")";
     Separator = ", ";
     Explanation += (Separator + "first-failing:" +
                     Predicate.getOrigPatFragRecord()->getRecord()->getName())
                        .str();
     break;
   }
 
   if (!HasUnsupportedPredicate)
     return Error::success();
 
   return failedImport(Explanation);
 }
 
 static Record *getInitValueAsRegClass(Init *V) {
   if (DefInit *VDefInit = dyn_cast<DefInit>(V)) {
     if (VDefInit->getDef()->isSubClassOf("RegisterOperand"))
       return VDefInit->getDef()->getValueAsDef("RegClass");
     if (VDefInit->getDef()->isSubClassOf("RegisterClass"))
       return VDefInit->getDef();
   }
   return nullptr;
 }
 
 std::string
 getNameForFeatureBitset(const std::vector<Record *> &FeatureBitset) {
   std::string Name = "GIFBS";
   for (const auto &Feature : FeatureBitset)
     Name += ("_" + Feature->getName()).str();
   return Name;
 }
 
 //===- MatchTable Helpers -------------------------------------------------===//
 
 class MatchTable;
 
 /// A record to be stored in a MatchTable.
 ///
 /// This class represents any and all output that may be required to emit the
 /// MatchTable. Instances  are most often configured to represent an opcode or
 /// value that will be emitted to the table with some formatting but it can also
 /// represent commas, comments, and other formatting instructions.
 struct MatchTableRecord {
   enum RecordFlagsBits {
     MTRF_None = 0x0,
     /// Causes EmitStr to be formatted as comment when emitted.
     MTRF_Comment = 0x1,
     /// Causes the record value to be followed by a comma when emitted.
     MTRF_CommaFollows = 0x2,
     /// Causes the record value to be followed by a line break when emitted.
     MTRF_LineBreakFollows = 0x4,
     /// Indicates that the record defines a label and causes an additional
     /// comment to be emitted containing the index of the label.
     MTRF_Label = 0x8,
     /// Causes the record to be emitted as the index of the label specified by
     /// LabelID along with a comment indicating where that label is.
     MTRF_JumpTarget = 0x10,
     /// Causes the formatter to add a level of indentation before emitting the
     /// record.
     MTRF_Indent = 0x20,
     /// Causes the formatter to remove a level of indentation after emitting the
     /// record.
     MTRF_Outdent = 0x40,
   };
 
   /// When MTRF_Label or MTRF_JumpTarget is used, indicates a label id to
   /// reference or define.
   unsigned LabelID;
   /// The string to emit. Depending on the MTRF_* flags it may be a comment, a
   /// value, a label name.
   std::string EmitStr;
 
 private:
   /// The number of MatchTable elements described by this record. Comments are 0
   /// while values are typically 1. Values >1 may occur when we need to emit
   /// values that exceed the size of a MatchTable element.
   unsigned NumElements;
 
 public:
   /// A bitfield of RecordFlagsBits flags.
   unsigned Flags;
 
   /// The actual run-time value, if known
   int64_t RawValue;
 
   MatchTableRecord(Optional<unsigned> LabelID_, StringRef EmitStr,
                    unsigned NumElements, unsigned Flags,
                    int64_t RawValue = std::numeric_limits<int64_t>::min())
       : LabelID(LabelID_.hasValue() ? LabelID_.getValue() : ~0u),
         EmitStr(EmitStr), NumElements(NumElements), Flags(Flags),
         RawValue(RawValue) {
 
     assert((!LabelID_.hasValue() || LabelID != ~0u) &&
            "This value is reserved for non-labels");
   }
   MatchTableRecord(const MatchTableRecord &Other) = default;
   MatchTableRecord(MatchTableRecord &&Other) = default;
 
   /// Useful if a Match Table Record gets optimized out
   void turnIntoComment() {
     Flags |= MTRF_Comment;
     Flags &= ~MTRF_CommaFollows;
     NumElements = 0;
   }
 
   /// For Jump Table generation purposes
   bool operator<(const MatchTableRecord &Other) const {
     return RawValue < Other.RawValue;
   }
   int64_t getRawValue() const { return RawValue; }
 
   void emit(raw_ostream &OS, bool LineBreakNextAfterThis,
             const MatchTable &Table) const;
   unsigned size() const { return NumElements; }
 };
 
 class Matcher;
 
 /// Holds the contents of a generated MatchTable to enable formatting and the
 /// necessary index tracking needed to support GIM_Try.
 class MatchTable {
   /// An unique identifier for the table. The generated table will be named
   /// MatchTable${ID}.
   unsigned ID;
   /// The records that make up the table. Also includes comments describing the
   /// values being emitted and line breaks to format it.
   std::vector<MatchTableRecord> Contents;
   /// The currently defined labels.
   DenseMap<unsigned, unsigned> LabelMap;
   /// Tracks the sum of MatchTableRecord::NumElements as the table is built.
   unsigned CurrentSize = 0;
   /// A unique identifier for a MatchTable label.
   unsigned CurrentLabelID = 0;
   /// Determines if the table should be instrumented for rule coverage tracking.
   bool IsWithCoverage;
 
 public:
   static MatchTableRecord LineBreak;
   static MatchTableRecord Comment(StringRef Comment) {
     return MatchTableRecord(None, Comment, 0, MatchTableRecord::MTRF_Comment);
   }
   static MatchTableRecord Opcode(StringRef Opcode, int IndentAdjust = 0) {
     unsigned ExtraFlags = 0;
     if (IndentAdjust > 0)
       ExtraFlags |= MatchTableRecord::MTRF_Indent;
     if (IndentAdjust < 0)
       ExtraFlags |= MatchTableRecord::MTRF_Outdent;
 
     return MatchTableRecord(None, Opcode, 1,
                             MatchTableRecord::MTRF_CommaFollows | ExtraFlags);
   }
   static MatchTableRecord NamedValue(StringRef NamedValue) {
     return MatchTableRecord(None, NamedValue, 1,
                             MatchTableRecord::MTRF_CommaFollows);
   }
   static MatchTableRecord NamedValue(StringRef NamedValue, int64_t RawValue) {
     return MatchTableRecord(None, NamedValue, 1,
                             MatchTableRecord::MTRF_CommaFollows, RawValue);
   }
   static MatchTableRecord NamedValue(StringRef Namespace,
                                      StringRef NamedValue) {
     return MatchTableRecord(None, (Namespace + "::" + NamedValue).str(), 1,
                             MatchTableRecord::MTRF_CommaFollows);
   }
   static MatchTableRecord NamedValue(StringRef Namespace, StringRef NamedValue,
                                      int64_t RawValue) {
     return MatchTableRecord(None, (Namespace + "::" + NamedValue).str(), 1,
                             MatchTableRecord::MTRF_CommaFollows, RawValue);
   }
   static MatchTableRecord IntValue(int64_t IntValue) {
     return MatchTableRecord(None, llvm::to_string(IntValue), 1,
                             MatchTableRecord::MTRF_CommaFollows);
   }
   static MatchTableRecord Label(unsigned LabelID) {
     return MatchTableRecord(LabelID, "Label " + llvm::to_string(LabelID), 0,
                             MatchTableRecord::MTRF_Label |
                                 MatchTableRecord::MTRF_Comment |
                                 MatchTableRecord::MTRF_LineBreakFollows);
   }
   static MatchTableRecord JumpTarget(unsigned LabelID) {
     return MatchTableRecord(LabelID, "Label " + llvm::to_string(LabelID), 1,
                             MatchTableRecord::MTRF_JumpTarget |
                                 MatchTableRecord::MTRF_Comment |
                                 MatchTableRecord::MTRF_CommaFollows);
   }
 
   static MatchTable buildTable(ArrayRef<Matcher *> Rules, bool WithCoverage);
 
   MatchTable(bool WithCoverage, unsigned ID = 0)
       : ID(ID), IsWithCoverage(WithCoverage) {}
 
   bool isWithCoverage() const { return IsWithCoverage; }
 
   void push_back(const MatchTableRecord &Value) {
     if (Value.Flags & MatchTableRecord::MTRF_Label)
       defineLabel(Value.LabelID);
     Contents.push_back(Value);
     CurrentSize += Value.size();
   }
 
   unsigned allocateLabelID() { return CurrentLabelID++; }
 
   void defineLabel(unsigned LabelID) {
     LabelMap.insert(std::make_pair(LabelID, CurrentSize));
   }
 
   unsigned getLabelIndex(unsigned LabelID) const {
     const auto I = LabelMap.find(LabelID);
     assert(I != LabelMap.end() && "Use of undeclared label");
     return I->second;
   }
 
   void emitUse(raw_ostream &OS) const { OS << "MatchTable" << ID; }
 
   void emitDeclaration(raw_ostream &OS) const {
     unsigned Indentation = 4;
     OS << "  constexpr static int64_t MatchTable" << ID << "[] = {";
     LineBreak.emit(OS, true, *this);
     OS << std::string(Indentation, ' ');
 
     for (auto I = Contents.begin(), E = Contents.end(); I != E;
          ++I) {
       bool LineBreakIsNext = false;
       const auto &NextI = std::next(I);
 
       if (NextI != E) {
         if (NextI->EmitStr == "" &&
             NextI->Flags == MatchTableRecord::MTRF_LineBreakFollows)
           LineBreakIsNext = true;
       }
 
       if (I->Flags & MatchTableRecord::MTRF_Indent)
         Indentation += 2;
 
       I->emit(OS, LineBreakIsNext, *this);
       if (I->Flags & MatchTableRecord::MTRF_LineBreakFollows)
         OS << std::string(Indentation, ' ');
 
       if (I->Flags & MatchTableRecord::MTRF_Outdent)
         Indentation -= 2;
     }
     OS << "};\n";
   }
 };
 
 MatchTableRecord MatchTable::LineBreak = {
     None, "" /* Emit String */, 0 /* Elements */,
     MatchTableRecord::MTRF_LineBreakFollows};
 
 void MatchTableRecord::emit(raw_ostream &OS, bool LineBreakIsNextAfterThis,
                             const MatchTable &Table) const {
   bool UseLineComment =
       LineBreakIsNextAfterThis | (Flags & MTRF_LineBreakFollows);
   if (Flags & (MTRF_JumpTarget | MTRF_CommaFollows))
     UseLineComment = false;
 
   if (Flags & MTRF_Comment)
     OS << (UseLineComment ? "// " : "/*");
 
   OS << EmitStr;
   if (Flags & MTRF_Label)
     OS << ": @" << Table.getLabelIndex(LabelID);
 
   if (Flags & MTRF_Comment && !UseLineComment)
     OS << "*/";
 
   if (Flags & MTRF_JumpTarget) {
     if (Flags & MTRF_Comment)
       OS << " ";
     OS << Table.getLabelIndex(LabelID);
   }
 
   if (Flags & MTRF_CommaFollows) {
     OS << ",";
     if (!LineBreakIsNextAfterThis && !(Flags & MTRF_LineBreakFollows))
       OS << " ";
   }
 
   if (Flags & MTRF_LineBreakFollows)
     OS << "\n";
 }
 
 MatchTable &operator<<(MatchTable &Table, const MatchTableRecord &Value) {
   Table.push_back(Value);
   return Table;
 }
 
 //===- Matchers -----------------------------------------------------------===//
 
 class OperandMatcher;
 class MatchAction;
 class PredicateMatcher;
 class RuleMatcher;
 
 class Matcher {
 public:
   virtual ~Matcher() = default;
   virtual void optimize() {}
   virtual void emit(MatchTable &Table) = 0;
 
   virtual bool hasFirstCondition() const = 0;
   virtual const PredicateMatcher &getFirstCondition() const = 0;
   virtual std::unique_ptr<PredicateMatcher> popFirstCondition() = 0;
 };
 
 MatchTable MatchTable::buildTable(ArrayRef<Matcher *> Rules,
                                   bool WithCoverage) {
   MatchTable Table(WithCoverage);
   for (Matcher *Rule : Rules)
     Rule->emit(Table);
 
   return Table << MatchTable::Opcode("GIM_Reject") << MatchTable::LineBreak;
 }
 
 class GroupMatcher final : public Matcher {
   /// Conditions that form a common prefix of all the matchers contained.
   SmallVector<std::unique_ptr<PredicateMatcher>, 1> Conditions;
 
   /// All the nested matchers, sharing a common prefix.
   std::vector<Matcher *> Matchers;
 
   /// An owning collection for any auxiliary matchers created while optimizing
   /// nested matchers contained.
   std::vector<std::unique_ptr<Matcher>> MatcherStorage;
 
 public:
   /// Add a matcher to the collection of nested matchers if it meets the
   /// requirements, and return true. If it doesn't, do nothing and return false.
   ///
   /// Expected to preserve its argument, so it could be moved out later on.
   bool addMatcher(Matcher &Candidate);
 
   /// Mark the matcher as fully-built and ensure any invariants expected by both
   /// optimize() and emit(...) methods. Generally, both sequences of calls
   /// are expected to lead to a sensible result:
   ///
   /// addMatcher(...)*; finalize(); optimize(); emit(...); and
   /// addMatcher(...)*; finalize(); emit(...);
   ///
   /// or generally
   ///
   /// addMatcher(...)*; finalize(); { optimize()*; emit(...); }*
   ///
   /// Multiple calls to optimize() are expected to be handled gracefully, though
   /// optimize() is not expected to be idempotent. Multiple calls to finalize()
   /// aren't generally supported. emit(...) is expected to be non-mutating and
   /// producing the exact same results upon repeated calls.
   ///
   /// addMatcher() calls after the finalize() call are not supported.
   ///
   /// finalize() and optimize() are both allowed to mutate the contained
   /// matchers, so moving them out after finalize() is not supported.
   void finalize();
   void optimize() override {}
   void emit(MatchTable &Table) override;
 
   /// Could be used to move out the matchers added previously, unless finalize()
   /// has been already called. If any of the matchers are moved out, the group
   /// becomes safe to destroy, but not safe to re-use for anything else.
   iterator_range<std::vector<Matcher *>::iterator> matchers() {
     return make_range(Matchers.begin(), Matchers.end());
   }
   size_t size() const { return Matchers.size(); }
   bool empty() const { return Matchers.empty(); }
 
   std::unique_ptr<PredicateMatcher> popFirstCondition() override {
     assert(!Conditions.empty() &&
            "Trying to pop a condition from a condition-less group");
     std::unique_ptr<PredicateMatcher> P = std::move(Conditions.front());
     Conditions.erase(Conditions.begin());
     return P;
   }
   const PredicateMatcher &getFirstCondition() const override {
     assert(!Conditions.empty() &&
            "Trying to get a condition from a condition-less group");
     return *Conditions.front();
   }
   bool hasFirstCondition() const override { return !Conditions.empty(); }
 
 private:
   /// See if a candidate matcher could be added to this group solely by
   /// analyzing its first condition.
   bool candidateConditionMatches(const PredicateMatcher &Predicate) const;
 };
 
 /// Generates code to check that a match rule matches.
 class RuleMatcher : public Matcher {
 public:
   using ActionList = std::list<std::unique_ptr<MatchAction>>;
   using action_iterator = ActionList::iterator;
 
 protected:
   /// A list of matchers that all need to succeed for the current rule to match.
   /// FIXME: This currently supports a single match position but could be
   /// extended to support multiple positions to support div/rem fusion or
   /// load-multiple instructions.
   using MatchersTy = std::vector<std::unique_ptr<InstructionMatcher>> ;
   MatchersTy Matchers;
 
   /// A list of actions that need to be taken when all predicates in this rule
   /// have succeeded.
   ActionList Actions;
 
   using DefinedInsnVariablesMap = std::map<InstructionMatcher *, unsigned>;
 
   /// A map of instruction matchers to the local variables
   DefinedInsnVariablesMap InsnVariableIDs;
 
   using MutatableInsnSet = SmallPtrSet<InstructionMatcher *, 4>;
 
   // The set of instruction matchers that have not yet been claimed for mutation
   // by a BuildMI.
   MutatableInsnSet MutatableInsns;
 
   /// A map of named operands defined by the matchers that may be referenced by
   /// the renderers.
   StringMap<OperandMatcher *> DefinedOperands;
 
   /// ID for the next instruction variable defined with implicitlyDefineInsnVar()
   unsigned NextInsnVarID;
 
   /// ID for the next output instruction allocated with allocateOutputInsnID()
   unsigned NextOutputInsnID;
 
   /// ID for the next temporary register ID allocated with allocateTempRegID()
   unsigned NextTempRegID;
 
   std::vector<Record *> RequiredFeatures;
   std::vector<std::unique_ptr<PredicateMatcher>> EpilogueMatchers;
 
   ArrayRef<SMLoc> SrcLoc;
 
   typedef std::tuple<Record *, unsigned, unsigned>
       DefinedComplexPatternSubOperand;
   typedef StringMap<DefinedComplexPatternSubOperand>
       DefinedComplexPatternSubOperandMap;
   /// A map of Symbolic Names to ComplexPattern sub-operands.
   DefinedComplexPatternSubOperandMap ComplexSubOperands;
 
   uint64_t RuleID;
   static uint64_t NextRuleID;
 
 public:
   RuleMatcher(ArrayRef<SMLoc> SrcLoc)
       : Matchers(), Actions(), InsnVariableIDs(), MutatableInsns(),
         DefinedOperands(), NextInsnVarID(0), NextOutputInsnID(0),
         NextTempRegID(0), SrcLoc(SrcLoc), ComplexSubOperands(),
         RuleID(NextRuleID++) {}
   RuleMatcher(RuleMatcher &&Other) = default;
   RuleMatcher &operator=(RuleMatcher &&Other) = default;
 
   uint64_t getRuleID() const { return RuleID; }
 
   InstructionMatcher &addInstructionMatcher(StringRef SymbolicName);
   void addRequiredFeature(Record *Feature);
   const std::vector<Record *> &getRequiredFeatures() const;
 
   template <class Kind, class... Args> Kind &addAction(Args &&... args);
   template <class Kind, class... Args>
   action_iterator insertAction(action_iterator InsertPt, Args &&... args);
 
   /// Define an instruction without emitting any code to do so.
   unsigned implicitlyDefineInsnVar(InstructionMatcher &Matcher);
 
   unsigned getInsnVarID(InstructionMatcher &InsnMatcher) const;
   DefinedInsnVariablesMap::const_iterator defined_insn_vars_begin() const {
     return InsnVariableIDs.begin();
   }
   DefinedInsnVariablesMap::const_iterator defined_insn_vars_end() const {
     return InsnVariableIDs.end();
   }
   iterator_range<typename DefinedInsnVariablesMap::const_iterator>
   defined_insn_vars() const {
     return make_range(defined_insn_vars_begin(), defined_insn_vars_end());
   }
 
   MutatableInsnSet::const_iterator mutatable_insns_begin() const {
     return MutatableInsns.begin();
   }
   MutatableInsnSet::const_iterator mutatable_insns_end() const {
     return MutatableInsns.end();
   }
   iterator_range<typename MutatableInsnSet::const_iterator>
   mutatable_insns() const {
     return make_range(mutatable_insns_begin(), mutatable_insns_end());
   }
   void reserveInsnMatcherForMutation(InstructionMatcher *InsnMatcher) {
     bool R = MutatableInsns.erase(InsnMatcher);
     assert(R && "Reserving a mutatable insn that isn't available");
     (void)R;
   }
 
   action_iterator actions_begin() { return Actions.begin(); }
   action_iterator actions_end() { return Actions.end(); }
   iterator_range<action_iterator> actions() {
     return make_range(actions_begin(), actions_end());
   }
 
   void defineOperand(StringRef SymbolicName, OperandMatcher &OM);
 
   void defineComplexSubOperand(StringRef SymbolicName, Record *ComplexPattern,
                                unsigned RendererID, unsigned SubOperandID) {
     assert(ComplexSubOperands.count(SymbolicName) == 0 && "Already defined");
     ComplexSubOperands[SymbolicName] =
         std::make_tuple(ComplexPattern, RendererID, SubOperandID);
   }
   Optional<DefinedComplexPatternSubOperand>
   getComplexSubOperand(StringRef SymbolicName) const {
     const auto &I = ComplexSubOperands.find(SymbolicName);
     if (I == ComplexSubOperands.end())
       return None;
     return I->second;
   }
 
   InstructionMatcher &getInstructionMatcher(StringRef SymbolicName) const;
   const OperandMatcher &getOperandMatcher(StringRef Name) const;
 
   void optimize() override;
   void emit(MatchTable &Table) override;
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   bool isHigherPriorityThan(const RuleMatcher &B) const;
 
   /// Report the maximum number of temporary operands needed by the rule
   /// matcher.
   unsigned countRendererFns() const;
 
   std::unique_ptr<PredicateMatcher> popFirstCondition() override;
   const PredicateMatcher &getFirstCondition() const override;
-  LLTCodeGen getFirstConditionAsRootType();
   bool hasFirstCondition() const override;
   unsigned getNumOperands() const;
   StringRef getOpcode() const;
 
   // FIXME: Remove this as soon as possible
   InstructionMatcher &insnmatchers_front() const { return *Matchers.front(); }
 
   unsigned allocateOutputInsnID() { return NextOutputInsnID++; }
   unsigned allocateTempRegID() { return NextTempRegID++; }
 
   iterator_range<MatchersTy::iterator> insnmatchers() {
     return make_range(Matchers.begin(), Matchers.end());
   }
   bool insnmatchers_empty() const { return Matchers.empty(); }
   void insnmatchers_pop_front() { Matchers.erase(Matchers.begin()); }
 };
 
 uint64_t RuleMatcher::NextRuleID = 0;
 
 using action_iterator = RuleMatcher::action_iterator;
 
 template <class PredicateTy> class PredicateListMatcher {
 private:
   /// Template instantiations should specialize this to return a string to use
   /// for the comment emitted when there are no predicates.
   std::string getNoPredicateComment() const;
 
 protected:
   using PredicatesTy = std::deque<std::unique_ptr<PredicateTy>>;
   PredicatesTy Predicates;
 
   /// Track if the list of predicates was manipulated by one of the optimization
   /// methods.
   bool Optimized = false;
 
 public:
   /// Construct a new predicate and add it to the matcher.
   template <class Kind, class... Args>
   Optional<Kind *> addPredicate(Args &&... args);
 
   typename PredicatesTy::iterator predicates_begin() {
     return Predicates.begin();
   }
   typename PredicatesTy::iterator predicates_end() {
     return Predicates.end();
   }
   iterator_range<typename PredicatesTy::iterator> predicates() {
     return make_range(predicates_begin(), predicates_end());
   }
   typename PredicatesTy::size_type predicates_size() const {
     return Predicates.size();
   }
   bool predicates_empty() const { return Predicates.empty(); }
 
   std::unique_ptr<PredicateTy> predicates_pop_front() {
     std::unique_ptr<PredicateTy> Front = std::move(Predicates.front());
     Predicates.pop_front();
     Optimized = true;
     return Front;
   }
 
   void prependPredicate(std::unique_ptr<PredicateTy> &&Predicate) {
     Predicates.push_front(std::move(Predicate));
   }
 
   void eraseNullPredicates() {
     const auto NewEnd =
         std::stable_partition(Predicates.begin(), Predicates.end(),
                               std::logical_not<std::unique_ptr<PredicateTy>>());
     if (NewEnd != Predicates.begin()) {
       Predicates.erase(Predicates.begin(), NewEnd);
       Optimized = true;
     }
   }
 
   /// Emit MatchTable opcodes that tests whether all the predicates are met.
   template <class... Args>
   void emitPredicateListOpcodes(MatchTable &Table, Args &&... args) {
     if (Predicates.empty() && !Optimized) {
       Table << MatchTable::Comment(getNoPredicateComment())
             << MatchTable::LineBreak;
       return;
     }
 
     for (const auto &Predicate : predicates())
       Predicate->emitPredicateOpcodes(Table, std::forward<Args>(args)...);
   }
 };
 
 class PredicateMatcher {
 public:
   /// This enum is used for RTTI and also defines the priority that is given to
   /// the predicate when generating the matcher code. Kinds with higher priority
   /// must be tested first.
   ///
   /// The relative priority of OPM_LLT, OPM_RegBank, and OPM_MBB do not matter
   /// but OPM_Int must have priority over OPM_RegBank since constant integers
   /// are represented by a virtual register defined by a G_CONSTANT instruction.
   ///
   /// Note: The relative priority between IPM_ and OPM_ does not matter, they
   /// are currently not compared between each other.
   enum PredicateKind {
     IPM_Opcode,
     IPM_NumOperands,
     IPM_ImmPredicate,
     IPM_AtomicOrderingMMO,
     IPM_MemoryLLTSize,
     IPM_MemoryVsLLTSize,
     OPM_SameOperand,
     OPM_ComplexPattern,
     OPM_IntrinsicID,
     OPM_Instruction,
     OPM_Int,
     OPM_LiteralInt,
     OPM_LLT,
     OPM_PointerToAny,
     OPM_RegBank,
     OPM_MBB,
   };
 
 protected:
   PredicateKind Kind;
   unsigned InsnVarID;
   unsigned OpIdx;
 
 public:
   PredicateMatcher(PredicateKind Kind, unsigned InsnVarID, unsigned OpIdx = ~0)
       : Kind(Kind), InsnVarID(InsnVarID), OpIdx(OpIdx) {}
 
   unsigned getInsnVarID() const { return InsnVarID; }
   unsigned getOpIdx() const { return OpIdx; }
 
   virtual ~PredicateMatcher() = default;
   /// Emit MatchTable opcodes that check the predicate for the given operand.
   virtual void emitPredicateOpcodes(MatchTable &Table,
                                     RuleMatcher &Rule) const = 0;
 
   PredicateKind getKind() const { return Kind; }
 
   virtual bool isIdentical(const PredicateMatcher &B) const {
     return B.getKind() == getKind() && InsnVarID == B.InsnVarID &&
            OpIdx == B.OpIdx;
   }
 
   virtual bool isIdenticalDownToValue(const PredicateMatcher &B) const {
     return hasValue() && PredicateMatcher::isIdentical(B);
   }
 
   virtual MatchTableRecord getValue() const {
     assert(hasValue() && "Can not get a value of a value-less predicate!");
     llvm_unreachable("Not implemented yet");
   }
   virtual bool hasValue() const { return false; }
 
   /// Report the maximum number of temporary operands needed by the predicate
   /// matcher.
   virtual unsigned countRendererFns() const { return 0; }
 };
 
 /// Generates code to check a predicate of an operand.
 ///
 /// Typical predicates include:
 /// * Operand is a particular register.
 /// * Operand is assigned a particular register bank.
 /// * Operand is an MBB.
 class OperandPredicateMatcher : public PredicateMatcher {
 public:
   OperandPredicateMatcher(PredicateKind Kind, unsigned InsnVarID,
                           unsigned OpIdx)
       : PredicateMatcher(Kind, InsnVarID, OpIdx) {}
   virtual ~OperandPredicateMatcher() {}
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   virtual bool isHigherPriorityThan(const OperandPredicateMatcher &B) const;
 };
 
 template <>
 std::string
 PredicateListMatcher<OperandPredicateMatcher>::getNoPredicateComment() const {
   return "No operand predicates";
 }
 
 /// Generates code to check that a register operand is defined by the same exact
 /// one as another.
 class SameOperandMatcher : public OperandPredicateMatcher {
   std::string MatchingName;
 
 public:
   SameOperandMatcher(unsigned InsnVarID, unsigned OpIdx, StringRef MatchingName)
       : OperandPredicateMatcher(OPM_SameOperand, InsnVarID, OpIdx),
         MatchingName(MatchingName) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_SameOperand;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override;
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return OperandPredicateMatcher::isIdentical(B) &&
            MatchingName == cast<SameOperandMatcher>(&B)->MatchingName;
   }
 };
 
 /// Generates code to check that an operand is a particular LLT.
 class LLTOperandMatcher : public OperandPredicateMatcher {
 protected:
   LLTCodeGen Ty;
 
 public:
   static std::map<LLTCodeGen, unsigned> TypeIDValues;
 
   static void initTypeIDValuesMap() {
     TypeIDValues.clear();
 
     unsigned ID = 0;
     for (const LLTCodeGen LLTy : KnownTypes)
       TypeIDValues[LLTy] = ID++;
   }
 
   LLTOperandMatcher(unsigned InsnVarID, unsigned OpIdx, const LLTCodeGen &Ty)
       : OperandPredicateMatcher(OPM_LLT, InsnVarID, OpIdx), Ty(Ty) {
     KnownTypes.insert(Ty);
   }
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_LLT;
   }
   bool isIdentical(const PredicateMatcher &B) const override {
     return OperandPredicateMatcher::isIdentical(B) &&
            Ty == cast<LLTOperandMatcher>(&B)->Ty;
   }
   MatchTableRecord getValue() const override {
     const auto VI = TypeIDValues.find(Ty);
     if (VI == TypeIDValues.end())
       return MatchTable::NamedValue(getTy().getCxxEnumValue());
     return MatchTable::NamedValue(getTy().getCxxEnumValue(), VI->second);
   }
   bool hasValue() const override {
     if (TypeIDValues.size() != KnownTypes.size())
       initTypeIDValuesMap();
     return TypeIDValues.count(Ty);
   }
 
   LLTCodeGen getTy() const { return Ty; }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckType") << MatchTable::Comment("MI")
           << MatchTable::IntValue(InsnVarID) << MatchTable::Comment("Op")
           << MatchTable::IntValue(OpIdx) << MatchTable::Comment("Type")
           << getValue() << MatchTable::LineBreak;
   }
 };
 
 std::map<LLTCodeGen, unsigned> LLTOperandMatcher::TypeIDValues;
 
 /// Generates code to check that an operand is a pointer to any address space.
 ///
 /// In SelectionDAG, the types did not describe pointers or address spaces. As a
 /// result, iN is used to describe a pointer of N bits to any address space and
 /// PatFrag predicates are typically used to constrain the address space. There's
 /// no reliable means to derive the missing type information from the pattern so
 /// imported rules must test the components of a pointer separately.
 ///
 /// If SizeInBits is zero, then the pointer size will be obtained from the
 /// subtarget.
 class PointerToAnyOperandMatcher : public OperandPredicateMatcher {
 protected:
   unsigned SizeInBits;
 
 public:
   PointerToAnyOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
                              unsigned SizeInBits)
       : OperandPredicateMatcher(OPM_PointerToAny, InsnVarID, OpIdx),
         SizeInBits(SizeInBits) {}
 
   static bool classof(const OperandPredicateMatcher *P) {
     return P->getKind() == OPM_PointerToAny;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckPointerToAny")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
           << MatchTable::Comment("SizeInBits")
           << MatchTable::IntValue(SizeInBits) << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that an operand is a particular target constant.
 class ComplexPatternOperandMatcher : public OperandPredicateMatcher {
 protected:
   const OperandMatcher &Operand;
   const Record &TheDef;
 
   unsigned getAllocatedTemporariesBaseID() const;
 
 public:
   bool isIdentical(const PredicateMatcher &B) const override { return false; }
 
   ComplexPatternOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
                                const OperandMatcher &Operand,
                                const Record &TheDef)
       : OperandPredicateMatcher(OPM_ComplexPattern, InsnVarID, OpIdx),
         Operand(Operand), TheDef(TheDef) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_ComplexPattern;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     unsigned ID = getAllocatedTemporariesBaseID();
     Table << MatchTable::Opcode("GIM_CheckComplexPattern")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
           << MatchTable::Comment("Renderer") << MatchTable::IntValue(ID)
           << MatchTable::NamedValue(("GICP_" + TheDef.getName()).str())
           << MatchTable::LineBreak;
   }
 
   unsigned countRendererFns() const override {
     return 1;
   }
 };
 
 /// Generates code to check that an operand is in a particular register bank.
 class RegisterBankOperandMatcher : public OperandPredicateMatcher {
 protected:
   const CodeGenRegisterClass &RC;
 
 public:
   RegisterBankOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
                              const CodeGenRegisterClass &RC)
       : OperandPredicateMatcher(OPM_RegBank, InsnVarID, OpIdx), RC(RC) {}
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return OperandPredicateMatcher::isIdentical(B) &&
            RC.getDef() == cast<RegisterBankOperandMatcher>(&B)->RC.getDef();
   }
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_RegBank;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckRegBankForClass")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
           << MatchTable::Comment("RC")
           << MatchTable::NamedValue(RC.getQualifiedName() + "RegClassID")
           << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that an operand is a basic block.
 class MBBOperandMatcher : public OperandPredicateMatcher {
 public:
   MBBOperandMatcher(unsigned InsnVarID, unsigned OpIdx)
       : OperandPredicateMatcher(OPM_MBB, InsnVarID, OpIdx) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_MBB;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckIsMBB") << MatchTable::Comment("MI")
           << MatchTable::IntValue(InsnVarID) << MatchTable::Comment("Op")
           << MatchTable::IntValue(OpIdx) << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that an operand is a G_CONSTANT with a particular
 /// int.
 class ConstantIntOperandMatcher : public OperandPredicateMatcher {
 protected:
   int64_t Value;
 
 public:
   ConstantIntOperandMatcher(unsigned InsnVarID, unsigned OpIdx, int64_t Value)
       : OperandPredicateMatcher(OPM_Int, InsnVarID, OpIdx), Value(Value) {}
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return OperandPredicateMatcher::isIdentical(B) &&
            Value == cast<ConstantIntOperandMatcher>(&B)->Value;
   }
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_Int;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckConstantInt")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
           << MatchTable::IntValue(Value) << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that an operand is a raw int (where MO.isImm() or
 /// MO.isCImm() is true).
 class LiteralIntOperandMatcher : public OperandPredicateMatcher {
 protected:
   int64_t Value;
 
 public:
   LiteralIntOperandMatcher(unsigned InsnVarID, unsigned OpIdx, int64_t Value)
       : OperandPredicateMatcher(OPM_LiteralInt, InsnVarID, OpIdx),
         Value(Value) {}
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return OperandPredicateMatcher::isIdentical(B) &&
            Value == cast<LiteralIntOperandMatcher>(&B)->Value;
   }
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_LiteralInt;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckLiteralInt")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
           << MatchTable::IntValue(Value) << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that an operand is an intrinsic ID.
 class IntrinsicIDOperandMatcher : public OperandPredicateMatcher {
 protected:
   const CodeGenIntrinsic *II;
 
 public:
   IntrinsicIDOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
                             const CodeGenIntrinsic *II)
       : OperandPredicateMatcher(OPM_IntrinsicID, InsnVarID, OpIdx), II(II) {}
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return OperandPredicateMatcher::isIdentical(B) &&
            II == cast<IntrinsicIDOperandMatcher>(&B)->II;
   }
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_IntrinsicID;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckIntrinsicID")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
           << MatchTable::NamedValue("Intrinsic::" + II->EnumName)
           << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that a set of predicates match for a particular
 /// operand.
 class OperandMatcher : public PredicateListMatcher<OperandPredicateMatcher> {
 protected:
   InstructionMatcher &Insn;
   unsigned OpIdx;
   std::string SymbolicName;
 
   /// The index of the first temporary variable allocated to this operand. The
   /// number of allocated temporaries can be found with
   /// countRendererFns().
   unsigned AllocatedTemporariesBaseID;
 
 public:
   OperandMatcher(InstructionMatcher &Insn, unsigned OpIdx,
                  const std::string &SymbolicName,
                  unsigned AllocatedTemporariesBaseID)
       : Insn(Insn), OpIdx(OpIdx), SymbolicName(SymbolicName),
         AllocatedTemporariesBaseID(AllocatedTemporariesBaseID) {}
 
   bool hasSymbolicName() const { return !SymbolicName.empty(); }
   const StringRef getSymbolicName() const { return SymbolicName; }
   void setSymbolicName(StringRef Name) {
     assert(SymbolicName.empty() && "Operand already has a symbolic name");
     SymbolicName = Name;
   }
 
   /// Construct a new operand predicate and add it to the matcher.
   template <class Kind, class... Args>
   Optional<Kind *> addPredicate(Args &&... args) {
     if (isSameAsAnotherOperand())
       return None;
     Predicates.emplace_back(llvm::make_unique<Kind>(
         getInsnVarID(), getOpIdx(), std::forward<Args>(args)...));
     return static_cast<Kind *>(Predicates.back().get());
   }
 
   unsigned getOpIdx() const { return OpIdx; }
   unsigned getInsnVarID() const;
 
   std::string getOperandExpr(unsigned InsnVarID) const {
     return "State.MIs[" + llvm::to_string(InsnVarID) + "]->getOperand(" +
            llvm::to_string(OpIdx) + ")";
   }
 
   InstructionMatcher &getInstructionMatcher() const { return Insn; }
 
   Error addTypeCheckPredicate(const TypeSetByHwMode &VTy,
                               bool OperandIsAPointer);
 
   /// Emit MatchTable opcodes that test whether the instruction named in
   /// InsnVarID matches all the predicates and all the operands.
   void emitPredicateOpcodes(MatchTable &Table, RuleMatcher &Rule) {
     if (!Optimized) {
       std::string Comment;
       raw_string_ostream CommentOS(Comment);
       CommentOS << "MIs[" << getInsnVarID() << "] ";
       if (SymbolicName.empty())
         CommentOS << "Operand " << OpIdx;
       else
         CommentOS << SymbolicName;
       Table << MatchTable::Comment(CommentOS.str()) << MatchTable::LineBreak;
     }
 
     emitPredicateListOpcodes(Table, Rule);
   }
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   bool isHigherPriorityThan(OperandMatcher &B) {
     // Operand matchers involving more predicates have higher priority.
     if (predicates_size() > B.predicates_size())
       return true;
     if (predicates_size() < B.predicates_size())
       return false;
 
     // This assumes that predicates are added in a consistent order.
     for (auto &&Predicate : zip(predicates(), B.predicates())) {
       if (std::get<0>(Predicate)->isHigherPriorityThan(*std::get<1>(Predicate)))
         return true;
       if (std::get<1>(Predicate)->isHigherPriorityThan(*std::get<0>(Predicate)))
         return false;
     }
 
     return false;
   };
 
   /// Report the maximum number of temporary operands needed by the operand
   /// matcher.
   unsigned countRendererFns() {
     return std::accumulate(
         predicates().begin(), predicates().end(), 0,
         [](unsigned A,
            const std::unique_ptr<OperandPredicateMatcher> &Predicate) {
           return A + Predicate->countRendererFns();
         });
   }
 
   unsigned getAllocatedTemporariesBaseID() const {
     return AllocatedTemporariesBaseID;
   }
 
   bool isSameAsAnotherOperand() {
     for (const auto &Predicate : predicates())
       if (isa<SameOperandMatcher>(Predicate))
         return true;
     return false;
   }
 };
 
 Error OperandMatcher::addTypeCheckPredicate(const TypeSetByHwMode &VTy,
                                             bool OperandIsAPointer) {
   if (!VTy.isMachineValueType())
     return failedImport("unsupported typeset");
 
   if (VTy.getMachineValueType() == MVT::iPTR && OperandIsAPointer) {
     addPredicate<PointerToAnyOperandMatcher>(0);
     return Error::success();
   }
 
   auto OpTyOrNone = MVTToLLT(VTy.getMachineValueType().SimpleTy);
   if (!OpTyOrNone)
     return failedImport("unsupported type");
 
   if (OperandIsAPointer)
     addPredicate<PointerToAnyOperandMatcher>(OpTyOrNone->get().getSizeInBits());
   else
     addPredicate<LLTOperandMatcher>(*OpTyOrNone);
   return Error::success();
 }
 
 unsigned ComplexPatternOperandMatcher::getAllocatedTemporariesBaseID() const {
   return Operand.getAllocatedTemporariesBaseID();
 }
 
 /// Generates code to check a predicate on an instruction.
 ///
 /// Typical predicates include:
 /// * The opcode of the instruction is a particular value.
 /// * The nsw/nuw flag is/isn't set.
 class InstructionPredicateMatcher : public PredicateMatcher {
 public:
   InstructionPredicateMatcher(PredicateKind Kind, unsigned InsnVarID)
       : PredicateMatcher(Kind, InsnVarID) {}
   virtual ~InstructionPredicateMatcher() {}
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   virtual bool
   isHigherPriorityThan(const InstructionPredicateMatcher &B) const {
     return Kind < B.Kind;
   };
 };
 
 template <>
 std::string
 PredicateListMatcher<PredicateMatcher>::getNoPredicateComment() const {
   return "No instruction predicates";
 }
 
 /// Generates code to check the opcode of an instruction.
 class InstructionOpcodeMatcher : public InstructionPredicateMatcher {
 protected:
   const CodeGenInstruction *I;
 
   static DenseMap<const CodeGenInstruction *, unsigned> OpcodeValues;
 
 public:
   static void initOpcodeValuesMap(const CodeGenTarget &Target) {
     OpcodeValues.clear();
 
     unsigned OpcodeValue = 0;
     for (const CodeGenInstruction *I : Target.getInstructionsByEnumValue())
       OpcodeValues[I] = OpcodeValue++;
   }
 
   InstructionOpcodeMatcher(unsigned InsnVarID, const CodeGenInstruction *I)
       : InstructionPredicateMatcher(IPM_Opcode, InsnVarID), I(I) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == IPM_Opcode;
   }
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return InstructionPredicateMatcher::isIdentical(B) &&
            I == cast<InstructionOpcodeMatcher>(&B)->I;
   }
   MatchTableRecord getValue() const override {
     const auto VI = OpcodeValues.find(I);
     if (VI != OpcodeValues.end())
       return MatchTable::NamedValue(I->Namespace, I->TheDef->getName(),
                                     VI->second);
     return MatchTable::NamedValue(I->Namespace, I->TheDef->getName());
   }
   bool hasValue() const override { return OpcodeValues.count(I); }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckOpcode") << MatchTable::Comment("MI")
           << MatchTable::IntValue(InsnVarID) << getValue()
           << MatchTable::LineBreak;
   }
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   bool
   isHigherPriorityThan(const InstructionPredicateMatcher &B) const override {
     if (InstructionPredicateMatcher::isHigherPriorityThan(B))
       return true;
     if (B.InstructionPredicateMatcher::isHigherPriorityThan(*this))
       return false;
 
     // Prioritize opcodes for cosmetic reasons in the generated source. Although
     // this is cosmetic at the moment, we may want to drive a similar ordering
     // using instruction frequency information to improve compile time.
     if (const InstructionOpcodeMatcher *BO =
             dyn_cast<InstructionOpcodeMatcher>(&B))
       return I->TheDef->getName() < BO->I->TheDef->getName();
 
     return false;
   };
 
   bool isConstantInstruction() const {
     return I->TheDef->getName() == "G_CONSTANT";
   }
 
   StringRef getOpcode() const { return I->TheDef->getName(); }
   unsigned getNumOperands() const { return I->Operands.size(); }
 
   StringRef getOperandType(unsigned OpIdx) const {
     return I->Operands[OpIdx].OperandType;
   }
 };
 
 DenseMap<const CodeGenInstruction *, unsigned>
     InstructionOpcodeMatcher::OpcodeValues;
 
 class InstructionNumOperandsMatcher final : public InstructionPredicateMatcher {
   unsigned NumOperands = 0;
 
 public:
   InstructionNumOperandsMatcher(unsigned InsnVarID, unsigned NumOperands)
       : InstructionPredicateMatcher(IPM_NumOperands, InsnVarID),
         NumOperands(NumOperands) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == IPM_NumOperands;
   }
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return InstructionPredicateMatcher::isIdentical(B) &&
            NumOperands == cast<InstructionNumOperandsMatcher>(&B)->NumOperands;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckNumOperands")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Expected")
           << MatchTable::IntValue(NumOperands) << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that this instruction is a constant whose value
 /// meets an immediate predicate.
 ///
 /// Immediates are slightly odd since they are typically used like an operand
 /// but are represented as an operator internally. We typically write simm8:$src
 /// in a tablegen pattern, but this is just syntactic sugar for
 /// (imm:i32)<<P:Predicate_simm8>>:$imm which more directly describes the nodes
 /// that will be matched and the predicate (which is attached to the imm
 /// operator) that will be tested. In SelectionDAG this describes a
 /// ConstantSDNode whose internal value will be tested using the simm8 predicate.
 ///
 /// The corresponding GlobalISel representation is %1 = G_CONSTANT iN Value. In
 /// this representation, the immediate could be tested with an
 /// InstructionMatcher, InstructionOpcodeMatcher, OperandMatcher, and a
 /// OperandPredicateMatcher-subclass to check the Value meets the predicate but
 /// there are two implementation issues with producing that matcher
 /// configuration from the SelectionDAG pattern:
 /// * ImmLeaf is a PatFrag whose root is an InstructionMatcher. This means that
 ///   were we to sink the immediate predicate to the operand we would have to
 ///   have two partial implementations of PatFrag support, one for immediates
 ///   and one for non-immediates.
 /// * At the point we handle the predicate, the OperandMatcher hasn't been
 ///   created yet. If we were to sink the predicate to the OperandMatcher we
 ///   would also have to complicate (or duplicate) the code that descends and
 ///   creates matchers for the subtree.
 /// Overall, it's simpler to handle it in the place it was found.
 class InstructionImmPredicateMatcher : public InstructionPredicateMatcher {
 protected:
   TreePredicateFn Predicate;
 
 public:
   InstructionImmPredicateMatcher(unsigned InsnVarID,
                                  const TreePredicateFn &Predicate)
       : InstructionPredicateMatcher(IPM_ImmPredicate, InsnVarID),
         Predicate(Predicate) {}
 
   bool isIdentical(const PredicateMatcher &B) const override {
     return InstructionPredicateMatcher::isIdentical(B) &&
            Predicate.getOrigPatFragRecord() ==
                cast<InstructionImmPredicateMatcher>(&B)
                    ->Predicate.getOrigPatFragRecord();
   }
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == IPM_ImmPredicate;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode(getMatchOpcodeForPredicate(Predicate))
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("Predicate")
           << MatchTable::NamedValue(getEnumNameForPredicate(Predicate))
           << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that a memory instruction has a atomic ordering
 /// MachineMemoryOperand.
 class AtomicOrderingMMOPredicateMatcher : public InstructionPredicateMatcher {
 public:
   enum AOComparator {
     AO_Exactly,
     AO_OrStronger,
     AO_WeakerThan,
   };
 
 protected:
   StringRef Order;
   AOComparator Comparator;
 
 public:
   AtomicOrderingMMOPredicateMatcher(unsigned InsnVarID, StringRef Order,
                                     AOComparator Comparator = AO_Exactly)
       : InstructionPredicateMatcher(IPM_AtomicOrderingMMO, InsnVarID),
         Order(Order), Comparator(Comparator) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == IPM_AtomicOrderingMMO;
   }
 
   bool isIdentical(const PredicateMatcher &B) const override {
     if (!InstructionPredicateMatcher::isIdentical(B))
       return false;
     const auto &R = *cast<AtomicOrderingMMOPredicateMatcher>(&B);
     return Order == R.Order && Comparator == R.Comparator;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     StringRef Opcode = "GIM_CheckAtomicOrdering";
 
     if (Comparator == AO_OrStronger)
       Opcode = "GIM_CheckAtomicOrderingOrStrongerThan";
     if (Comparator == AO_WeakerThan)
       Opcode = "GIM_CheckAtomicOrderingWeakerThan";
 
     Table << MatchTable::Opcode(Opcode) << MatchTable::Comment("MI")
           << MatchTable::IntValue(InsnVarID) << MatchTable::Comment("Order")
           << MatchTable::NamedValue(("(int64_t)AtomicOrdering::" + Order).str())
           << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that the size of an MMO is exactly N bytes.
 class MemorySizePredicateMatcher : public InstructionPredicateMatcher {
 protected:
   unsigned MMOIdx;
   uint64_t Size;
 
 public:
   MemorySizePredicateMatcher(unsigned InsnVarID, unsigned MMOIdx, unsigned Size)
       : InstructionPredicateMatcher(IPM_MemoryLLTSize, InsnVarID),
         MMOIdx(MMOIdx), Size(Size) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == IPM_MemoryLLTSize;
   }
   bool isIdentical(const PredicateMatcher &B) const override {
     return InstructionPredicateMatcher::isIdentical(B) &&
            MMOIdx == cast<MemorySizePredicateMatcher>(&B)->MMOIdx &&
            Size == cast<MemorySizePredicateMatcher>(&B)->Size;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIM_CheckMemorySizeEqualTo")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("MMO") << MatchTable::IntValue(MMOIdx)
           << MatchTable::Comment("Size") << MatchTable::IntValue(Size)
           << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that the size of an MMO is less-than, equal-to, or
 /// greater than a given LLT.
 class MemoryVsLLTSizePredicateMatcher : public InstructionPredicateMatcher {
 public:
   enum RelationKind {
     GreaterThan,
     EqualTo,
     LessThan,
   };
 
 protected:
   unsigned MMOIdx;
   RelationKind Relation;
   unsigned OpIdx;
 
 public:
   MemoryVsLLTSizePredicateMatcher(unsigned InsnVarID, unsigned MMOIdx,
                                   enum RelationKind Relation,
                                   unsigned OpIdx)
       : InstructionPredicateMatcher(IPM_MemoryVsLLTSize, InsnVarID),
         MMOIdx(MMOIdx), Relation(Relation), OpIdx(OpIdx) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == IPM_MemoryVsLLTSize;
   }
   bool isIdentical(const PredicateMatcher &B) const override {
     return InstructionPredicateMatcher::isIdentical(B) &&
            MMOIdx == cast<MemoryVsLLTSizePredicateMatcher>(&B)->MMOIdx &&
            Relation == cast<MemoryVsLLTSizePredicateMatcher>(&B)->Relation &&
            OpIdx == cast<MemoryVsLLTSizePredicateMatcher>(&B)->OpIdx;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode(Relation == EqualTo
                                     ? "GIM_CheckMemorySizeEqualToLLT"
                                     : Relation == GreaterThan
                                           ? "GIM_CheckMemorySizeGreaterThanLLT"
                                           : "GIM_CheckMemorySizeLessThanLLT")
           << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
           << MatchTable::Comment("MMO") << MatchTable::IntValue(MMOIdx)
           << MatchTable::Comment("OpIdx") << MatchTable::IntValue(OpIdx)
           << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to check that a set of predicates and operands match for a
 /// particular instruction.
 ///
 /// Typical predicates include:
 /// * Has a specific opcode.
 /// * Has an nsw/nuw flag or doesn't.
 class InstructionMatcher final : public PredicateListMatcher<PredicateMatcher> {
 protected:
   typedef std::vector<std::unique_ptr<OperandMatcher>> OperandVec;
 
   RuleMatcher &Rule;
 
   /// The operands to match. All rendered operands must be present even if the
   /// condition is always true.
   OperandVec Operands;
   bool NumOperandsCheck = true;
 
   std::string SymbolicName;
   unsigned InsnVarID;
 
 public:
   InstructionMatcher(RuleMatcher &Rule, StringRef SymbolicName)
       : Rule(Rule), SymbolicName(SymbolicName) {
     // We create a new instruction matcher.
     // Get a new ID for that instruction.
     InsnVarID = Rule.implicitlyDefineInsnVar(*this);
   }
 
   /// Construct a new instruction predicate and add it to the matcher.
   template <class Kind, class... Args>
   Optional<Kind *> addPredicate(Args &&... args) {
     Predicates.emplace_back(
         llvm::make_unique<Kind>(getInsnVarID(), std::forward<Args>(args)...));
     return static_cast<Kind *>(Predicates.back().get());
   }
 
   RuleMatcher &getRuleMatcher() const { return Rule; }
 
   unsigned getInsnVarID() const { return InsnVarID; }
 
   /// Add an operand to the matcher.
   OperandMatcher &addOperand(unsigned OpIdx, const std::string &SymbolicName,
                              unsigned AllocatedTemporariesBaseID) {
     Operands.emplace_back(new OperandMatcher(*this, OpIdx, SymbolicName,
                                              AllocatedTemporariesBaseID));
     if (!SymbolicName.empty())
       Rule.defineOperand(SymbolicName, *Operands.back());
 
     return *Operands.back();
   }
 
   OperandMatcher &getOperand(unsigned OpIdx) {
     auto I = std::find_if(Operands.begin(), Operands.end(),
                           [&OpIdx](const std::unique_ptr<OperandMatcher> &X) {
                             return X->getOpIdx() == OpIdx;
                           });
     if (I != Operands.end())
       return **I;
     llvm_unreachable("Failed to lookup operand");
   }
 
   StringRef getSymbolicName() const { return SymbolicName; }
   unsigned getNumOperands() const { return Operands.size(); }
   OperandVec::iterator operands_begin() { return Operands.begin(); }
   OperandVec::iterator operands_end() { return Operands.end(); }
   iterator_range<OperandVec::iterator> operands() {
     return make_range(operands_begin(), operands_end());
   }
   OperandVec::const_iterator operands_begin() const { return Operands.begin(); }
   OperandVec::const_iterator operands_end() const { return Operands.end(); }
   iterator_range<OperandVec::const_iterator> operands() const {
     return make_range(operands_begin(), operands_end());
   }
   bool operands_empty() const { return Operands.empty(); }
 
   void pop_front() { Operands.erase(Operands.begin()); }
 
   void optimize();
 
   /// Emit MatchTable opcodes that test whether the instruction named in
   /// InsnVarName matches all the predicates and all the operands.
   void emitPredicateOpcodes(MatchTable &Table, RuleMatcher &Rule) {
     if (NumOperandsCheck)
       InstructionNumOperandsMatcher(InsnVarID, getNumOperands())
           .emitPredicateOpcodes(Table, Rule);
 
     emitPredicateListOpcodes(Table, Rule);
 
     for (const auto &Operand : Operands)
       Operand->emitPredicateOpcodes(Table, Rule);
   }
 
   /// Compare the priority of this object and B.
   ///
   /// Returns true if this object is more important than B.
   bool isHigherPriorityThan(InstructionMatcher &B) {
     // Instruction matchers involving more operands have higher priority.
     if (Operands.size() > B.Operands.size())
       return true;
     if (Operands.size() < B.Operands.size())
       return false;
 
     for (auto &&P : zip(predicates(), B.predicates())) {
       auto L = static_cast<InstructionPredicateMatcher *>(std::get<0>(P).get());
       auto R = static_cast<InstructionPredicateMatcher *>(std::get<1>(P).get());
       if (L->isHigherPriorityThan(*R))
         return true;
       if (R->isHigherPriorityThan(*L))
         return false;
     }
 
     for (const auto &Operand : zip(Operands, B.Operands)) {
       if (std::get<0>(Operand)->isHigherPriorityThan(*std::get<1>(Operand)))
         return true;
       if (std::get<1>(Operand)->isHigherPriorityThan(*std::get<0>(Operand)))
         return false;
     }
 
     return false;
   };
 
   /// Report the maximum number of temporary operands needed by the instruction
   /// matcher.
   unsigned countRendererFns() {
     return std::accumulate(
                predicates().begin(), predicates().end(), 0,
                [](unsigned A,
                   const std::unique_ptr<PredicateMatcher> &Predicate) {
                  return A + Predicate->countRendererFns();
                }) +
            std::accumulate(
                Operands.begin(), Operands.end(), 0,
                [](unsigned A, const std::unique_ptr<OperandMatcher> &Operand) {
                  return A + Operand->countRendererFns();
                });
   }
 
   InstructionOpcodeMatcher &getOpcodeMatcher() {
     for (auto &P : predicates())
       if (auto *OpMatcher = dyn_cast<InstructionOpcodeMatcher>(P.get()))
         return *OpMatcher;
     llvm_unreachable("Didn't find an opcode matcher");
   }
 
   bool isConstantInstruction() {
     return getOpcodeMatcher().isConstantInstruction();
   }
 
   StringRef getOpcode() { return getOpcodeMatcher().getOpcode(); }
 };
 
 StringRef RuleMatcher::getOpcode() const {
   return Matchers.front()->getOpcode();
 }
 
 unsigned RuleMatcher::getNumOperands() const {
   return Matchers.front()->getNumOperands();
 }
 
-LLTCodeGen RuleMatcher::getFirstConditionAsRootType() {
-  InstructionMatcher &InsnMatcher = *Matchers.front();
-  if (!InsnMatcher.predicates_empty())
-    if (const auto *TM =
-            dyn_cast<LLTOperandMatcher>(&**InsnMatcher.predicates_begin()))
-      if (TM->getInsnVarID() == 0 && TM->getOpIdx() == 0)
-        return TM->getTy();
-  return {};
-}
-
 /// Generates code to check that the operand is a register defined by an
 /// instruction that matches the given instruction matcher.
 ///
 /// For example, the pattern:
 ///   (set $dst, (G_MUL (G_ADD $src1, $src2), $src3))
 /// would use an InstructionOperandMatcher for operand 1 of the G_MUL to match
 /// the:
 ///   (G_ADD $src1, $src2)
 /// subpattern.
 class InstructionOperandMatcher : public OperandPredicateMatcher {
 protected:
   std::unique_ptr<InstructionMatcher> InsnMatcher;
 
 public:
   InstructionOperandMatcher(unsigned InsnVarID, unsigned OpIdx,
                             RuleMatcher &Rule, StringRef SymbolicName)
       : OperandPredicateMatcher(OPM_Instruction, InsnVarID, OpIdx),
         InsnMatcher(new InstructionMatcher(Rule, SymbolicName)) {}
 
   static bool classof(const PredicateMatcher *P) {
     return P->getKind() == OPM_Instruction;
   }
 
   InstructionMatcher &getInsnMatcher() const { return *InsnMatcher; }
 
   void emitCaptureOpcodes(MatchTable &Table, RuleMatcher &Rule) const {
     const unsigned NewInsnVarID = InsnMatcher->getInsnVarID();
     Table << MatchTable::Opcode("GIM_RecordInsn")
           << MatchTable::Comment("DefineMI")
           << MatchTable::IntValue(NewInsnVarID) << MatchTable::Comment("MI")
           << MatchTable::IntValue(getInsnVarID())
           << MatchTable::Comment("OpIdx") << MatchTable::IntValue(getOpIdx())
           << MatchTable::Comment("MIs[" + llvm::to_string(NewInsnVarID) + "]")
           << MatchTable::LineBreak;
   }
 
   void emitPredicateOpcodes(MatchTable &Table,
                             RuleMatcher &Rule) const override {
     emitCaptureOpcodes(Table, Rule);
     InsnMatcher->emitPredicateOpcodes(Table, Rule);
   }
 
   bool isHigherPriorityThan(const OperandPredicateMatcher &B) const override {
     if (OperandPredicateMatcher::isHigherPriorityThan(B))
       return true;
     if (B.OperandPredicateMatcher::isHigherPriorityThan(*this))
       return false;
 
     if (const InstructionOperandMatcher *BP =
             dyn_cast<InstructionOperandMatcher>(&B))
       if (InsnMatcher->isHigherPriorityThan(*BP->InsnMatcher))
         return true;
     return false;
   }
 };
 
 void InstructionMatcher::optimize() {
   SmallVector<std::unique_ptr<PredicateMatcher>, 8> Stash;
   const auto &OpcMatcher = getOpcodeMatcher();
 
   Stash.push_back(predicates_pop_front());
   if (Stash.back().get() == &OpcMatcher) {
     if (NumOperandsCheck && OpcMatcher.getNumOperands() < getNumOperands())
       Stash.emplace_back(
           new InstructionNumOperandsMatcher(InsnVarID, getNumOperands()));
     NumOperandsCheck = false;
   }
 
   if (InsnVarID > 0) {
     assert(!Operands.empty() && "Nested instruction is expected to def a vreg");
     for (auto &OP : Operands[0]->predicates())
       OP.reset();
     Operands[0]->eraseNullPredicates();
   }
   while (!Stash.empty())
     prependPredicate(Stash.pop_back_val());
 }
 
 //===- Actions ------------------------------------------------------------===//
 class OperandRenderer {
 public:
   enum RendererKind {
     OR_Copy,
     OR_CopyOrAddZeroReg,
     OR_CopySubReg,
     OR_CopyConstantAsImm,
     OR_CopyFConstantAsFPImm,
     OR_Imm,
     OR_Register,
     OR_TempRegister,
     OR_ComplexPattern,
     OR_Custom
   };
 
 protected:
   RendererKind Kind;
 
 public:
   OperandRenderer(RendererKind Kind) : Kind(Kind) {}
   virtual ~OperandRenderer() {}
 
   RendererKind getKind() const { return Kind; }
 
   virtual void emitRenderOpcodes(MatchTable &Table,
                                  RuleMatcher &Rule) const = 0;
 };
 
 /// A CopyRenderer emits code to copy a single operand from an existing
 /// instruction to the one being built.
 class CopyRenderer : public OperandRenderer {
 protected:
   unsigned NewInsnID;
   /// The name of the operand.
   const StringRef SymbolicName;
 
 public:
   CopyRenderer(unsigned NewInsnID, StringRef SymbolicName)
       : OperandRenderer(OR_Copy), NewInsnID(NewInsnID),
         SymbolicName(SymbolicName) {
     assert(!SymbolicName.empty() && "Cannot copy from an unspecified source");
   }
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_Copy;
   }
 
   const StringRef getSymbolicName() const { return SymbolicName; }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     const OperandMatcher &Operand = Rule.getOperandMatcher(SymbolicName);
     unsigned OldInsnVarID = Rule.getInsnVarID(Operand.getInstructionMatcher());
     Table << MatchTable::Opcode("GIR_Copy") << MatchTable::Comment("NewInsnID")
           << MatchTable::IntValue(NewInsnID) << MatchTable::Comment("OldInsnID")
           << MatchTable::IntValue(OldInsnVarID) << MatchTable::Comment("OpIdx")
           << MatchTable::IntValue(Operand.getOpIdx())
           << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
   }
 };
 
 /// A CopyOrAddZeroRegRenderer emits code to copy a single operand from an
 /// existing instruction to the one being built. If the operand turns out to be
 /// a 'G_CONSTANT 0' then it replaces the operand with a zero register.
 class CopyOrAddZeroRegRenderer : public OperandRenderer {
 protected:
   unsigned NewInsnID;
   /// The name of the operand.
   const StringRef SymbolicName;
   const Record *ZeroRegisterDef;
 
 public:
   CopyOrAddZeroRegRenderer(unsigned NewInsnID,
                            StringRef SymbolicName, Record *ZeroRegisterDef)
       : OperandRenderer(OR_CopyOrAddZeroReg), NewInsnID(NewInsnID),
         SymbolicName(SymbolicName), ZeroRegisterDef(ZeroRegisterDef) {
     assert(!SymbolicName.empty() && "Cannot copy from an unspecified source");
   }
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_CopyOrAddZeroReg;
   }
 
   const StringRef getSymbolicName() const { return SymbolicName; }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     const OperandMatcher &Operand = Rule.getOperandMatcher(SymbolicName);
     unsigned OldInsnVarID = Rule.getInsnVarID(Operand.getInstructionMatcher());
     Table << MatchTable::Opcode("GIR_CopyOrAddZeroReg")
           << MatchTable::Comment("NewInsnID") << MatchTable::IntValue(NewInsnID)
           << MatchTable::Comment("OldInsnID")
           << MatchTable::IntValue(OldInsnVarID) << MatchTable::Comment("OpIdx")
           << MatchTable::IntValue(Operand.getOpIdx())
           << MatchTable::NamedValue(
                  (ZeroRegisterDef->getValue("Namespace")
                       ? ZeroRegisterDef->getValueAsString("Namespace")
                       : ""),
                  ZeroRegisterDef->getName())
           << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
   }
 };
 
 /// A CopyConstantAsImmRenderer emits code to render a G_CONSTANT instruction to
 /// an extended immediate operand.
 class CopyConstantAsImmRenderer : public OperandRenderer {
 protected:
   unsigned NewInsnID;
   /// The name of the operand.
   const std::string SymbolicName;
   bool Signed;
 
 public:
   CopyConstantAsImmRenderer(unsigned NewInsnID, StringRef SymbolicName)
       : OperandRenderer(OR_CopyConstantAsImm), NewInsnID(NewInsnID),
         SymbolicName(SymbolicName), Signed(true) {}
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_CopyConstantAsImm;
   }
 
   const StringRef getSymbolicName() const { return SymbolicName; }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     InstructionMatcher &InsnMatcher = Rule.getInstructionMatcher(SymbolicName);
     unsigned OldInsnVarID = Rule.getInsnVarID(InsnMatcher);
     Table << MatchTable::Opcode(Signed ? "GIR_CopyConstantAsSImm"
                                        : "GIR_CopyConstantAsUImm")
           << MatchTable::Comment("NewInsnID") << MatchTable::IntValue(NewInsnID)
           << MatchTable::Comment("OldInsnID")
           << MatchTable::IntValue(OldInsnVarID)
           << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
   }
 };
 
 /// A CopyFConstantAsFPImmRenderer emits code to render a G_FCONSTANT
 /// instruction to an extended immediate operand.
 class CopyFConstantAsFPImmRenderer : public OperandRenderer {
 protected:
   unsigned NewInsnID;
   /// The name of the operand.
   const std::string SymbolicName;
 
 public:
   CopyFConstantAsFPImmRenderer(unsigned NewInsnID, StringRef SymbolicName)
       : OperandRenderer(OR_CopyFConstantAsFPImm), NewInsnID(NewInsnID),
         SymbolicName(SymbolicName) {}
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_CopyFConstantAsFPImm;
   }
 
   const StringRef getSymbolicName() const { return SymbolicName; }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     InstructionMatcher &InsnMatcher = Rule.getInstructionMatcher(SymbolicName);
     unsigned OldInsnVarID = Rule.getInsnVarID(InsnMatcher);
     Table << MatchTable::Opcode("GIR_CopyFConstantAsFPImm")
           << MatchTable::Comment("NewInsnID") << MatchTable::IntValue(NewInsnID)
           << MatchTable::Comment("OldInsnID")
           << MatchTable::IntValue(OldInsnVarID)
           << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
   }
 };
 
 /// A CopySubRegRenderer emits code to copy a single register operand from an
 /// existing instruction to the one being built and indicate that only a
 /// subregister should be copied.
 class CopySubRegRenderer : public OperandRenderer {
 protected:
   unsigned NewInsnID;
   /// The name of the operand.
   const StringRef SymbolicName;
   /// The subregister to extract.
   const CodeGenSubRegIndex *SubReg;
 
 public:
   CopySubRegRenderer(unsigned NewInsnID, StringRef SymbolicName,
                      const CodeGenSubRegIndex *SubReg)
       : OperandRenderer(OR_CopySubReg), NewInsnID(NewInsnID),
         SymbolicName(SymbolicName), SubReg(SubReg) {}
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_CopySubReg;
   }
 
   const StringRef getSymbolicName() const { return SymbolicName; }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     const OperandMatcher &Operand = Rule.getOperandMatcher(SymbolicName);
     unsigned OldInsnVarID = Rule.getInsnVarID(Operand.getInstructionMatcher());
     Table << MatchTable::Opcode("GIR_CopySubReg")
           << MatchTable::Comment("NewInsnID") << MatchTable::IntValue(NewInsnID)
           << MatchTable::Comment("OldInsnID")
           << MatchTable::IntValue(OldInsnVarID) << MatchTable::Comment("OpIdx")
           << MatchTable::IntValue(Operand.getOpIdx())
           << MatchTable::Comment("SubRegIdx")
           << MatchTable::IntValue(SubReg->EnumValue)
           << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
   }
 };
 
 /// Adds a specific physical register to the instruction being built.
 /// This is typically useful for WZR/XZR on AArch64.
 class AddRegisterRenderer : public OperandRenderer {
 protected:
   unsigned InsnID;
   const Record *RegisterDef;
 
 public:
   AddRegisterRenderer(unsigned InsnID, const Record *RegisterDef)
       : OperandRenderer(OR_Register), InsnID(InsnID), RegisterDef(RegisterDef) {
   }
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_Register;
   }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIR_AddRegister")
           << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
           << MatchTable::NamedValue(
                  (RegisterDef->getValue("Namespace")
                       ? RegisterDef->getValueAsString("Namespace")
                       : ""),
                  RegisterDef->getName())
           << MatchTable::LineBreak;
   }
 };
 
 /// Adds a specific temporary virtual register to the instruction being built.
 /// This is used to chain instructions together when emitting multiple
 /// instructions.
 class TempRegRenderer : public OperandRenderer {
 protected:
   unsigned InsnID;
   unsigned TempRegID;
   bool IsDef;
 
 public:
   TempRegRenderer(unsigned InsnID, unsigned TempRegID, bool IsDef = false)
       : OperandRenderer(OR_Register), InsnID(InsnID), TempRegID(TempRegID),
         IsDef(IsDef) {}
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_TempRegister;
   }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIR_AddTempRegister")
           << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
           << MatchTable::Comment("TempRegID") << MatchTable::IntValue(TempRegID)
           << MatchTable::Comment("TempRegFlags");
     if (IsDef)
       Table << MatchTable::NamedValue("RegState::Define");
     else
       Table << MatchTable::IntValue(0);
     Table << MatchTable::LineBreak;
   }
 };
 
 /// Adds a specific immediate to the instruction being built.
 class ImmRenderer : public OperandRenderer {
 protected:
   unsigned InsnID;
   int64_t Imm;
 
 public:
   ImmRenderer(unsigned InsnID, int64_t Imm)
       : OperandRenderer(OR_Imm), InsnID(InsnID), Imm(Imm) {}
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_Imm;
   }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIR_AddImm") << MatchTable::Comment("InsnID")
           << MatchTable::IntValue(InsnID) << MatchTable::Comment("Imm")
           << MatchTable::IntValue(Imm) << MatchTable::LineBreak;
   }
 };
 
 /// Adds operands by calling a renderer function supplied by the ComplexPattern
 /// matcher function.
 class RenderComplexPatternOperand : public OperandRenderer {
 private:
   unsigned InsnID;
   const Record &TheDef;
   /// The name of the operand.
   const StringRef SymbolicName;
   /// The renderer number. This must be unique within a rule since it's used to
   /// identify a temporary variable to hold the renderer function.
   unsigned RendererID;
   /// When provided, this is the suboperand of the ComplexPattern operand to
   /// render. Otherwise all the suboperands will be rendered.
   Optional<unsigned> SubOperand;
 
   unsigned getNumOperands() const {
     return TheDef.getValueAsDag("Operands")->getNumArgs();
   }
 
 public:
   RenderComplexPatternOperand(unsigned InsnID, const Record &TheDef,
                               StringRef SymbolicName, unsigned RendererID,
                               Optional<unsigned> SubOperand = None)
       : OperandRenderer(OR_ComplexPattern), InsnID(InsnID), TheDef(TheDef),
         SymbolicName(SymbolicName), RendererID(RendererID),
         SubOperand(SubOperand) {}
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_ComplexPattern;
   }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode(SubOperand.hasValue() ? "GIR_ComplexSubOperandRenderer"
                                                       : "GIR_ComplexRenderer")
           << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
           << MatchTable::Comment("RendererID")
           << MatchTable::IntValue(RendererID);
     if (SubOperand.hasValue())
       Table << MatchTable::Comment("SubOperand")
             << MatchTable::IntValue(SubOperand.getValue());
     Table << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
   }
 };
 
 class CustomRenderer : public OperandRenderer {
 protected:
   unsigned InsnID;
   const Record &Renderer;
   /// The name of the operand.
   const std::string SymbolicName;
 
 public:
   CustomRenderer(unsigned InsnID, const Record &Renderer,
                  StringRef SymbolicName)
       : OperandRenderer(OR_Custom), InsnID(InsnID), Renderer(Renderer),
         SymbolicName(SymbolicName) {}
 
   static bool classof(const OperandRenderer *R) {
     return R->getKind() == OR_Custom;
   }
 
   void emitRenderOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     InstructionMatcher &InsnMatcher = Rule.getInstructionMatcher(SymbolicName);
     unsigned OldInsnVarID = Rule.getInsnVarID(InsnMatcher);
     Table << MatchTable::Opcode("GIR_CustomRenderer")
           << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
           << MatchTable::Comment("OldInsnID")
           << MatchTable::IntValue(OldInsnVarID)
           << MatchTable::Comment("Renderer")
           << MatchTable::NamedValue(
                  "GICR_" + Renderer.getValueAsString("RendererFn").str())
           << MatchTable::Comment(SymbolicName) << MatchTable::LineBreak;
   }
 };
 
 /// An action taken when all Matcher predicates succeeded for a parent rule.
 ///
 /// Typical actions include:
 /// * Changing the opcode of an instruction.
 /// * Adding an operand to an instruction.
 class MatchAction {
 public:
   virtual ~MatchAction() {}
 
   /// Emit the MatchTable opcodes to implement the action.
   virtual void emitActionOpcodes(MatchTable &Table,
                                  RuleMatcher &Rule) const = 0;
 };
 
 /// Generates a comment describing the matched rule being acted upon.
 class DebugCommentAction : public MatchAction {
 private:
   std::string S;
 
 public:
   DebugCommentAction(StringRef S) : S(S) {}
 
   void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Comment(S) << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to build an instruction or mutate an existing instruction
 /// into the desired instruction when this is possible.
 class BuildMIAction : public MatchAction {
 private:
   unsigned InsnID;
   const CodeGenInstruction *I;
   InstructionMatcher *Matched;
   std::vector<std::unique_ptr<OperandRenderer>> OperandRenderers;
 
   /// True if the instruction can be built solely by mutating the opcode.
   bool canMutate(RuleMatcher &Rule, const InstructionMatcher *Insn) const {
     if (!Insn)
       return false;
 
     if (OperandRenderers.size() != Insn->getNumOperands())
       return false;
 
     for (const auto &Renderer : enumerate(OperandRenderers)) {
       if (const auto *Copy = dyn_cast<CopyRenderer>(&*Renderer.value())) {
         const OperandMatcher &OM = Rule.getOperandMatcher(Copy->getSymbolicName());
         if (Insn != &OM.getInstructionMatcher() ||
             OM.getOpIdx() != Renderer.index())
           return false;
       } else
         return false;
     }
 
     return true;
   }
 
 public:
   BuildMIAction(unsigned InsnID, const CodeGenInstruction *I)
       : InsnID(InsnID), I(I), Matched(nullptr) {}
 
   unsigned getInsnID() const { return InsnID; }
   const CodeGenInstruction *getCGI() const { return I; }
 
   void chooseInsnToMutate(RuleMatcher &Rule) {
     for (auto *MutateCandidate : Rule.mutatable_insns()) {
       if (canMutate(Rule, MutateCandidate)) {
         // Take the first one we're offered that we're able to mutate.
         Rule.reserveInsnMatcherForMutation(MutateCandidate);
         Matched = MutateCandidate;
         return;
       }
     }
   }
 
   template <class Kind, class... Args>
   Kind &addRenderer(Args&&... args) {
     OperandRenderers.emplace_back(
         llvm::make_unique<Kind>(InsnID, std::forward<Args>(args)...));
     return *static_cast<Kind *>(OperandRenderers.back().get());
   }
 
   void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     if (Matched) {
       assert(canMutate(Rule, Matched) &&
              "Arranged to mutate an insn that isn't mutatable");
 
       unsigned RecycleInsnID = Rule.getInsnVarID(*Matched);
       Table << MatchTable::Opcode("GIR_MutateOpcode")
             << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
             << MatchTable::Comment("RecycleInsnID")
             << MatchTable::IntValue(RecycleInsnID)
             << MatchTable::Comment("Opcode")
             << MatchTable::NamedValue(I->Namespace, I->TheDef->getName())
             << MatchTable::LineBreak;
 
       if (!I->ImplicitDefs.empty() || !I->ImplicitUses.empty()) {
         for (auto Def : I->ImplicitDefs) {
           auto Namespace = Def->getValue("Namespace")
                                ? Def->getValueAsString("Namespace")
                                : "";
           Table << MatchTable::Opcode("GIR_AddImplicitDef")
                 << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
                 << MatchTable::NamedValue(Namespace, Def->getName())
                 << MatchTable::LineBreak;
         }
         for (auto Use : I->ImplicitUses) {
           auto Namespace = Use->getValue("Namespace")
                                ? Use->getValueAsString("Namespace")
                                : "";
           Table << MatchTable::Opcode("GIR_AddImplicitUse")
                 << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
                 << MatchTable::NamedValue(Namespace, Use->getName())
                 << MatchTable::LineBreak;
         }
       }
       return;
     }
 
     // TODO: Simple permutation looks like it could be almost as common as
     //       mutation due to commutative operations.
 
     Table << MatchTable::Opcode("GIR_BuildMI") << MatchTable::Comment("InsnID")
           << MatchTable::IntValue(InsnID) << MatchTable::Comment("Opcode")
           << MatchTable::NamedValue(I->Namespace, I->TheDef->getName())
           << MatchTable::LineBreak;
     for (const auto &Renderer : OperandRenderers)
       Renderer->emitRenderOpcodes(Table, Rule);
 
     if (I->mayLoad || I->mayStore) {
       Table << MatchTable::Opcode("GIR_MergeMemOperands")
             << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
             << MatchTable::Comment("MergeInsnID's");
       // Emit the ID's for all the instructions that are matched by this rule.
       // TODO: Limit this to matched instructions that mayLoad/mayStore or have
       //       some other means of having a memoperand. Also limit this to
       //       emitted instructions that expect to have a memoperand too. For
       //       example, (G_SEXT (G_LOAD x)) that results in separate load and
       //       sign-extend instructions shouldn't put the memoperand on the
       //       sign-extend since it has no effect there.
       std::vector<unsigned> MergeInsnIDs;
       for (const auto &IDMatcherPair : Rule.defined_insn_vars())
         MergeInsnIDs.push_back(IDMatcherPair.second);
       llvm::sort(MergeInsnIDs.begin(), MergeInsnIDs.end());
       for (const auto &MergeInsnID : MergeInsnIDs)
         Table << MatchTable::IntValue(MergeInsnID);
       Table << MatchTable::NamedValue("GIU_MergeMemOperands_EndOfList")
             << MatchTable::LineBreak;
     }
 
     // FIXME: This is a hack but it's sufficient for ISel. We'll need to do
     //        better for combines. Particularly when there are multiple match
     //        roots.
     if (InsnID == 0)
       Table << MatchTable::Opcode("GIR_EraseFromParent")
             << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
             << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to constrain the operands of an output instruction to the
 /// register classes specified by the definition of that instruction.
 class ConstrainOperandsToDefinitionAction : public MatchAction {
   unsigned InsnID;
 
 public:
   ConstrainOperandsToDefinitionAction(unsigned InsnID) : InsnID(InsnID) {}
 
   void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIR_ConstrainSelectedInstOperands")
           << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
           << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to constrain the specified operand of an output instruction
 /// to the specified register class.
 class ConstrainOperandToRegClassAction : public MatchAction {
   unsigned InsnID;
   unsigned OpIdx;
   const CodeGenRegisterClass &RC;
 
 public:
   ConstrainOperandToRegClassAction(unsigned InsnID, unsigned OpIdx,
                                    const CodeGenRegisterClass &RC)
       : InsnID(InsnID), OpIdx(OpIdx), RC(RC) {}
 
   void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIR_ConstrainOperandRC")
           << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
           << MatchTable::Comment("Op") << MatchTable::IntValue(OpIdx)
           << MatchTable::Comment("RC " + RC.getName())
           << MatchTable::IntValue(RC.EnumValue) << MatchTable::LineBreak;
   }
 };
 
 /// Generates code to create a temporary register which can be used to chain
 /// instructions together.
 class MakeTempRegisterAction : public MatchAction {
 private:
   LLTCodeGen Ty;
   unsigned TempRegID;
 
 public:
   MakeTempRegisterAction(const LLTCodeGen &Ty, unsigned TempRegID)
       : Ty(Ty), TempRegID(TempRegID) {}
 
   void emitActionOpcodes(MatchTable &Table, RuleMatcher &Rule) const override {
     Table << MatchTable::Opcode("GIR_MakeTempReg")
           << MatchTable::Comment("TempRegID") << MatchTable::IntValue(TempRegID)
           << MatchTable::Comment("TypeID")
           << MatchTable::NamedValue(Ty.getCxxEnumValue())
           << MatchTable::LineBreak;
   }
 };
 
 InstructionMatcher &RuleMatcher::addInstructionMatcher(StringRef SymbolicName) {
   Matchers.emplace_back(new InstructionMatcher(*this, SymbolicName));
   MutatableInsns.insert(Matchers.back().get());
   return *Matchers.back();
 }
 
 void RuleMatcher::addRequiredFeature(Record *Feature) {
   RequiredFeatures.push_back(Feature);
 }
 
 const std::vector<Record *> &RuleMatcher::getRequiredFeatures() const {
   return RequiredFeatures;
 }
 
 // Emplaces an action of the specified Kind at the end of the action list.
 //
 // Returns a reference to the newly created action.
 //
 // Like std::vector::emplace_back(), may invalidate all iterators if the new
 // size exceeds the capacity. Otherwise, only invalidates the past-the-end
 // iterator.
 template <class Kind, class... Args>
 Kind &RuleMatcher::addAction(Args &&... args) {
   Actions.emplace_back(llvm::make_unique<Kind>(std::forward<Args>(args)...));
   return *static_cast<Kind *>(Actions.back().get());
 }
 
 // Emplaces an action of the specified Kind before the given insertion point.
 //
 // Returns an iterator pointing at the newly created instruction.
 //
 // Like std::vector::insert(), may invalidate all iterators if the new size
 // exceeds the capacity. Otherwise, only invalidates the iterators from the
 // insertion point onwards.
 template <class Kind, class... Args>
 action_iterator RuleMatcher::insertAction(action_iterator InsertPt,
                                           Args &&... args) {
   return Actions.emplace(InsertPt,
                          llvm::make_unique<Kind>(std::forward<Args>(args)...));
 }
 
 unsigned RuleMatcher::implicitlyDefineInsnVar(InstructionMatcher &Matcher) {
   unsigned NewInsnVarID = NextInsnVarID++;
   InsnVariableIDs[&Matcher] = NewInsnVarID;
   return NewInsnVarID;
 }
 
 unsigned RuleMatcher::getInsnVarID(InstructionMatcher &InsnMatcher) const {
   const auto &I = InsnVariableIDs.find(&InsnMatcher);
   if (I != InsnVariableIDs.end())
     return I->second;
   llvm_unreachable("Matched Insn was not captured in a local variable");
 }
 
 void RuleMatcher::defineOperand(StringRef SymbolicName, OperandMatcher &OM) {
   if (DefinedOperands.find(SymbolicName) == DefinedOperands.end()) {
     DefinedOperands[SymbolicName] = &OM;
     return;
   }
 
   // If the operand is already defined, then we must ensure both references in
   // the matcher have the exact same node.
   OM.addPredicate<SameOperandMatcher>(OM.getSymbolicName());
 }
 
 InstructionMatcher &
 RuleMatcher::getInstructionMatcher(StringRef SymbolicName) const {
   for (const auto &I : InsnVariableIDs)
     if (I.first->getSymbolicName() == SymbolicName)
       return *I.first;
   llvm_unreachable(
       ("Failed to lookup instruction " + SymbolicName).str().c_str());
 }
 
 const OperandMatcher &
 RuleMatcher::getOperandMatcher(StringRef Name) const {
   const auto &I = DefinedOperands.find(Name);
 
   if (I == DefinedOperands.end())
     PrintFatalError(SrcLoc, "Operand " + Name + " was not declared in matcher");
 
   return *I->second;
 }
 
 void RuleMatcher::emit(MatchTable &Table) {
   if (Matchers.empty())
     llvm_unreachable("Unexpected empty matcher!");
 
   // The representation supports rules that require multiple roots such as:
   //    %ptr(p0) = ...
   //    %elt0(s32) = G_LOAD %ptr
   //    %1(p0) = G_ADD %ptr, 4
   //    %elt1(s32) = G_LOAD p0 %1
   // which could be usefully folded into:
   //    %ptr(p0) = ...
   //    %elt0(s32), %elt1(s32) = TGT_LOAD_PAIR %ptr
   // on some targets but we don't need to make use of that yet.
   assert(Matchers.size() == 1 && "Cannot handle multi-root matchers yet");
 
   unsigned LabelID = Table.allocateLabelID();
   Table << MatchTable::Opcode("GIM_Try", +1)
         << MatchTable::Comment("On fail goto")
         << MatchTable::JumpTarget(LabelID)
         << MatchTable::Comment(("Rule ID " + Twine(RuleID) + " //").str())
         << MatchTable::LineBreak;
 
   if (!RequiredFeatures.empty()) {
     Table << MatchTable::Opcode("GIM_CheckFeatures")
           << MatchTable::NamedValue(getNameForFeatureBitset(RequiredFeatures))
           << MatchTable::LineBreak;
   }
 
   Matchers.front()->emitPredicateOpcodes(Table, *this);
 
   // We must also check if it's safe to fold the matched instructions.
   if (InsnVariableIDs.size() >= 2) {
     // Invert the map to create stable ordering (by var names)
     SmallVector<unsigned, 2> InsnIDs;
     for (const auto &Pair : InsnVariableIDs) {
       // Skip the root node since it isn't moving anywhere. Everything else is
       // sinking to meet it.
       if (Pair.first == Matchers.front().get())
         continue;
 
       InsnIDs.push_back(Pair.second);
     }
     llvm::sort(InsnIDs.begin(), InsnIDs.end());
 
     for (const auto &InsnID : InsnIDs) {
       // Reject the difficult cases until we have a more accurate check.
       Table << MatchTable::Opcode("GIM_CheckIsSafeToFold")
             << MatchTable::Comment("InsnID") << MatchTable::IntValue(InsnID)
             << MatchTable::LineBreak;
 
       // FIXME: Emit checks to determine it's _actually_ safe to fold and/or
       //        account for unsafe cases.
       //
       //        Example:
       //          MI1--> %0 = ...
       //                 %1 = ... %0
       //          MI0--> %2 = ... %0
       //          It's not safe to erase MI1. We currently handle this by not
       //          erasing %0 (even when it's dead).
       //
       //        Example:
       //          MI1--> %0 = load volatile @a
       //                 %1 = load volatile @a
       //          MI0--> %2 = ... %0
       //          It's not safe to sink %0's def past %1. We currently handle
       //          this by rejecting all loads.
       //
       //        Example:
       //          MI1--> %0 = load @a
       //                 %1 = store @a
       //          MI0--> %2 = ... %0
       //          It's not safe to sink %0's def past %1. We currently handle
       //          this by rejecting all loads.
       //
       //        Example:
       //                   G_CONDBR %cond, @BB1
       //                 BB0:
       //          MI1-->   %0 = load @a
       //                   G_BR @BB1
       //                 BB1:
       //          MI0-->   %2 = ... %0
       //          It's not always safe to sink %0 across control flow. In this
       //          case it may introduce a memory fault. We currentl handle this
       //          by rejecting all loads.
     }
   }
 
   for (const auto &PM : EpilogueMatchers)
     PM->emitPredicateOpcodes(Table, *this);
 
   for (const auto &MA : Actions)
     MA->emitActionOpcodes(Table, *this);
 
   if (Table.isWithCoverage())
     Table << MatchTable::Opcode("GIR_Coverage") << MatchTable::IntValue(RuleID)
           << MatchTable::LineBreak;
   else
     Table << MatchTable::Comment(("GIR_Coverage, " + Twine(RuleID) + ",").str())
           << MatchTable::LineBreak;
 
   Table << MatchTable::Opcode("GIR_Done", -1) << MatchTable::LineBreak
         << MatchTable::Label(LabelID);
   ++NumPatternEmitted;
 }
 
 bool RuleMatcher::isHigherPriorityThan(const RuleMatcher &B) const {
   // Rules involving more match roots have higher priority.
   if (Matchers.size() > B.Matchers.size())
     return true;
   if (Matchers.size() < B.Matchers.size())
     return false;
 
   for (const auto &Matcher : zip(Matchers, B.Matchers)) {
     if (std::get<0>(Matcher)->isHigherPriorityThan(*std::get<1>(Matcher)))
       return true;
     if (std::get<1>(Matcher)->isHigherPriorityThan(*std::get<0>(Matcher)))
       return false;
   }
 
   return false;
 }
 
 unsigned RuleMatcher::countRendererFns() const {
   return std::accumulate(
       Matchers.begin(), Matchers.end(), 0,
       [](unsigned A, const std::unique_ptr<InstructionMatcher> &Matcher) {
         return A + Matcher->countRendererFns();
       });
 }
 
 bool OperandPredicateMatcher::isHigherPriorityThan(
     const OperandPredicateMatcher &B) const {
   // Generally speaking, an instruction is more important than an Int or a
   // LiteralInt because it can cover more nodes but theres an exception to
   // this. G_CONSTANT's are less important than either of those two because they
   // are more permissive.
 
   const InstructionOperandMatcher *AOM =
       dyn_cast<InstructionOperandMatcher>(this);
   const InstructionOperandMatcher *BOM =
       dyn_cast<InstructionOperandMatcher>(&B);
   bool AIsConstantInsn = AOM && AOM->getInsnMatcher().isConstantInstruction();
   bool BIsConstantInsn = BOM && BOM->getInsnMatcher().isConstantInstruction();
 
   if (AOM && BOM) {
     // The relative priorities between a G_CONSTANT and any other instruction
     // don't actually matter but this code is needed to ensure a strict weak
     // ordering. This is particularly important on Windows where the rules will
     // be incorrectly sorted without it.
     if (AIsConstantInsn != BIsConstantInsn)
       return AIsConstantInsn < BIsConstantInsn;
     return false;
   }
 
   if (AOM && AIsConstantInsn && (B.Kind == OPM_Int || B.Kind == OPM_LiteralInt))
     return false;
   if (BOM && BIsConstantInsn && (Kind == OPM_Int || Kind == OPM_LiteralInt))
     return true;
 
   return Kind < B.Kind;
 }
 
 void SameOperandMatcher::emitPredicateOpcodes(MatchTable &Table,
                                               RuleMatcher &Rule) const {
   const OperandMatcher &OtherOM = Rule.getOperandMatcher(MatchingName);
   unsigned OtherInsnVarID = Rule.getInsnVarID(OtherOM.getInstructionMatcher());
   assert(OtherInsnVarID == OtherOM.getInstructionMatcher().getInsnVarID());
 
   Table << MatchTable::Opcode("GIM_CheckIsSameOperand")
         << MatchTable::Comment("MI") << MatchTable::IntValue(InsnVarID)
         << MatchTable::Comment("OpIdx") << MatchTable::IntValue(OpIdx)
         << MatchTable::Comment("OtherMI")
         << MatchTable::IntValue(OtherInsnVarID)
         << MatchTable::Comment("OtherOpIdx")
         << MatchTable::IntValue(OtherOM.getOpIdx())
         << MatchTable::LineBreak;
 }
 
 //===- GlobalISelEmitter class --------------------------------------------===//
 
 class GlobalISelEmitter {
 public:
   explicit GlobalISelEmitter(RecordKeeper &RK);
   void run(raw_ostream &OS);
 
 private:
   const RecordKeeper &RK;
   const CodeGenDAGPatterns CGP;
   const CodeGenTarget &Target;
   CodeGenRegBank CGRegs;
 
   /// Keep track of the equivalence between SDNodes and Instruction by mapping
   /// SDNodes to the GINodeEquiv mapping. We need to map to the GINodeEquiv to
   /// check for attributes on the relation such as CheckMMOIsNonAtomic.
   /// This is defined using 'GINodeEquiv' in the target description.
   DenseMap<Record *, Record *> NodeEquivs;
 
   /// Keep track of the equivalence between ComplexPattern's and
   /// GIComplexOperandMatcher. Map entries are specified by subclassing
   /// GIComplexPatternEquiv.
   DenseMap<const Record *, const Record *> ComplexPatternEquivs;
 
   /// Keep track of the equivalence between SDNodeXForm's and
   /// GICustomOperandRenderer. Map entries are specified by subclassing
   /// GISDNodeXFormEquiv.
   DenseMap<const Record *, const Record *> SDNodeXFormEquivs;
 
   /// Keep track of Scores of PatternsToMatch similar to how the DAG does.
   /// This adds compatibility for RuleMatchers to use this for ordering rules.
   DenseMap<uint64_t, int> RuleMatcherScores;
 
   // Map of predicates to their subtarget features.
   SubtargetFeatureInfoMap SubtargetFeatures;
 
   // Rule coverage information.
   Optional<CodeGenCoverage> RuleCoverage;
 
   void gatherOpcodeValues();
   void gatherTypeIDValues();
   void gatherNodeEquivs();
   Record *findNodeEquiv(Record *N) const;
   const CodeGenInstruction *getEquivNode(Record &Equiv,
                                          const TreePatternNode *N) const;
 
   Error importRulePredicates(RuleMatcher &M, ArrayRef<Predicate> Predicates);
   Expected<InstructionMatcher &> createAndImportSelDAGMatcher(
       RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
       const TreePatternNode *Src, unsigned &TempOpIdx) const;
   Error importComplexPatternOperandMatcher(OperandMatcher &OM, Record *R,
                                            unsigned &TempOpIdx) const;
   Error importChildMatcher(RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
                            const TreePatternNode *SrcChild,
                            bool OperandIsAPointer, unsigned OpIdx,
                            unsigned &TempOpIdx) const;
 
   Expected<BuildMIAction &>
   createAndImportInstructionRenderer(RuleMatcher &M,
                                      const TreePatternNode *Dst);
   Expected<action_iterator> createAndImportSubInstructionRenderer(
       action_iterator InsertPt, RuleMatcher &M, const TreePatternNode *Dst,
       unsigned TempReg);
   Expected<action_iterator>
   createInstructionRenderer(action_iterator InsertPt, RuleMatcher &M,
                             const TreePatternNode *Dst);
   void importExplicitDefRenderers(BuildMIAction &DstMIBuilder);
   Expected<action_iterator>
   importExplicitUseRenderers(action_iterator InsertPt, RuleMatcher &M,
                              BuildMIAction &DstMIBuilder,
                              const llvm::TreePatternNode *Dst);
   Expected<action_iterator>
   importExplicitUseRenderer(action_iterator InsertPt, RuleMatcher &Rule,
                             BuildMIAction &DstMIBuilder,
                             TreePatternNode *DstChild);
   Error importDefaultOperandRenderers(BuildMIAction &DstMIBuilder,
                                       DagInit *DefaultOps) const;
   Error
   importImplicitDefRenderers(BuildMIAction &DstMIBuilder,
                              const std::vector<Record *> &ImplicitDefs) const;
 
   void emitImmPredicates(raw_ostream &OS, StringRef TypeIdentifier,
                          StringRef Type,
                          std::function<bool(const Record *R)> Filter);
 
   /// Analyze pattern \p P, returning a matcher for it if possible.
   /// Otherwise, return an Error explaining why we don't support it.
   Expected<RuleMatcher> runOnPattern(const PatternToMatch &P);
 
   void declareSubtargetFeature(Record *Predicate);
 
   MatchTable buildMatchTable(MutableArrayRef<RuleMatcher> Rules, bool Optimize,
                              bool WithCoverage);
 
 public:
   /// Takes a sequence of \p Rules and group them based on the predicates
   /// they share. \p MatcherStorage is used as a memory container
   /// for the group that are created as part of this process.
   ///
   /// What this optimization does looks like if GroupT = GroupMatcher:
   /// Output without optimization:
   /// \verbatim
   /// # R1
   ///  # predicate A
   ///  # predicate B
   ///  ...
   /// # R2
   ///  # predicate A // <-- effectively this is going to be checked twice.
   ///                //     Once in R1 and once in R2.
   ///  # predicate C
   /// \endverbatim
   /// Output with optimization:
   /// \verbatim
   /// # Group1_2
   ///  # predicate A // <-- Check is now shared.
   ///  # R1
   ///   # predicate B
   ///  # R2
   ///   # predicate C
   /// \endverbatim
   template <class GroupT>
   static std::vector<Matcher *> optimizeRules(
       ArrayRef<Matcher *> Rules,
       std::vector<std::unique_ptr<Matcher>> &MatcherStorage);
 };
 
 void GlobalISelEmitter::gatherOpcodeValues() {
   InstructionOpcodeMatcher::initOpcodeValuesMap(Target);
 }
 
 void GlobalISelEmitter::gatherTypeIDValues() {
   LLTOperandMatcher::initTypeIDValuesMap();
 }
 
 void GlobalISelEmitter::gatherNodeEquivs() {
   assert(NodeEquivs.empty());
   for (Record *Equiv : RK.getAllDerivedDefinitions("GINodeEquiv"))
     NodeEquivs[Equiv->getValueAsDef("Node")] = Equiv;
 
   assert(ComplexPatternEquivs.empty());
   for (Record *Equiv : RK.getAllDerivedDefinitions("GIComplexPatternEquiv")) {
     Record *SelDAGEquiv = Equiv->getValueAsDef("SelDAGEquivalent");
     if (!SelDAGEquiv)
       continue;
     ComplexPatternEquivs[SelDAGEquiv] = Equiv;
  }
 
  assert(SDNodeXFormEquivs.empty());
  for (Record *Equiv : RK.getAllDerivedDefinitions("GISDNodeXFormEquiv")) {
    Record *SelDAGEquiv = Equiv->getValueAsDef("SelDAGEquivalent");
    if (!SelDAGEquiv)
      continue;
    SDNodeXFormEquivs[SelDAGEquiv] = Equiv;
  }
 }
 
 Record *GlobalISelEmitter::findNodeEquiv(Record *N) const {
   return NodeEquivs.lookup(N);
 }
 
 const CodeGenInstruction *
 GlobalISelEmitter::getEquivNode(Record &Equiv, const TreePatternNode *N) const {
   for (const auto &Predicate : N->getPredicateFns()) {
     if (!Equiv.isValueUnset("IfSignExtend") && Predicate.isLoad() &&
         Predicate.isSignExtLoad())
       return &Target.getInstruction(Equiv.getValueAsDef("IfSignExtend"));
     if (!Equiv.isValueUnset("IfZeroExtend") && Predicate.isLoad() &&
         Predicate.isZeroExtLoad())
       return &Target.getInstruction(Equiv.getValueAsDef("IfZeroExtend"));
   }
   return &Target.getInstruction(Equiv.getValueAsDef("I"));
 }
 
 GlobalISelEmitter::GlobalISelEmitter(RecordKeeper &RK)
     : RK(RK), CGP(RK), Target(CGP.getTargetInfo()),
       CGRegs(RK, Target.getHwModes()) {}
 
 //===- Emitter ------------------------------------------------------------===//
 
 Error
 GlobalISelEmitter::importRulePredicates(RuleMatcher &M,
                                         ArrayRef<Predicate> Predicates) {
   for (const Predicate &P : Predicates) {
     if (!P.Def)
       continue;
     declareSubtargetFeature(P.Def);
     M.addRequiredFeature(P.Def);
   }
 
   return Error::success();
 }
 
 Expected<InstructionMatcher &> GlobalISelEmitter::createAndImportSelDAGMatcher(
     RuleMatcher &Rule, InstructionMatcher &InsnMatcher,
     const TreePatternNode *Src, unsigned &TempOpIdx) const {
   Record *SrcGIEquivOrNull = nullptr;
   const CodeGenInstruction *SrcGIOrNull = nullptr;
 
   // Start with the defined operands (i.e., the results of the root operator).
   if (Src->getExtTypes().size() > 1)
     return failedImport("Src pattern has multiple results");
 
   if (Src->isLeaf()) {
     Init *SrcInit = Src->getLeafValue();
     if (isa<IntInit>(SrcInit)) {
       InsnMatcher.addPredicate<InstructionOpcodeMatcher>(
           &Target.getInstruction(RK.getDef("G_CONSTANT")));
     } else
       return failedImport(
           "Unable to deduce gMIR opcode to handle Src (which is a leaf)");
   } else {
     SrcGIEquivOrNull = findNodeEquiv(Src->getOperator());
     if (!SrcGIEquivOrNull)
       return failedImport("Pattern operator lacks an equivalent Instruction" +
                           explainOperator(Src->getOperator()));
     SrcGIOrNull = getEquivNode(*SrcGIEquivOrNull, Src);
 
     // The operators look good: match the opcode
     InsnMatcher.addPredicate<InstructionOpcodeMatcher>(SrcGIOrNull);
   }
 
   unsigned OpIdx = 0;
   for (const TypeSetByHwMode &VTy : Src->getExtTypes()) {
     // Results don't have a name unless they are the root node. The caller will
     // set the name if appropriate.
     OperandMatcher &OM = InsnMatcher.addOperand(OpIdx++, "", TempOpIdx);
     if (auto Error = OM.addTypeCheckPredicate(VTy, false /* OperandIsAPointer */))
       return failedImport(toString(std::move(Error)) +
                           " for result of Src pattern operator");
   }
 
   for (const auto &Predicate : Src->getPredicateFns()) {
     if (Predicate.isAlwaysTrue())
       continue;
 
     if (Predicate.isImmediatePattern()) {
       InsnMatcher.addPredicate<InstructionImmPredicateMatcher>(Predicate);
       continue;
     }
 
     // G_LOAD is used for both non-extending and any-extending loads. 
     if (Predicate.isLoad() && Predicate.isNonExtLoad()) {
       InsnMatcher.addPredicate<MemoryVsLLTSizePredicateMatcher>(
           0, MemoryVsLLTSizePredicateMatcher::EqualTo, 0);
       continue;
     }
     if (Predicate.isLoad() && Predicate.isAnyExtLoad()) {
       InsnMatcher.addPredicate<MemoryVsLLTSizePredicateMatcher>(
           0, MemoryVsLLTSizePredicateMatcher::LessThan, 0);
       continue;
     }
 
     // No check required. We already did it by swapping the opcode.
     if (!SrcGIEquivOrNull->isValueUnset("IfSignExtend") &&
         Predicate.isSignExtLoad())
       continue;
 
     // No check required. We already did it by swapping the opcode.
     if (!SrcGIEquivOrNull->isValueUnset("IfZeroExtend") &&
         Predicate.isZeroExtLoad())
       continue;
 
     // No check required. G_STORE by itself is a non-extending store.
     if (Predicate.isNonTruncStore())
       continue;
 
     if (Predicate.isLoad() || Predicate.isStore() || Predicate.isAtomic()) {
       if (Predicate.getMemoryVT() != nullptr) {
         Optional<LLTCodeGen> MemTyOrNone =
             MVTToLLT(getValueType(Predicate.getMemoryVT()));
 
         if (!MemTyOrNone)
           return failedImport("MemVT could not be converted to LLT");
 
         // MMO's work in bytes so we must take care of unusual types like i1
         // don't round down.
         unsigned MemSizeInBits =
             llvm::alignTo(MemTyOrNone->get().getSizeInBits(), 8);
 
         InsnMatcher.addPredicate<MemorySizePredicateMatcher>(
             0, MemSizeInBits / 8);
         continue;
       }
     }
 
     if (Predicate.isLoad() || Predicate.isStore()) {
       // No check required. A G_LOAD/G_STORE is an unindexed load.
       if (Predicate.isUnindexed())
         continue;
     }
 
     if (Predicate.isAtomic()) {
       if (Predicate.isAtomicOrderingMonotonic()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
             "Monotonic");
         continue;
       }
       if (Predicate.isAtomicOrderingAcquire()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>("Acquire");
         continue;
       }
       if (Predicate.isAtomicOrderingRelease()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>("Release");
         continue;
       }
       if (Predicate.isAtomicOrderingAcquireRelease()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
             "AcquireRelease");
         continue;
       }
       if (Predicate.isAtomicOrderingSequentiallyConsistent()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
             "SequentiallyConsistent");
         continue;
       }
 
       if (Predicate.isAtomicOrderingAcquireOrStronger()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
             "Acquire", AtomicOrderingMMOPredicateMatcher::AO_OrStronger);
         continue;
       }
       if (Predicate.isAtomicOrderingWeakerThanAcquire()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
             "Acquire", AtomicOrderingMMOPredicateMatcher::AO_WeakerThan);
         continue;
       }
 
       if (Predicate.isAtomicOrderingReleaseOrStronger()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
             "Release", AtomicOrderingMMOPredicateMatcher::AO_OrStronger);
         continue;
       }
       if (Predicate.isAtomicOrderingWeakerThanRelease()) {
         InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>(
             "Release", AtomicOrderingMMOPredicateMatcher::AO_WeakerThan);
         continue;
       }
     }
 
     return failedImport("Src pattern child has predicate (" +
                         explainPredicates(Src) + ")");
   }
   if (SrcGIEquivOrNull && SrcGIEquivOrNull->getValueAsBit("CheckMMOIsNonAtomic"))
     InsnMatcher.addPredicate<AtomicOrderingMMOPredicateMatcher>("NotAtomic");
 
   if (Src->isLeaf()) {
     Init *SrcInit = Src->getLeafValue();
     if (IntInit *SrcIntInit = dyn_cast<IntInit>(SrcInit)) {
       OperandMatcher &OM =
           InsnMatcher.addOperand(OpIdx++, Src->getName(), TempOpIdx);
       OM.addPredicate<LiteralIntOperandMatcher>(SrcIntInit->getValue());
     } else
       return failedImport(
           "Unable to deduce gMIR opcode to handle Src (which is a leaf)");
   } else {
     assert(SrcGIOrNull &&
            "Expected to have already found an equivalent Instruction");
     if (SrcGIOrNull->TheDef->getName() == "G_CONSTANT" ||
         SrcGIOrNull->TheDef->getName() == "G_FCONSTANT") {
       // imm/fpimm still have operands but we don't need to do anything with it
       // here since we don't support ImmLeaf predicates yet. However, we still
       // need to note the hidden operand to get GIM_CheckNumOperands correct.
       InsnMatcher.addOperand(OpIdx++, "", TempOpIdx);
       return InsnMatcher;
     }
 
     // Match the used operands (i.e. the children of the operator).
     for (unsigned i = 0, e = Src->getNumChildren(); i != e; ++i) {
       TreePatternNode *SrcChild = Src->getChild(i);
 
       // SelectionDAG allows pointers to be represented with iN since it doesn't
       // distinguish between pointers and integers but they are different types in GlobalISel.
       // Coerce integers to pointers to address space 0 if the context indicates a pointer.
       bool OperandIsAPointer = SrcGIOrNull->isOperandAPointer(i);
 
       // For G_INTRINSIC/G_INTRINSIC_W_SIDE_EFFECTS, the operand immediately
       // following the defs is an intrinsic ID.
       if ((SrcGIOrNull->TheDef->getName() == "G_INTRINSIC" ||
            SrcGIOrNull->TheDef->getName() == "G_INTRINSIC_W_SIDE_EFFECTS") &&
           i == 0) {
         if (const CodeGenIntrinsic *II = Src->getIntrinsicInfo(CGP)) {
           OperandMatcher &OM =
               InsnMatcher.addOperand(OpIdx++, SrcChild->getName(), TempOpIdx);
           OM.addPredicate<IntrinsicIDOperandMatcher>(II);
           continue;
         }
 
         return failedImport("Expected IntInit containing instrinsic ID)");
       }
 
       if (auto Error =
               importChildMatcher(Rule, InsnMatcher, SrcChild, OperandIsAPointer,
                                  OpIdx++, TempOpIdx))
         return std::move(Error);
     }
   }
 
   return InsnMatcher;
 }
 
 Error GlobalISelEmitter::importComplexPatternOperandMatcher(
     OperandMatcher &OM, Record *R, unsigned &TempOpIdx) const {
   const auto &ComplexPattern = ComplexPatternEquivs.find(R);
   if (ComplexPattern == ComplexPatternEquivs.end())
     return failedImport("SelectionDAG ComplexPattern (" + R->getName() +
                         ") not mapped to GlobalISel");
 
   OM.addPredicate<ComplexPatternOperandMatcher>(OM, *ComplexPattern->second);
   TempOpIdx++;
   return Error::success();
 }
 
 Error GlobalISelEmitter::importChildMatcher(RuleMatcher &Rule,
                                             InstructionMatcher &InsnMatcher,
                                             const TreePatternNode *SrcChild,
                                             bool OperandIsAPointer,
                                             unsigned OpIdx,
                                             unsigned &TempOpIdx) const {
   OperandMatcher &OM =
       InsnMatcher.addOperand(OpIdx, SrcChild->getName(), TempOpIdx);
   if (OM.isSameAsAnotherOperand())
     return Error::success();
 
   ArrayRef<TypeSetByHwMode> ChildTypes = SrcChild->getExtTypes();
   if (ChildTypes.size() != 1)
     return failedImport("Src pattern child has multiple results");
 
   // Check MBB's before the type check since they are not a known type.
   if (!SrcChild->isLeaf()) {
     if (SrcChild->getOperator()->isSubClassOf("SDNode")) {
       auto &ChildSDNI = CGP.getSDNodeInfo(SrcChild->getOperator());
       if (ChildSDNI.getSDClassName() == "BasicBlockSDNode") {
         OM.addPredicate<MBBOperandMatcher>();
         return Error::success();
       }
     }
   }
 
   if (auto Error =
           OM.addTypeCheckPredicate(ChildTypes.front(), OperandIsAPointer))
     return failedImport(toString(std::move(Error)) + " for Src operand (" +
                         to_string(*SrcChild) + ")");
 
   // Check for nested instructions.
   if (!SrcChild->isLeaf()) {
     if (SrcChild->getOperator()->isSubClassOf("ComplexPattern")) {
       // When a ComplexPattern is used as an operator, it should do the same
       // thing as when used as a leaf. However, the children of the operator
       // name the sub-operands that make up the complex operand and we must
       // prepare to reference them in the renderer too.
       unsigned RendererID = TempOpIdx;
       if (auto Error = importComplexPatternOperandMatcher(
               OM, SrcChild->getOperator(), TempOpIdx))
         return Error;
 
       for (unsigned i = 0, e = SrcChild->getNumChildren(); i != e; ++i) {
         auto *SubOperand = SrcChild->getChild(i);
         if (!SubOperand->getName().empty())
           Rule.defineComplexSubOperand(SubOperand->getName(),
                                        SrcChild->getOperator(), RendererID, i);
       }
 
       return Error::success();
     }
 
     auto MaybeInsnOperand = OM.addPredicate<InstructionOperandMatcher>(
         InsnMatcher.getRuleMatcher(), SrcChild->getName());
     if (!MaybeInsnOperand.hasValue()) {
       // This isn't strictly true. If the user were to provide exactly the same
       // matchers as the original operand then we could allow it. However, it's
       // simpler to not permit the redundant specification.
       return failedImport("Nested instruction cannot be the same as another operand");
     }
 
     // Map the node to a gMIR instruction.
     InstructionOperandMatcher &InsnOperand = **MaybeInsnOperand;
     auto InsnMatcherOrError = createAndImportSelDAGMatcher(
         Rule, InsnOperand.getInsnMatcher(), SrcChild, TempOpIdx);
     if (auto Error = InsnMatcherOrError.takeError())
       return Error;
 
     return Error::success();
   }
 
   if (SrcChild->hasAnyPredicate())
     return failedImport("Src pattern child has unsupported predicate");
 
   // Check for constant immediates.
   if (auto *ChildInt = dyn_cast<IntInit>(SrcChild->getLeafValue())) {
     OM.addPredicate<ConstantIntOperandMatcher>(ChildInt->getValue());
     return Error::success();
   }
 
   // Check for def's like register classes or ComplexPattern's.
   if (auto *ChildDefInit = dyn_cast<DefInit>(SrcChild->getLeafValue())) {
     auto *ChildRec = ChildDefInit->getDef();
 
     // Check for register classes.
     if (ChildRec->isSubClassOf("RegisterClass") ||
         ChildRec->isSubClassOf("RegisterOperand")) {
       OM.addPredicate<RegisterBankOperandMatcher>(
           Target.getRegisterClass(getInitValueAsRegClass(ChildDefInit)));
       return Error::success();
     }
 
     // Check for ValueType.
     if (ChildRec->isSubClassOf("ValueType")) {
       // We already added a type check as standard practice so this doesn't need
       // to do anything.
       return Error::success();
     }
 
     // Check for ComplexPattern's.
     if (ChildRec->isSubClassOf("ComplexPattern"))
       return importComplexPatternOperandMatcher(OM, ChildRec, TempOpIdx);
 
     if (ChildRec->isSubClassOf("ImmLeaf")) {
       return failedImport(
           "Src pattern child def is an unsupported tablegen class (ImmLeaf)");
     }
 
     return failedImport(
         "Src pattern child def is an unsupported tablegen class");
   }
 
   return failedImport("Src pattern child is an unsupported kind");
 }
 
 Expected<action_iterator> GlobalISelEmitter::importExplicitUseRenderer(
     action_iterator InsertPt, RuleMatcher &Rule, BuildMIAction &DstMIBuilder,
     TreePatternNode *DstChild) {
 
   const auto &SubOperand = Rule.getComplexSubOperand(DstChild->getName());
   if (SubOperand.hasValue()) {
     DstMIBuilder.addRenderer<RenderComplexPatternOperand>(
         *std::get<0>(*SubOperand), DstChild->getName(),
         std::get<1>(*SubOperand), std::get<2>(*SubOperand));
     return InsertPt;
   }
 
   if (!DstChild->isLeaf()) {
 
     if (DstChild->getOperator()->isSubClassOf("SDNodeXForm")) {
       auto Child = DstChild->getChild(0);
       auto I = SDNodeXFormEquivs.find(DstChild->getOperator());
       if (I != SDNodeXFormEquivs.end()) {
         DstMIBuilder.addRenderer<CustomRenderer>(*I->second, Child->getName());
         return InsertPt;
       }
       return failedImport("SDNodeXForm " + Child->getName() +
                           " has no custom renderer");
     }
 
     // We accept 'bb' here. It's an operator because BasicBlockSDNode isn't
     // inline, but in MI it's just another operand.
     if (DstChild->getOperator()->isSubClassOf("SDNode")) {
       auto &ChildSDNI = CGP.getSDNodeInfo(DstChild->getOperator());
       if (ChildSDNI.getSDClassName() == "BasicBlockSDNode") {
         DstMIBuilder.addRenderer<CopyRenderer>(DstChild->getName());
         return InsertPt;
       }
     }
 
     // Similarly, imm is an operator in TreePatternNode's view but must be
     // rendered as operands.
     // FIXME: The target should be able to choose sign-extended when appropriate
     //        (e.g. on Mips).
     if (DstChild->getOperator()->getName() == "imm") {
       DstMIBuilder.addRenderer<CopyConstantAsImmRenderer>(DstChild->getName());
       return InsertPt;
     } else if (DstChild->getOperator()->getName() == "fpimm") {
       DstMIBuilder.addRenderer<CopyFConstantAsFPImmRenderer>(
           DstChild->getName());
       return InsertPt;
     }
 
     if (DstChild->getOperator()->isSubClassOf("Instruction")) {
       ArrayRef<TypeSetByHwMode> ChildTypes = DstChild->getExtTypes();
       if (ChildTypes.size() != 1)
         return failedImport("Dst pattern child has multiple results");
 
       Optional<LLTCodeGen> OpTyOrNone = None;
       if (ChildTypes.front().isMachineValueType())
         OpTyOrNone =
             MVTToLLT(ChildTypes.front().getMachineValueType().SimpleTy);
       if (!OpTyOrNone)
         return failedImport("Dst operand has an unsupported type");
 
       unsigned TempRegID = Rule.allocateTempRegID();
       InsertPt = Rule.insertAction<MakeTempRegisterAction>(
           InsertPt, OpTyOrNone.getValue(), TempRegID);
       DstMIBuilder.addRenderer<TempRegRenderer>(TempRegID);
 
       auto InsertPtOrError = createAndImportSubInstructionRenderer(
           ++InsertPt, Rule, DstChild, TempRegID);
       if (auto Error = InsertPtOrError.takeError())
         return std::move(Error);
       return InsertPtOrError.get();
     }
 
     return failedImport("Dst pattern child isn't a leaf node or an MBB" + llvm::to_string(*DstChild));
   }
 
   // It could be a specific immediate in which case we should just check for
   // that immediate.
   if (const IntInit *ChildIntInit =
           dyn_cast<IntInit>(DstChild->getLeafValue())) {
     DstMIBuilder.addRenderer<ImmRenderer>(ChildIntInit->getValue());
     return InsertPt;
   }
 
   // Otherwise, we're looking for a bog-standard RegisterClass operand.
   if (auto *ChildDefInit = dyn_cast<DefInit>(DstChild->getLeafValue())) {
     auto *ChildRec = ChildDefInit->getDef();
 
     ArrayRef<TypeSetByHwMode> ChildTypes = DstChild->getExtTypes();
     if (ChildTypes.size() != 1)
       return failedImport("Dst pattern child has multiple results");
 
     Optional<LLTCodeGen> OpTyOrNone = None;
     if (ChildTypes.front().isMachineValueType())
       OpTyOrNone = MVTToLLT(ChildTypes.front().getMachineValueType().SimpleTy);
     if (!OpTyOrNone)
       return failedImport("Dst operand has an unsupported type");
 
     if (ChildRec->isSubClassOf("Register")) {
       DstMIBuilder.addRenderer<AddRegisterRenderer>(ChildRec);
       return InsertPt;
     }
 
     if (ChildRec->isSubClassOf("RegisterClass") ||
         ChildRec->isSubClassOf("RegisterOperand") ||
         ChildRec->isSubClassOf("ValueType")) {
       if (ChildRec->isSubClassOf("RegisterOperand") &&
           !ChildRec->isValueUnset("GIZeroRegister")) {
         DstMIBuilder.addRenderer<CopyOrAddZeroRegRenderer>(
             DstChild->getName(), ChildRec->getValueAsDef("GIZeroRegister"));
         return InsertPt;
       }
 
       DstMIBuilder.addRenderer<CopyRenderer>(DstChild->getName());
       return InsertPt;
     }
 
     if (ChildRec->isSubClassOf("ComplexPattern")) {
       const auto &ComplexPattern = ComplexPatternEquivs.find(ChildRec);
       if (ComplexPattern == ComplexPatternEquivs.end())
         return failedImport(
             "SelectionDAG ComplexPattern not mapped to GlobalISel");
 
       const OperandMatcher &OM = Rule.getOperandMatcher(DstChild->getName());
       DstMIBuilder.addRenderer<RenderComplexPatternOperand>(
           *ComplexPattern->second, DstChild->getName(),
           OM.getAllocatedTemporariesBaseID());
       return InsertPt;
     }
 
     return failedImport(
         "Dst pattern child def is an unsupported tablegen class");
   }
 
   return failedImport("Dst pattern child is an unsupported kind");
 }
 
 Expected<BuildMIAction &> GlobalISelEmitter::createAndImportInstructionRenderer(
     RuleMatcher &M, const TreePatternNode *Dst) {
   auto InsertPtOrError = createInstructionRenderer(M.actions_end(), M, Dst);
   if (auto Error = InsertPtOrError.takeError())
     return std::move(Error);
 
   action_iterator InsertPt = InsertPtOrError.get();
   BuildMIAction &DstMIBuilder = *static_cast<BuildMIAction *>(InsertPt->get());
 
   importExplicitDefRenderers(DstMIBuilder);
 
   if (auto Error = importExplicitUseRenderers(InsertPt, M, DstMIBuilder, Dst)
                        .takeError())
     return std::move(Error);
 
   return DstMIBuilder;
 }
 
 Expected<action_iterator>
 GlobalISelEmitter::createAndImportSubInstructionRenderer(
     const action_iterator InsertPt, RuleMatcher &M, const TreePatternNode *Dst,
     unsigned TempRegID) {
   auto InsertPtOrError = createInstructionRenderer(InsertPt, M, Dst);
 
   // TODO: Assert there's exactly one result.
 
   if (auto Error = InsertPtOrError.takeError())
     return std::move(Error);
 
   BuildMIAction &DstMIBuilder =
       *static_cast<BuildMIAction *>(InsertPtOrError.get()->get());
 
   // Assign the result to TempReg.
   DstMIBuilder.addRenderer<TempRegRenderer>(TempRegID, true);
 
   InsertPtOrError =
       importExplicitUseRenderers(InsertPtOrError.get(), M, DstMIBuilder, Dst);
   if (auto Error = InsertPtOrError.takeError())
     return std::move(Error);
 
   M.insertAction<ConstrainOperandsToDefinitionAction>(InsertPt,
                                                       DstMIBuilder.getInsnID());
   return InsertPtOrError.get();
 }
 
 Expected<action_iterator> GlobalISelEmitter::createInstructionRenderer(
     action_iterator InsertPt, RuleMatcher &M, const TreePatternNode *Dst) {
   Record *DstOp = Dst->getOperator();
   if (!DstOp->isSubClassOf("Instruction")) {
     if (DstOp->isSubClassOf("ValueType"))
       return failedImport(
           "Pattern operator isn't an instruction (it's a ValueType)");
     return failedImport("Pattern operator isn't an instruction");
   }
   CodeGenInstruction *DstI = &Target.getInstruction(DstOp);
 
   // COPY_TO_REGCLASS is just a copy with a ConstrainOperandToRegClassAction
   // attached. Similarly for EXTRACT_SUBREG except that's a subregister copy.
   if (DstI->TheDef->getName() == "COPY_TO_REGCLASS")
     DstI = &Target.getInstruction(RK.getDef("COPY"));
   else if (DstI->TheDef->getName() == "EXTRACT_SUBREG")
     DstI = &Target.getInstruction(RK.getDef("COPY"));
   else if (DstI->TheDef->getName() == "REG_SEQUENCE")
     return failedImport("Unable to emit REG_SEQUENCE");
 
   return M.insertAction<BuildMIAction>(InsertPt, M.allocateOutputInsnID(),
                                        DstI);
 }
 
 void GlobalISelEmitter::importExplicitDefRenderers(
     BuildMIAction &DstMIBuilder) {
   const CodeGenInstruction *DstI = DstMIBuilder.getCGI();
   for (unsigned I = 0; I < DstI->Operands.NumDefs; ++I) {
     const CGIOperandList::OperandInfo &DstIOperand = DstI->Operands[I];
     DstMIBuilder.addRenderer<CopyRenderer>(DstIOperand.Name);
   }
 }
 
 Expected<action_iterator> GlobalISelEmitter::importExplicitUseRenderers(
     action_iterator InsertPt, RuleMatcher &M, BuildMIAction &DstMIBuilder,
     const llvm::TreePatternNode *Dst) {
   const CodeGenInstruction *DstI = DstMIBuilder.getCGI();
   CodeGenInstruction *OrigDstI = &Target.getInstruction(Dst->getOperator());
 
   // EXTRACT_SUBREG needs to use a subregister COPY.
   if (OrigDstI->TheDef->getName() == "EXTRACT_SUBREG") {
     if (!Dst->getChild(0)->isLeaf())
       return failedImport("EXTRACT_SUBREG child #1 is not a leaf");
 
     if (DefInit *SubRegInit =
             dyn_cast<DefInit>(Dst->getChild(1)->getLeafValue())) {
       Record *RCDef = getInitValueAsRegClass(Dst->getChild(0)->getLeafValue());
       if (!RCDef)
         return failedImport("EXTRACT_SUBREG child #0 could not "
                             "be coerced to a register class");
 
       CodeGenRegisterClass *RC = CGRegs.getRegClass(RCDef);
       CodeGenSubRegIndex *SubIdx = CGRegs.getSubRegIdx(SubRegInit->getDef());
 
       const auto &SrcRCDstRCPair =
           RC->getMatchingSubClassWithSubRegs(CGRegs, SubIdx);
       if (SrcRCDstRCPair.hasValue()) {
         assert(SrcRCDstRCPair->second && "Couldn't find a matching subclass");
         if (SrcRCDstRCPair->first != RC)
           return failedImport("EXTRACT_SUBREG requires an additional COPY");
       }
 
       DstMIBuilder.addRenderer<CopySubRegRenderer>(Dst->getChild(0)->getName(),
                                                    SubIdx);
       return InsertPt;
     }
 
     return failedImport("EXTRACT_SUBREG child #1 is not a subreg index");
   }
 
   // Render the explicit uses.
   unsigned DstINumUses = OrigDstI->Operands.size() - OrigDstI->Operands.NumDefs;
   unsigned ExpectedDstINumUses = Dst->getNumChildren();
   if (OrigDstI->TheDef->getName() == "COPY_TO_REGCLASS") {
     DstINumUses--; // Ignore the class constraint.
     ExpectedDstINumUses--;
   }
 
   unsigned Child = 0;
   unsigned NumDefaultOps = 0;
   for (unsigned I = 0; I != DstINumUses; ++I) {
     const CGIOperandList::OperandInfo &DstIOperand =
         DstI->Operands[DstI->Operands.NumDefs + I];
 
     // If the operand has default values, introduce them now.
     // FIXME: Until we have a decent test case that dictates we should do
     // otherwise, we're going to assume that operands with default values cannot
     // be specified in the patterns. Therefore, adding them will not cause us to
     // end up with too many rendered operands.
     if (DstIOperand.Rec->isSubClassOf("OperandWithDefaultOps")) {
       DagInit *DefaultOps = DstIOperand.Rec->getValueAsDag("DefaultOps");
       if (auto Error = importDefaultOperandRenderers(DstMIBuilder, DefaultOps))
         return std::move(Error);
       ++NumDefaultOps;
       continue;
     }
 
     auto InsertPtOrError = importExplicitUseRenderer(InsertPt, M, DstMIBuilder,
                                                      Dst->getChild(Child));
     if (auto Error = InsertPtOrError.takeError())
       return std::move(Error);
     InsertPt = InsertPtOrError.get();
     ++Child;
   }
 
   if (NumDefaultOps + ExpectedDstINumUses != DstINumUses)
     return failedImport("Expected " + llvm::to_string(DstINumUses) +
                         " used operands but found " +
                         llvm::to_string(ExpectedDstINumUses) +
                         " explicit ones and " + llvm::to_string(NumDefaultOps) +
                         " default ones");
 
   return InsertPt;
 }
 
 Error GlobalISelEmitter::importDefaultOperandRenderers(
     BuildMIAction &DstMIBuilder, DagInit *DefaultOps) const {
   for (const auto *DefaultOp : DefaultOps->getArgs()) {
     // Look through ValueType operators.
     if (const DagInit *DefaultDagOp = dyn_cast<DagInit>(DefaultOp)) {
       if (const DefInit *DefaultDagOperator =
               dyn_cast<DefInit>(DefaultDagOp->getOperator())) {
         if (DefaultDagOperator->getDef()->isSubClassOf("ValueType"))
           DefaultOp = DefaultDagOp->getArg(0);
       }
     }
 
     if (const DefInit *DefaultDefOp = dyn_cast<DefInit>(DefaultOp)) {
       DstMIBuilder.addRenderer<AddRegisterRenderer>(DefaultDefOp->getDef());
       continue;
     }
 
     if (const IntInit *DefaultIntOp = dyn_cast<IntInit>(DefaultOp)) {
       DstMIBuilder.addRenderer<ImmRenderer>(DefaultIntOp->getValue());
       continue;
     }
 
     return failedImport("Could not add default op");
   }
 
   return Error::success();
 }
 
 Error GlobalISelEmitter::importImplicitDefRenderers(
     BuildMIAction &DstMIBuilder,
     const std::vector<Record *> &ImplicitDefs) const {
   if (!ImplicitDefs.empty())
     return failedImport("Pattern defines a physical register");
   return Error::success();
 }
 
 Expected<RuleMatcher> GlobalISelEmitter::runOnPattern(const PatternToMatch &P) {
   // Keep track of the matchers and actions to emit.
   int Score = P.getPatternComplexity(CGP);
   RuleMatcher M(P.getSrcRecord()->getLoc());
   RuleMatcherScores[M.getRuleID()] = Score;
   M.addAction<DebugCommentAction>(llvm::to_string(*P.getSrcPattern()) +
                                   "  =>  " +
                                   llvm::to_string(*P.getDstPattern()));
 
   if (auto Error = importRulePredicates(M, P.getPredicates()))
     return std::move(Error);
 
   // Next, analyze the pattern operators.
   TreePatternNode *Src = P.getSrcPattern();
   TreePatternNode *Dst = P.getDstPattern();
 
   // If the root of either pattern isn't a simple operator, ignore it.
   if (auto Err = isTrivialOperatorNode(Dst))
     return failedImport("Dst pattern root isn't a trivial operator (" +
                         toString(std::move(Err)) + ")");
   if (auto Err = isTrivialOperatorNode(Src))
     return failedImport("Src pattern root isn't a trivial operator (" +
                         toString(std::move(Err)) + ")");
 
   // The different predicates and matchers created during
   // addInstructionMatcher use the RuleMatcher M to set up their
   // instruction ID (InsnVarID) that are going to be used when
   // M is going to be emitted.
   // However, the code doing the emission still relies on the IDs
   // returned during that process by the RuleMatcher when issuing
   // the recordInsn opcodes.
   // Because of that:
   // 1. The order in which we created the predicates
   //    and such must be the same as the order in which we emit them,
   //    and
   // 2. We need to reset the generation of the IDs in M somewhere between
   //    addInstructionMatcher and emit
   //
   // FIXME: Long term, we don't want to have to rely on this implicit
   // naming being the same. One possible solution would be to have
   // explicit operator for operation capture and reference those.
   // The plus side is that it would expose opportunities to share
   // the capture accross rules. The downside is that it would
   // introduce a dependency between predicates (captures must happen
   // before their first use.)
   InstructionMatcher &InsnMatcherTemp = M.addInstructionMatcher(Src->getName());
   unsigned TempOpIdx = 0;
   auto InsnMatcherOrError =
       createAndImportSelDAGMatcher(M, InsnMatcherTemp, Src, TempOpIdx);
   if (auto Error = InsnMatcherOrError.takeError())
     return std::move(Error);
   InstructionMatcher &InsnMatcher = InsnMatcherOrError.get();
 
   if (Dst->isLeaf()) {
     Record *RCDef = getInitValueAsRegClass(Dst->getLeafValue());
 
     const CodeGenRegisterClass &RC = Target.getRegisterClass(RCDef);
     if (RCDef) {
       // We need to replace the def and all its uses with the specified
       // operand. However, we must also insert COPY's wherever needed.
       // For now, emit a copy and let the register allocator clean up.
       auto &DstI = Target.getInstruction(RK.getDef("COPY"));
       const auto &DstIOperand = DstI.Operands[0];
 
       OperandMatcher &OM0 = InsnMatcher.getOperand(0);
       OM0.setSymbolicName(DstIOperand.Name);
       M.defineOperand(OM0.getSymbolicName(), OM0);
       OM0.addPredicate<RegisterBankOperandMatcher>(RC);
 
       auto &DstMIBuilder =
           M.addAction<BuildMIAction>(M.allocateOutputInsnID(), &DstI);
       DstMIBuilder.addRenderer<CopyRenderer>(DstIOperand.Name);
       DstMIBuilder.addRenderer<CopyRenderer>(Dst->getName());
       M.addAction<ConstrainOperandToRegClassAction>(0, 0, RC);
 
       // We're done with this pattern!  It's eligible for GISel emission; return
       // it.
       ++NumPatternImported;
       return std::move(M);
     }
 
     return failedImport("Dst pattern root isn't a known leaf");
   }
 
   // Start with the defined operands (i.e., the results of the root operator).
   Record *DstOp = Dst->getOperator();
   if (!DstOp->isSubClassOf("Instruction"))
     return failedImport("Pattern operator isn't an instruction");
 
   auto &DstI = Target.getInstruction(DstOp);
   if (DstI.Operands.NumDefs != Src->getExtTypes().size())
     return failedImport("Src pattern results and dst MI defs are different (" +
                         to_string(Src->getExtTypes().size()) + " def(s) vs " +
                         to_string(DstI.Operands.NumDefs) + " def(s))");
 
   // The root of the match also has constraints on the register bank so that it
   // matches the result instruction.
   unsigned OpIdx = 0;
   for (const TypeSetByHwMode &VTy : Src->getExtTypes()) {
     (void)VTy;
 
     const auto &DstIOperand = DstI.Operands[OpIdx];
     Record *DstIOpRec = DstIOperand.Rec;
     if (DstI.TheDef->getName() == "COPY_TO_REGCLASS") {
       DstIOpRec = getInitValueAsRegClass(Dst->getChild(1)->getLeafValue());
 
       if (DstIOpRec == nullptr)
         return failedImport(
             "COPY_TO_REGCLASS operand #1 isn't a register class");
     } else if (DstI.TheDef->getName() == "EXTRACT_SUBREG") {
       if (!Dst->getChild(0)->isLeaf())
         return failedImport("EXTRACT_SUBREG operand #0 isn't a leaf");
 
       // We can assume that a subregister is in the same bank as it's super
       // register.
       DstIOpRec = getInitValueAsRegClass(Dst->getChild(0)->getLeafValue());
 
       if (DstIOpRec == nullptr)
         return failedImport(
             "EXTRACT_SUBREG operand #0 isn't a register class");
     } else if (DstIOpRec->isSubClassOf("RegisterOperand"))
       DstIOpRec = DstIOpRec->getValueAsDef("RegClass");
     else if (!DstIOpRec->isSubClassOf("RegisterClass"))
       return failedImport("Dst MI def isn't a register class" +
                           to_string(*Dst));
 
     OperandMatcher &OM = InsnMatcher.getOperand(OpIdx);
     OM.setSymbolicName(DstIOperand.Name);
     M.defineOperand(OM.getSymbolicName(), OM);
     OM.addPredicate<RegisterBankOperandMatcher>(
         Target.getRegisterClass(DstIOpRec));
     ++OpIdx;
   }
 
   auto DstMIBuilderOrError = createAndImportInstructionRenderer(M, Dst);
   if (auto Error = DstMIBuilderOrError.takeError())
     return std::move(Error);
   BuildMIAction &DstMIBuilder = DstMIBuilderOrError.get();
 
   // Render the implicit defs.
   // These are only added to the root of the result.
   if (auto Error = importImplicitDefRenderers(DstMIBuilder, P.getDstRegs()))
     return std::move(Error);
 
   DstMIBuilder.chooseInsnToMutate(M);
 
   // Constrain the registers to classes. This is normally derived from the
   // emitted instruction but a few instructions require special handling.
   if (DstI.TheDef->getName() == "COPY_TO_REGCLASS") {
     // COPY_TO_REGCLASS does not provide operand constraints itself but the
     // result is constrained to the class given by the second child.
     Record *DstIOpRec =
         getInitValueAsRegClass(Dst->getChild(1)->getLeafValue());
 
     if (DstIOpRec == nullptr)
       return failedImport("COPY_TO_REGCLASS operand #1 isn't a register class");
 
     M.addAction<ConstrainOperandToRegClassAction>(
         0, 0, Target.getRegisterClass(DstIOpRec));
 
     // We're done with this pattern!  It's eligible for GISel emission; return
     // it.
     ++NumPatternImported;
     return std::move(M);
   }
 
   if (DstI.TheDef->getName() == "EXTRACT_SUBREG") {
     // EXTRACT_SUBREG selects into a subregister COPY but unlike most
     // instructions, the result register class is controlled by the
     // subregisters of the operand. As a result, we must constrain the result
     // class rather than check that it's already the right one.
     if (!Dst->getChild(0)->isLeaf())
       return failedImport("EXTRACT_SUBREG child #1 is not a leaf");
 
     DefInit *SubRegInit = dyn_cast<DefInit>(Dst->getChild(1)->getLeafValue());
     if (!SubRegInit)
       return failedImport("EXTRACT_SUBREG child #1 is not a subreg index");
 
     // Constrain the result to the same register bank as the operand.
     Record *DstIOpRec =
         getInitValueAsRegClass(Dst->getChild(0)->getLeafValue());
 
     if (DstIOpRec == nullptr)
       return failedImport("EXTRACT_SUBREG operand #1 isn't a register class");
 
     CodeGenSubRegIndex *SubIdx = CGRegs.getSubRegIdx(SubRegInit->getDef());
     CodeGenRegisterClass *SrcRC = CGRegs.getRegClass(DstIOpRec);
 
     // It would be nice to leave this constraint implicit but we're required
     // to pick a register class so constrain the result to a register class
     // that can hold the correct MVT.
     //
     // FIXME: This may introduce an extra copy if the chosen class doesn't
     //        actually contain the subregisters.
     assert(Src->getExtTypes().size() == 1 &&
              "Expected Src of EXTRACT_SUBREG to have one result type");
 
     const auto &SrcRCDstRCPair =
         SrcRC->getMatchingSubClassWithSubRegs(CGRegs, SubIdx);
     assert(SrcRCDstRCPair->second && "Couldn't find a matching subclass");
     M.addAction<ConstrainOperandToRegClassAction>(0, 0, *SrcRCDstRCPair->second);
     M.addAction<ConstrainOperandToRegClassAction>(0, 1, *SrcRCDstRCPair->first);
 
     // We're done with this pattern!  It's eligible for GISel emission; return
     // it.
     ++NumPatternImported;
     return std::move(M);
   }
 
   M.addAction<ConstrainOperandsToDefinitionAction>(0);
 
   // We're done with this pattern!  It's eligible for GISel emission; return it.
   ++NumPatternImported;
   return std::move(M);
 }
 
 // Emit imm predicate table and an enum to reference them with.
 // The 'Predicate_' part of the name is redundant but eliminating it is more
 // trouble than it's worth.
 void GlobalISelEmitter::emitImmPredicates(
     raw_ostream &OS, StringRef TypeIdentifier, StringRef Type,
     std::function<bool(const Record *R)> Filter) {
   std::vector<const Record *> MatchedRecords;
   const auto &Defs = RK.getAllDerivedDefinitions("PatFrag");
   std::copy_if(Defs.begin(), Defs.end(), std::back_inserter(MatchedRecords),
                [&](Record *Record) {
                  return !Record->getValueAsString("ImmediateCode").empty() &&
                         Filter(Record);
                });
 
   if (!MatchedRecords.empty()) {
     OS << "// PatFrag predicates.\n"
        << "enum {\n";
     std::string EnumeratorSeparator =
         (" = GIPFP_" + TypeIdentifier + "_Invalid + 1,\n").str();
     for (const auto *Record : MatchedRecords) {
       OS << "  GIPFP_" << TypeIdentifier << "_Predicate_" << Record->getName()
          << EnumeratorSeparator;
       EnumeratorSeparator = ",\n";
     }
     OS << "};\n";
   }
 
   OS << "bool " << Target.getName() << "InstructionSelector::testImmPredicate_"
      << TypeIdentifier << "(unsigned PredicateID, " << Type
      << " Imm) const {\n";
   if (!MatchedRecords.empty())
     OS << "  switch (PredicateID) {\n";
   for (const auto *Record : MatchedRecords) {
     OS << "  case GIPFP_" << TypeIdentifier << "_Predicate_"
        << Record->getName() << ": {\n"
        << "    " << Record->getValueAsString("ImmediateCode") << "\n"
        << "    llvm_unreachable(\"ImmediateCode should have returned\");\n"
        << "    return false;\n"
        << "  }\n";
   }
   if (!MatchedRecords.empty())
     OS << "  }\n";
   OS << "  llvm_unreachable(\"Unknown predicate\");\n"
      << "  return false;\n"
      << "}\n";
 }
 
 template <class GroupT>
 std::vector<Matcher *> GlobalISelEmitter::optimizeRules(
     ArrayRef<Matcher *> Rules,
     std::vector<std::unique_ptr<Matcher>> &MatcherStorage) {
 
   std::vector<Matcher *> OptRules;
   std::unique_ptr<GroupT> CurrentGroup = make_unique<GroupT>();
   assert(CurrentGroup->empty() && "Newly created group isn't empty!");
   unsigned NumGroups = 0;
 
   auto ProcessCurrentGroup = [&]() {
     if (CurrentGroup->empty())
       // An empty group is good to be reused:
       return;
 
     // If the group isn't large enough to provide any benefit, move all the
     // added rules out of it and make sure to re-create the group to properly
     // re-initialize it:
     if (CurrentGroup->size() < 2)
       for (Matcher *M : CurrentGroup->matchers())
         OptRules.push_back(M);
     else {
       CurrentGroup->finalize();
       OptRules.push_back(CurrentGroup.get());
       MatcherStorage.emplace_back(std::move(CurrentGroup));
       ++NumGroups;
     }
     CurrentGroup = make_unique<GroupT>();
   };
   for (Matcher *Rule : Rules) {
     // Greedily add as many matchers as possible to the current group:
     if (CurrentGroup->addMatcher(*Rule))
       continue;
 
     ProcessCurrentGroup();
     assert(CurrentGroup->empty() && "A group wasn't properly re-initialized");
 
     // Try to add the pending matcher to a newly created empty group:
     if (!CurrentGroup->addMatcher(*Rule))
       // If we couldn't add the matcher to an empty group, that group type
       // doesn't support that kind of matchers at all, so just skip it:
       OptRules.push_back(Rule);
   }
   ProcessCurrentGroup();
 
   DEBUG(dbgs() << "NumGroups: " << NumGroups << "\n");
   assert(CurrentGroup->empty() && "The last group wasn't properly processed");
   return OptRules;
 }
 
 MatchTable
 GlobalISelEmitter::buildMatchTable(MutableArrayRef<RuleMatcher> Rules,
                                    bool Optimize, bool WithCoverage) {
   std::vector<Matcher *> InputRules;
   for (Matcher &Rule : Rules)
     InputRules.push_back(&Rule);
 
   if (!Optimize)
     return MatchTable::buildTable(InputRules, WithCoverage);
 
+  unsigned CurrentOrdering = 0;
+  StringMap<unsigned> OpcodeOrder;
+  for (RuleMatcher &Rule : Rules) {
+    const StringRef Opcode = Rule.getOpcode();
+    assert(!Opcode.empty() && "Didn't expect an undefined opcode");
+    if (OpcodeOrder.count(Opcode) == 0)
+      OpcodeOrder[Opcode] = CurrentOrdering++;
+  }
+
+  std::stable_sort(InputRules.begin(), InputRules.end(),
+                   [&OpcodeOrder](const Matcher *A, const Matcher *B) {
+                     auto *L = static_cast<const RuleMatcher *>(A);
+                     auto *R = static_cast<const RuleMatcher *>(B);
+                     return std::make_tuple(OpcodeOrder[L->getOpcode()],
+                                            L->getNumOperands()) <
+                            std::make_tuple(OpcodeOrder[R->getOpcode()],
+                                            R->getNumOperands());
+                   });
+
   for (Matcher *Rule : InputRules)
     Rule->optimize();
 
   std::vector<std::unique_ptr<Matcher>> MatcherStorage;
   std::vector<Matcher *> OptRules =
       optimizeRules<GroupMatcher>(InputRules, MatcherStorage);
 
   for (Matcher *Rule : OptRules)
     Rule->optimize();
 
   return MatchTable::buildTable(OptRules, WithCoverage);
 }
 
 void GlobalISelEmitter::run(raw_ostream &OS) {
   if (!UseCoverageFile.empty()) {
     RuleCoverage = CodeGenCoverage();
     auto RuleCoverageBufOrErr = MemoryBuffer::getFile(UseCoverageFile);
     if (!RuleCoverageBufOrErr) {
       PrintWarning(SMLoc(), "Missing rule coverage data");
       RuleCoverage = None;
     } else {
       if (!RuleCoverage->parse(*RuleCoverageBufOrErr.get(), Target.getName())) {
         PrintWarning(SMLoc(), "Ignoring invalid or missing rule coverage data");
         RuleCoverage = None;
       }
     }
   }
 
   // Track the run-time opcode values
   gatherOpcodeValues();
   // Track the run-time LLT ID values
   gatherTypeIDValues();
 
   // Track the GINodeEquiv definitions.
   gatherNodeEquivs();
 
   emitSourceFileHeader(("Global Instruction Selector for the " +
                        Target.getName() + " target").str(), OS);
   std::vector<RuleMatcher> Rules;
   // Look through the SelectionDAG patterns we found, possibly emitting some.
   for (const PatternToMatch &Pat : CGP.ptms()) {
     ++NumPatternTotal;
 
     auto MatcherOrErr = runOnPattern(Pat);
 
     // The pattern analysis can fail, indicating an unsupported pattern.
     // Report that if we've been asked to do so.
     if (auto Err = MatcherOrErr.takeError()) {
       if (WarnOnSkippedPatterns) {
         PrintWarning(Pat.getSrcRecord()->getLoc(),
                      "Skipped pattern: " + toString(std::move(Err)));
       } else {
         consumeError(std::move(Err));
       }
       ++NumPatternImportsSkipped;
       continue;
     }
 
     if (RuleCoverage) {
       if (RuleCoverage->isCovered(MatcherOrErr->getRuleID()))
         ++NumPatternsTested;
       else
         PrintWarning(Pat.getSrcRecord()->getLoc(),
                      "Pattern is not covered by a test");
     }
     Rules.push_back(std::move(MatcherOrErr.get()));
   }
 
   // Comparison function to order records by name.
   auto orderByName = [](const Record *A, const Record *B) {
     return A->getName() < B->getName();
   };
 
   std::vector<Record *> ComplexPredicates =
       RK.getAllDerivedDefinitions("GIComplexOperandMatcher");
   llvm::sort(ComplexPredicates.begin(), ComplexPredicates.end(), orderByName);
 
   std::vector<Record *> CustomRendererFns =
       RK.getAllDerivedDefinitions("GICustomOperandRenderer");
   llvm::sort(CustomRendererFns.begin(), CustomRendererFns.end(), orderByName);
 
   unsigned MaxTemporaries = 0;
   for (const auto &Rule : Rules)
     MaxTemporaries = std::max(MaxTemporaries, Rule.countRendererFns());
 
   OS << "#ifdef GET_GLOBALISEL_PREDICATE_BITSET\n"
      << "const unsigned MAX_SUBTARGET_PREDICATES = " << SubtargetFeatures.size()
      << ";\n"
      << "using PredicateBitset = "
         "llvm::PredicateBitsetImpl<MAX_SUBTARGET_PREDICATES>;\n"
      << "#endif // ifdef GET_GLOBALISEL_PREDICATE_BITSET\n\n";
 
   OS << "#ifdef GET_GLOBALISEL_TEMPORARIES_DECL\n"
      << "  mutable MatcherState State;\n"
      << "  typedef "
         "ComplexRendererFns("
      << Target.getName()
      << "InstructionSelector::*ComplexMatcherMemFn)(MachineOperand &) const;\n"
 
      << "  typedef void(" << Target.getName()
      << "InstructionSelector::*CustomRendererFn)(MachineInstrBuilder &, const "
         "MachineInstr&) "
         "const;\n"
      << "  const ISelInfoTy<PredicateBitset, ComplexMatcherMemFn, "
         "CustomRendererFn> "
         "ISelInfo;\n";
   OS << "  static " << Target.getName()
      << "InstructionSelector::ComplexMatcherMemFn ComplexPredicateFns[];\n"
      << "  static " << Target.getName()
      << "InstructionSelector::CustomRendererFn CustomRenderers[];\n"
      << "  bool testImmPredicate_I64(unsigned PredicateID, int64_t Imm) const "
         "override;\n"
      << "  bool testImmPredicate_APInt(unsigned PredicateID, const APInt &Imm) "
         "const override;\n"
      << "  bool testImmPredicate_APFloat(unsigned PredicateID, const APFloat "
         "&Imm) const override;\n"
      << "  const int64_t *getMatchTable() const override;\n"
      << "#endif // ifdef GET_GLOBALISEL_TEMPORARIES_DECL\n\n";
 
   OS << "#ifdef GET_GLOBALISEL_TEMPORARIES_INIT\n"
      << ", State(" << MaxTemporaries << "),\n"
      << "ISelInfo(TypeObjects, NumTypeObjects, FeatureBitsets"
      << ", ComplexPredicateFns, CustomRenderers)\n"
      << "#endif // ifdef GET_GLOBALISEL_TEMPORARIES_INIT\n\n";
 
   OS << "#ifdef GET_GLOBALISEL_IMPL\n";
   SubtargetFeatureInfo::emitSubtargetFeatureBitEnumeration(SubtargetFeatures,
                                                            OS);
 
   // Separate subtarget features by how often they must be recomputed.
   SubtargetFeatureInfoMap ModuleFeatures;
   std::copy_if(SubtargetFeatures.begin(), SubtargetFeatures.end(),
                std::inserter(ModuleFeatures, ModuleFeatures.end()),
                [](const SubtargetFeatureInfoMap::value_type &X) {
                  return !X.second.mustRecomputePerFunction();
                });
   SubtargetFeatureInfoMap FunctionFeatures;
   std::copy_if(SubtargetFeatures.begin(), SubtargetFeatures.end(),
                std::inserter(FunctionFeatures, FunctionFeatures.end()),
                [](const SubtargetFeatureInfoMap::value_type &X) {
                  return X.second.mustRecomputePerFunction();
                });
 
   SubtargetFeatureInfo::emitComputeAvailableFeatures(
       Target.getName(), "InstructionSelector", "computeAvailableModuleFeatures",
       ModuleFeatures, OS);
   SubtargetFeatureInfo::emitComputeAvailableFeatures(
       Target.getName(), "InstructionSelector",
       "computeAvailableFunctionFeatures", FunctionFeatures, OS,
       "const MachineFunction *MF");
 
   // Emit a table containing the LLT objects needed by the matcher and an enum
   // for the matcher to reference them with.
   std::vector<LLTCodeGen> TypeObjects;
   for (const auto &Ty : KnownTypes)
     TypeObjects.push_back(Ty);
   llvm::sort(TypeObjects.begin(), TypeObjects.end());
   OS << "// LLT Objects.\n"
      << "enum {\n";
   for (const auto &TypeObject : TypeObjects) {
     OS << "  ";
     TypeObject.emitCxxEnumValue(OS);
     OS << ",\n";
   }
   OS << "};\n";
   OS << "const static size_t NumTypeObjects = " << TypeObjects.size() << ";\n"
      << "const static LLT TypeObjects[] = {\n";
   for (const auto &TypeObject : TypeObjects) {
     OS << "  ";
     TypeObject.emitCxxConstructorCall(OS);
     OS << ",\n";
   }
   OS << "};\n\n";
 
   // Emit a table containing the PredicateBitsets objects needed by the matcher
   // and an enum for the matcher to reference them with.
   std::vector<std::vector<Record *>> FeatureBitsets;
   for (auto &Rule : Rules)
     FeatureBitsets.push_back(Rule.getRequiredFeatures());
   llvm::sort(
       FeatureBitsets.begin(), FeatureBitsets.end(),
       [&](const std::vector<Record *> &A, const std::vector<Record *> &B) {
         if (A.size() < B.size())
           return true;
         if (A.size() > B.size())
           return false;
         for (const auto &Pair : zip(A, B)) {
           if (std::get<0>(Pair)->getName() < std::get<1>(Pair)->getName())
             return true;
           if (std::get<0>(Pair)->getName() > std::get<1>(Pair)->getName())
             return false;
         }
         return false;
       });
   FeatureBitsets.erase(
       std::unique(FeatureBitsets.begin(), FeatureBitsets.end()),
       FeatureBitsets.end());
   OS << "// Feature bitsets.\n"
      << "enum {\n"
      << "  GIFBS_Invalid,\n";
   for (const auto &FeatureBitset : FeatureBitsets) {
     if (FeatureBitset.empty())
       continue;
     OS << "  " << getNameForFeatureBitset(FeatureBitset) << ",\n";
   }
   OS << "};\n"
      << "const static PredicateBitset FeatureBitsets[] {\n"
      << "  {}, // GIFBS_Invalid\n";
   for (const auto &FeatureBitset : FeatureBitsets) {
     if (FeatureBitset.empty())
       continue;
     OS << "  {";
     for (const auto &Feature : FeatureBitset) {
       const auto &I = SubtargetFeatures.find(Feature);
       assert(I != SubtargetFeatures.end() && "Didn't import predicate?");
       OS << I->second.getEnumBitName() << ", ";
     }
     OS << "},\n";
   }
   OS << "};\n\n";
 
   // Emit complex predicate table and an enum to reference them with.
   OS << "// ComplexPattern predicates.\n"
      << "enum {\n"
      << "  GICP_Invalid,\n";
   for (const auto &Record : ComplexPredicates)
     OS << "  GICP_" << Record->getName() << ",\n";
   OS << "};\n"
      << "// See constructor for table contents\n\n";
 
   emitImmPredicates(OS, "I64", "int64_t", [](const Record *R) {
     bool Unset;
     return !R->getValueAsBitOrUnset("IsAPFloat", Unset) &&
            !R->getValueAsBit("IsAPInt");
   });
   emitImmPredicates(OS, "APFloat", "const APFloat &", [](const Record *R) {
     bool Unset;
     return R->getValueAsBitOrUnset("IsAPFloat", Unset);
   });
   emitImmPredicates(OS, "APInt", "const APInt &", [](const Record *R) {
     return R->getValueAsBit("IsAPInt");
   });
   OS << "\n";
 
   OS << Target.getName() << "InstructionSelector::ComplexMatcherMemFn\n"
      << Target.getName() << "InstructionSelector::ComplexPredicateFns[] = {\n"
      << "  nullptr, // GICP_Invalid\n";
   for (const auto &Record : ComplexPredicates)
     OS << "  &" << Target.getName()
        << "InstructionSelector::" << Record->getValueAsString("MatcherFn")
        << ", // " << Record->getName() << "\n";
   OS << "};\n\n";
 
   OS << "// Custom renderers.\n"
      << "enum {\n"
      << "  GICR_Invalid,\n";
   for (const auto &Record : CustomRendererFns)
     OS << "  GICR_" << Record->getValueAsString("RendererFn") << ", \n";
   OS << "};\n";
 
   OS << Target.getName() << "InstructionSelector::CustomRendererFn\n"
      << Target.getName() << "InstructionSelector::CustomRenderers[] = {\n"
      << "  nullptr, // GICP_Invalid\n";
   for (const auto &Record : CustomRendererFns)
     OS << "  &" << Target.getName()
        << "InstructionSelector::" << Record->getValueAsString("RendererFn")
        << ", // " << Record->getName() << "\n";
   OS << "};\n\n";
 
   std::stable_sort(Rules.begin(), Rules.end(), [&](const RuleMatcher &A,
                                                    const RuleMatcher &B) {
     int ScoreA = RuleMatcherScores[A.getRuleID()];
     int ScoreB = RuleMatcherScores[B.getRuleID()];
     if (ScoreA > ScoreB)
       return true;
     if (ScoreB > ScoreA)
       return false;
     if (A.isHigherPriorityThan(B)) {
       assert(!B.isHigherPriorityThan(A) && "Cannot be more important "
                                            "and less important at "
                                            "the same time");
       return true;
     }
     return false;
   });
 
   OS << "bool " << Target.getName()
      << "InstructionSelector::selectImpl(MachineInstr &I, CodeGenCoverage "
         "&CoverageInfo) const {\n"
      << "  MachineFunction &MF = *I.getParent()->getParent();\n"
      << "  MachineRegisterInfo &MRI = MF.getRegInfo();\n"
      << "  // FIXME: This should be computed on a per-function basis rather "
         "than per-insn.\n"
      << "  AvailableFunctionFeatures = computeAvailableFunctionFeatures(&STI, "
         "&MF);\n"
      << "  const PredicateBitset AvailableFeatures = getAvailableFeatures();\n"
      << "  NewMIVector OutMIs;\n"
      << "  State.MIs.clear();\n"
      << "  State.MIs.push_back(&I);\n\n"
      << "  if (executeMatchTable(*this, OutMIs, State, ISelInfo"
      << ", getMatchTable(), TII, MRI, TRI, RBI, AvailableFeatures"
      << ", CoverageInfo)) {\n"
      << "    return true;\n"
      << "  }\n\n"
      << "  return false;\n"
      << "}\n\n";
 
   const MatchTable Table =
       buildMatchTable(Rules, OptimizeMatchTable, GenerateCoverage);
   OS << "const int64_t *" << Target.getName()
      << "InstructionSelector::getMatchTable() const {\n";
   Table.emitDeclaration(OS);
   OS << "  return ";
   Table.emitUse(OS);
   OS << ";\n}\n";
   OS << "#endif // ifdef GET_GLOBALISEL_IMPL\n";
 
   OS << "#ifdef GET_GLOBALISEL_PREDICATES_DECL\n"
      << "PredicateBitset AvailableModuleFeatures;\n"
      << "mutable PredicateBitset AvailableFunctionFeatures;\n"
      << "PredicateBitset getAvailableFeatures() const {\n"
      << "  return AvailableModuleFeatures | AvailableFunctionFeatures;\n"
      << "}\n"
      << "PredicateBitset\n"
      << "computeAvailableModuleFeatures(const " << Target.getName()
      << "Subtarget *Subtarget) const;\n"
      << "PredicateBitset\n"
      << "computeAvailableFunctionFeatures(const " << Target.getName()
      << "Subtarget *Subtarget,\n"
      << "                                 const MachineFunction *MF) const;\n"
      << "#endif // ifdef GET_GLOBALISEL_PREDICATES_DECL\n";
 
   OS << "#ifdef GET_GLOBALISEL_PREDICATES_INIT\n"
      << "AvailableModuleFeatures(computeAvailableModuleFeatures(&STI)),\n"
      << "AvailableFunctionFeatures()\n"
      << "#endif // ifdef GET_GLOBALISEL_PREDICATES_INIT\n";
 }
 
 void GlobalISelEmitter::declareSubtargetFeature(Record *Predicate) {
   if (SubtargetFeatures.count(Predicate) == 0)
     SubtargetFeatures.emplace(
         Predicate, SubtargetFeatureInfo(Predicate, SubtargetFeatures.size()));
 }
 
 void RuleMatcher::optimize() {
   for (auto &Item : InsnVariableIDs) {
     InstructionMatcher &InsnMatcher = *Item.first;
     for (auto &OM : InsnMatcher.operands()) {
       // Register Banks checks rarely fail, but often crash as targets usually
       // provide only partially defined RegisterBankInfo::getRegBankFromRegClass
       // method. Often the problem is hidden as non-optimized MatchTable checks
       // banks rather late, most notably after checking target / function /
       // module features and a few opcodes. That makes these checks a)
       // beneficial to delay until the very end (we don't want to perform a lot
       // of checks that all pass and then fail at the very end) b) not safe to
       // have as early checks.
       for (auto &OP : OM->predicates())
         if (isa<RegisterBankOperandMatcher>(OP) ||
             isa<ComplexPatternOperandMatcher>(OP))
           EpilogueMatchers.emplace_back(std::move(OP));
       OM->eraseNullPredicates();
     }
     InsnMatcher.optimize();
   }
   llvm::sort(
       EpilogueMatchers.begin(), EpilogueMatchers.end(),
       [](const std::unique_ptr<PredicateMatcher> &L,
          const std::unique_ptr<PredicateMatcher> &R) {
         return std::make_tuple(L->getKind(), L->getInsnVarID(), L->getOpIdx()) <
                std::make_tuple(R->getKind(), R->getInsnVarID(), R->getOpIdx());
       });
 }
 
 bool RuleMatcher::hasFirstCondition() const {
   if (insnmatchers_empty())
     return false;
   InstructionMatcher &Matcher = insnmatchers_front();
   if (!Matcher.predicates_empty())
     return true;
   for (auto &OM : Matcher.operands())
     for (auto &OP : OM->predicates())
       if (!isa<InstructionOperandMatcher>(OP))
         return true;
   return false;
 }
 
 const PredicateMatcher &RuleMatcher::getFirstCondition() const {
   assert(!insnmatchers_empty() &&
          "Trying to get a condition from an empty RuleMatcher");
 
   InstructionMatcher &Matcher = insnmatchers_front();
   if (!Matcher.predicates_empty())
     return **Matcher.predicates_begin();
   // If there is no more predicate on the instruction itself, look at its
   // operands.
   for (auto &OM : Matcher.operands())
     for (auto &OP : OM->predicates())
       if (!isa<InstructionOperandMatcher>(OP))
         return *OP;
 
   llvm_unreachable("Trying to get a condition from an InstructionMatcher with "
                    "no conditions");
 }
 
 std::unique_ptr<PredicateMatcher> RuleMatcher::popFirstCondition() {
   assert(!insnmatchers_empty() &&
          "Trying to pop a condition from an empty RuleMatcher");
 
   InstructionMatcher &Matcher = insnmatchers_front();
   if (!Matcher.predicates_empty())
     return Matcher.predicates_pop_front();
   // If there is no more predicate on the instruction itself, look at its
   // operands.
   for (auto &OM : Matcher.operands())
     for (auto &OP : OM->predicates())
       if (!isa<InstructionOperandMatcher>(OP)) {
         std::unique_ptr<PredicateMatcher> Result = std::move(OP);
         OM->eraseNullPredicates();
         return Result;
       }
 
   llvm_unreachable("Trying to pop a condition from an InstructionMatcher with "
                    "no conditions");
 }
 
 bool GroupMatcher::candidateConditionMatches(
     const PredicateMatcher &Predicate) const {
 
   if (empty()) {
     // Sharing predicates for nested instructions is not supported yet as we
     // currently don't hoist the GIM_RecordInsn's properly, therefore we can
     // only work on the original root instruction (InsnVarID == 0):
     if (Predicate.getInsnVarID() != 0)
       return false;
     // ... otherwise an empty group can handle any predicate with no specific
     // requirements:
     return true;
   }
 
   const Matcher &Representative = **Matchers.begin();
   const auto &RepresentativeCondition = Representative.getFirstCondition();
   // ... if not empty, the group can only accomodate matchers with the exact
   // same first condition:
   return Predicate.isIdentical(RepresentativeCondition);
 }
 
 bool GroupMatcher::addMatcher(Matcher &Candidate) {
   if (!Candidate.hasFirstCondition())
     return false;
 
   const PredicateMatcher &Predicate = Candidate.getFirstCondition();
   if (!candidateConditionMatches(Predicate))
     return false;
 
   Matchers.push_back(&Candidate);
   return true;
 }
 
 void GroupMatcher::finalize() {
   assert(Conditions.empty() && "Already finalized?");
   if (empty())
     return;
 
   Matcher &FirstRule = **Matchers.begin();
 
   Conditions.push_back(FirstRule.popFirstCondition());
   for (unsigned I = 1, E = Matchers.size(); I < E; ++I)
     Matchers[I]->popFirstCondition();
 }
 
 void GroupMatcher::emit(MatchTable &Table) {
   unsigned LabelID = ~0U;
   if (!Conditions.empty()) {
     LabelID = Table.allocateLabelID();
     Table << MatchTable::Opcode("GIM_Try", +1)
           << MatchTable::Comment("On fail goto")
           << MatchTable::JumpTarget(LabelID) << MatchTable::LineBreak;
   }
   for (auto &Condition : Conditions)
     Condition->emitPredicateOpcodes(
         Table, *static_cast<RuleMatcher *>(*Matchers.begin()));
 
   for (const auto &M : Matchers)
     M->emit(Table);
 
   // Exit the group
   if (!Conditions.empty())
     Table << MatchTable::Opcode("GIM_Reject", -1) << MatchTable::LineBreak
           << MatchTable::Label(LabelID);
 }
 
 unsigned OperandMatcher::getInsnVarID() const { return Insn.getInsnVarID(); }
 
 } // end anonymous namespace
 
 //===----------------------------------------------------------------------===//
 
 namespace llvm {
 void EmitGlobalISel(RecordKeeper &RK, raw_ostream &OS) {
   GlobalISelEmitter(RK).run(OS);
 }
 } // End llvm namespace

llvm-svn: 332999
2018-05-22 16:51:54 +00:00
Simon Dardis 13de555737 [FastISel] Permit instructions to be skipped for FastISel generation.
Some ISA's such as microMIPS32(R6) have instructions which are near identical
for code generation purposes, e.g. xor and xor16. These instructions take the
same value types for operands and return values, have the same
instruction predicates and map to the same ISD opcode. (These instructions do
differ by register classes.)

In such cases, the FastISel generator rejects the instruction definition.

This patch borrows the 'FastIselShouldIgnore' bit from rL129692 and enables
applying it to an instruction definition.

Reviewers: mcrosier

Differential Revision: https://reviews.llvm.org/D46953

llvm-svn: 332983
2018-05-22 14:36:58 +00:00
Roman Tereshin 19da667599 [GlobalISel][InstructionSelect] Removing redundant num operands and nested def operands checks, perf patch 2
This patch continues a series of patches that decrease time spent by
GlobalISel in its InstructionSelect pass by roughly 60% for -O0 builds
for large inputs as measured on sqlite3-amalgamation
(http://sqlite.org/download.html) targeting AArch64.

This commit specifically removes number of operands checks that are
redundant if the instruction's opcode already guarantees that number
of operands (or more), and also avoids any kind of checks on a def
operand of a nested instruction as everything about it was already
checked at its use.

The expected performance implication is about 3% off InstructionSelect
comparing to the baseline (before the series of patches)

This patch also contains a bit of NFC changes required for further
patches in the series.

Every commit planned shares the same Phabricator Review.

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 332945
2018-05-22 04:31:50 +00:00
Roman Tereshin f1aa348b31 Reapply r332907 "[GlobalISel] Improving InstructionSelect's performance by reducing MatchTable..."
Apparently the compile time problem was caused by the fact that not
all compilers / STL implementations can automatically convert
std::unique_ptr<Derived> to std::unique_ptr<Base>. Fixed (hopefully)
by making sure it's std::unique_ptr<Derived>&& (rvalue ref) to
std::unique_ptr<Base> conversion instead.

llvm-svn: 332917
2018-05-21 23:28:51 +00:00
Roman Tereshin 8bdf7be5bb Revert r332907 "[GlobalISel] Improving InstructionSelect's performance by reducing MatchTable..."
There is a compile time error I didn't see locally, investigating now.

llvm-svn: 332912
2018-05-21 22:21:24 +00:00
Roman Tereshin f0dc9fa934 [GlobalISel] Improving InstructionSelect's performance by reducing MatchTable, mostly NFC, perf patch 1
This patch starts a series of patches that decrease time spent by
GlobalISel in its InstructionSelect pass by roughly 60% for -O0 builds
for large inputs as measured on sqlite3-amalgamation
(http://sqlite.org/download.html) targeting AArch64.

The performance improvements are achieved solely by reducing the
number of matching GIM_* opcodes executed by the MatchTable's
interpreter during the selection by approx. a factor of 30, which also
brings contribution of this particular part of the selection process
to the overall runtime of InstructionSelect pass down from approx.
60-70% to 5-7%, thus making further improvements in this particular
direction not very profitable.

The improvements described above are expected for any target that
doesn't have many complex patterns. The targets that do should
strictly benefit from the changes, but by how much exactly is hard to
estimate beforehand. It's also likely that such target WILL benefit
from further improvements to MatchTable, most likely the ones that
bring it closer to a perfect decision tree.

This commit specifically is rather large mostly NFC commit that does
necessary preparation work and refactoring, there will be a following
series of small patches introducing a specific optimization each
shortly after.

This commit specifically is expected to cause a small compile time
regression (around 2.5% of InstructionSelect pass time), which should
be fixed by the next commit of the series.

Every commit planned shares the same Phabricator Review.

Reviewers: qcolombet, dsanders, bogner, aemerson, javed.absar

Reviewed By: qcolombet

Subscribers: rovka, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D44700

llvm-svn: 332907
2018-05-21 22:04:39 +00:00
Nicola Zaghen d34e60ca85 Rename DEBUG macro to LLVM_DEBUG.
The DEBUG() macro is very generic so it might clash with other projects.
The renaming was done as follows:
- git grep -l 'DEBUG' | xargs sed -i 's/\bDEBUG\s\?(/LLVM_DEBUG(/g'
- git diff -U0 master | ../clang/tools/clang-format/clang-format-diff.py -i -p1 -style LLVM
- Manual change to APInt
- Manually chage DOCS as regex doesn't match it.

In the transition period the DEBUG() macro is still present and aliased
to the LLVM_DEBUG() one.

Differential Revision: https://reviews.llvm.org/D43624

llvm-svn: 332240
2018-05-14 12:53:11 +00:00
Sam Clegg 16c16827bc [WebAssembly] Initial Disassembler.
This implements a new table-gen emitter to create tables for
a wasm disassembler, and a dissassembler to use them.

Comes with 2 tests, that tests a few instructions manually. Is also able to
disassemble large .wasm files with objdump reasonably.

Not working so well, to be addressed in followups:
- objdump appears to be passing an incorrect starting point.
- since the disassembler works an instruction at a time, and it is
  disassembling stack instruction, it has no idea of pseudo register assignments.
  These registers are required for the instruction printing code that follows.
  For now, all such registers appear in the output as $0.

Patch by Wouter van Oortmerssen

Differential Revision: https://reviews.llvm.org/D45848

llvm-svn: 332052
2018-05-10 22:16:44 +00:00
Daniel Sanders f84bc3793e [globalisel] Update GlobalISel emitter to match new representation of extending loads
Summary:
Previously, a extending load was represented at (G_*EXT (G_LOAD x)).
This had a few drawbacks:
* G_LOAD had to be legal for all sizes you could extend from, even if
  registers didn't naturally hold those sizes.
* All sizes you could extend from had to be allocatable just in case the
  extend went missing (e.g. by optimization).
* At minimum, G_*EXT and G_TRUNC had to be legal for these sizes. As we
  improve optimization of extends and truncates, this legality requirement
  would spread without considerable care w.r.t when certain combines were
  permitted.
* The SelectionDAG importer required some ugly and fragile pattern
  rewriting to translate patterns into this style.

This patch changes the representation to:
* (G_[SZ]EXTLOAD x)
* (G_LOAD x) any-extends when MMO.getSize() * 8 < ResultTy.getSizeInBits()
which resolves these issues by allowing targets to work entirely in their
native register sizes, and by having a more direct translation from
SelectionDAG patterns.

Each extending load can be lowered by the legalizer into separate extends
and loads, however a target that supports s1 will need the any-extending
load to extend to at least s8 since LLVM does not represent memory accesses
smaller than 8 bit. The legalizer can widenScalar G_LOAD into an
any-extending load but sign/zero-extending loads need help from something
else like a combiner pass. A follow-up patch that adds combiner helpers for
for this will follow.

The new representation requires that the MMO correctly reflect the memory
access so this has been corrected in a couple tests. I've also moved the
extending loads to their own tests since they are (mostly) separate opcodes
now. Additionally, the re-write appears to have invalidated two tests from
select-with-no-legality-check.mir since the matcher table no longer contains
loads that result in s1's and they aren't legal in AArch64 anymore.

Depends on D45540

Reviewers: ab, aditya_nandakumar, bogner, rtereshin, volkan, rovka, javed.absar

Reviewed By: rtereshin

Subscribers: javed.absar, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D45541

llvm-svn: 331601
2018-05-05 20:53:24 +00:00
Roman Tereshin beb393126b [GlobalISel][InstructionSelect] Making Coverage Info generation optional on per-match table basis
to make sure that Testgen always has access to coverage info even if
the match table used by the selector itself is stripped off that
information for performance reasons.

Reviewers: dsanders, aemerson

Reviewed By: dsanders

Subscribers: rovka, kristof.beyls, llvm-commits, dsanders

Differential Revision: https://reviews.llvm.org/D46098

llvm-svn: 331398
2018-05-02 20:15:11 +00:00
Roman Tereshin 2d6d3762a0 [GlobalISel][InstructionSelect] Refactoring buildMatchTable out, NFC
to share it between the Instruction Selector in optimized and
non-optimized modes both and the Testgen.

Reviewers: dsanders, aemerson

Reviewed By: dsanders

Subscribers: rovka, kristof.beyls, llvm-commits, dsanders

Differential Revision: https://reviews.llvm.org/D46097

llvm-svn: 331396
2018-05-02 20:08:14 +00:00
Roman Tereshin 2df4c22915 [GlobalISel][InstructionSelect] Refactoring out a getMatchTable virtual method + other small NFC's
The main goal is to share getMatchTable between the Instruction
Selector and the Testgen.

The commit also contains some NFC only loosely related to refactoring
out the getMatchTable, but strongly related to the initial Testgen
patch (see https://reviews.llvm.org/D43962)

Reviewers: dsanders, aemerson

Reviewed By: dsanders

Subscribers: rovka, kristof.beyls, llvm-commits, dsanders

Differential Revision: https://reviews.llvm.org/D46096

llvm-svn: 331395
2018-05-02 20:07:15 +00:00
Adrian Prantl 5f8f34e459 Remove \brief commands from doxygen comments.
We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.

Patch produced by

  for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done

Differential Revision: https://reviews.llvm.org/D46290

llvm-svn: 331272
2018-05-01 15:54:18 +00:00
Gabor Buella c8ded04e85 [X86] movdiri and movdir64b instructions
Reviewers: spatel, craig.topper, RKSimon

Reviewed By: craig.topper, RKSimon

Differential Revision: https://reviews.llvm.org/D45983

llvm-svn: 331248
2018-05-01 10:01:16 +00:00
Craig Topper 33dc01d105 [X86] Remove 'opaque ptr' from the intel syntax parser and printer.
Previously for instructions like fxsave we would print "opaque ptr" as part of the memory operand. Now we print nothing.

We also no longer accept "opaque ptr" in the parser. We still accept any size to be specified for these instructions, but we may want to consider only parsing when no explicit size is specified. This what gas does.

llvm-svn: 331243
2018-05-01 04:42:00 +00:00
Nico Weber 432a38838d IWYU for llvm-config.h in llvm, additions.
See r331124 for how I made a list of files missing the include.
I then ran this Python script:

    for f in open('filelist.txt'):
        f = f.strip()
        fl = open(f).readlines()

        found = False
        for i in xrange(len(fl)):
            p = '#include "llvm/'
            if not fl[i].startswith(p):
                continue
            if fl[i][len(p):] > 'Config':
                fl.insert(i, '#include "llvm/Config/llvm-config.h"\n')
                found = True
                break
        if not found:
            print 'not found', f
        else:
            open(f, 'w').write(''.join(fl))

and then looked through everything with `svn diff | diffstat -l | xargs -n 1000 gvim -p`
and tried to fix include ordering and whatnot.

No intended behavior change.

llvm-svn: 331184
2018-04-30 14:59:11 +00:00
Gabor Buella 61ff2864cb NFC - Typo fixes lib/VMCore -> lib/IR
llvm-svn: 331166
2018-04-30 10:18:11 +00:00
Andrea Di Biagio b31f918979 [Tablegen] SubtargetEmitter: move the logic that prints predicates for variant scheduling classes to helper functions. NFC
llvm-svn: 330968
2018-04-26 18:03:24 +00:00
Andrea Di Biagio 38fe227fd9 [Tablegen] Simplify code in CodeGenSchedule. NFCI
llvm-svn: 330935
2018-04-26 12:56:26 +00:00
Sander de Smalen a2fb1d18d2 [AsmMatcher] Extend PredicateMethod with optional DiagnosticPredicate
An optional, light-weight and backward-compatible mechanism to allow
specifying that a diagnostic _only_ applies to a partial mismatch (NearMiss),
rather than a full mismatch.

Patch [1/2] in a series to improve assembler diagnostics for SVE.
-  Patch [1/2]: https://reviews.llvm.org/D45879
-  Patch [2/2]: https://reviews.llvm.org/D45880

Reviewers: olista01, stoklund, craig.topper, mcrosier, rengolin, echristo, fhahn, SjoerdMeijer, evandro, javed.absar

Reviewed By: olista01

Differential Revision: https://reviews.llvm.org/D45879

llvm-svn: 330930
2018-04-26 09:24:45 +00:00
Craig Topper 773ead2908 [TableGen] Fix bad indentation in tablegen output file.
llvm-svn: 330801
2018-04-25 06:24:51 +00:00
Gabor Buella 1a2ce572bf [X86] Revert r330638 - accidental commit
llvm-svn: 330640
2018-04-23 20:05:51 +00:00
Gabor Buella 213a7cda1f [X86] movdiri and movdir64b instructions
Reviewers: craig.topper
llvm-svn: 330638
2018-04-23 20:00:59 +00:00
Craig Topper 9dcc50fcef [X86] Remove an unnecessary HANDLE_OPTIONAL line from the disassembler operand processing.
llvm-svn: 330534
2018-04-22 06:40:37 +00:00
Craig Topper e33ed7d667 [X86] Remove DATA32_PREFIX. Hack the printing for DATA16_PREFIX to print 'data32' in 16-bit mode. Hack the asm parser to convert 'data32' to 'data16' in 16-bit mode.
Improve the error messages to match GNU assembler.

This also allows us to remove the hack from the disassembler table building.

llvm-svn: 330531
2018-04-22 00:52:02 +00:00
Craig Topper 6496d51284 [X86] Remove non-existant instruction name from X86DisassemblerTables.cpp.
This instruction was removed a long time so we don't need to check for it here.

llvm-svn: 330363
2018-04-19 20:44:15 +00:00
Simon Pilgrim b04cd1b9f3 [llvm-exegesis] Fix PfmIssueCountersTable creation
This patch ensures that the pfm issue counter tables are the correct size, accounting for the invalid resource entry at the beginning of the resource tables.

It also fixes an issue with pfm failing to match event counters due to a trailing comma added to all the event names.

I've also added a counter comment to each entry as it helps locate problems with the tables.

Note: I don't have access to a SandyBridge test machine, which is the only model to make use of multiple event counters being mapped to a single resource. I don't know if pfm accepts a comma-seperated list or not, but that is what it was doing.

Differential Revision: https://reviews.llvm.org/D45787

llvm-svn: 330317
2018-04-19 10:59:49 +00:00
Simon Pilgrim 4eb879c686 X86FoldTableEntry - avoid unnecessary std::string creation. NFCI.
llvm-svn: 329860
2018-04-11 23:08:30 +00:00
Simon Pilgrim 781dece55e Don't repeatedly evaluate size() in the for loop. NFCI.
llvm-svn: 329853
2018-04-11 22:24:48 +00:00
Clement Courbet bfa20dddcb [MC][TableGen] Fix r329675.
Caught by bots with -Wmissing-braces.

llvm-svn: 329676
2018-04-10 08:43:46 +00:00
Clement Courbet b449379eae [MC][TableGen] Add optional libpfm counter names for ProcResUnits.
Summary:
Subtargets can define the libpfm counter names that can be used to
measure cycles and uops issued on ProcResUnits.
This allows making llvm-exegesis available on more targets.
Fixes PR36984.

Reviewers: gchatelet, RKSimon, andreadb, craig.topper

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D45360

llvm-svn: 329675
2018-04-10 08:16:37 +00:00
Sameer AbuAsal c1b0e66b58 [RISCV] Tablegen-driven Instruction Compression.
Summary:

    This patch implements a tablegen-driven Instruction Compression
    mechanism for generating RISCV compressed instructions
    (C Extension) from the expanded instruction form.

    This tablegen backend processes CompressPat declarations in a
    td file and generates all the compile-time and runtime checks
    required to validate the declarations, validate the input
    operands and generate correct instructions.

    The checks include validating register operands, immediate
    operands, fixed register operands and fixed immediate operands.

    Example:
      class CompressPat<dag input, dag output> {
        dag Input  = input;
        dag Output    = output;
        list<Predicate> Predicates = [];
      }

      let Predicates = [HasStdExtC] in {
      def : CompressPat<(ADD GPRNoX0:$rs1, GPRNoX0:$rs1, GPRNoX0:$rs2),
                        (C_ADD GPRNoX0:$rs1, GPRNoX0:$rs2)>;
      }

    The result is an auto-generated header file
    'RISCVGenCompressEmitter.inc' which exports two functions for
    compressing/uncompressing MCInst instructions, plus
    some helper functions:

      bool compressInst(MCInst& OutInst, const MCInst &MI,
                        const MCSubtargetInfo &STI,
                        MCContext &Context);

      bool uncompressInst(MCInst& OutInst, const MCInst &MI,
                          const MCRegisterInfo &MRI,
                          const MCSubtargetInfo &STI);

    The clients that include this auto-generated header file and
    invoke these functions can compress an instruction before emitting
    it, in the target-specific ASM or ELF streamer, or can uncompress
    an instruction before printing it, when the expanded instruction
    format aliases is favored.

    The following clients were added to implement compression\uncompression
    for RISCV:

    1) RISCVAsmParser::MatchAndEmitInstruction:
       Inserted a call to compressInst() to compresses instructions
       parsed by llvm-mc coming from an ASM input.
    2) RISCVAsmPrinter::EmitInstruction:
       Inserted a call to compressInst() to compress instructions that
       were lowered from Machine Instructions (MachineInstr).
    3) RVInstPrinter::printInst:
       Inserted a call to uncompressInst() to print the expanded
       version of the instruction instead of the compressed one (e.g,
       add s0, s0, a5 instead of c.add s0, a5) when -riscv-no-aliases
       is not passed.

This patch squashes D45119, D42780 and D41932. It was reviewed in  smaller patches by
asb, efriedma, apazos and mgrang.

Reviewers: asb, efriedma, apazos, llvm-commits, sabuasal

Reviewed By: sabuasal

Subscribers: mgorny, eraman, asb, rbar, johnrusso, simoncook, jordy.potman.lists, apazos, niosHD, kito-cheng, shiva0217, zzheng

Differential Revision: https://reviews.llvm.org/D45385

llvm-svn: 329455
2018-04-06 21:07:05 +00:00
Mandeep Singh Grang 1b0e2f2a20 [TableGen] Change std::sort to llvm::sort in response to r327219
Summary:
r327219 added wrappers to std::sort which randomly shuffle the container before sorting.
This will help in uncovering non-determinism caused due to undefined sorting
order of objects having the same key.

To make use of that infrastructure we need to invoke llvm::sort instead of std::sort.

Note: This patch is one of a series of patches to replace *all* std::sort to llvm::sort.
Refer the comments section in D44363 for a list of all the required patches.

Reviewers: stoklund, kparzysz, dsanders

Reviewed By: dsanders

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D45144

llvm-svn: 329451
2018-04-06 20:18:05 +00:00
Craig Topper 665f74414d [X86] Disassembler support for having an ADSIZE prefix affect instructions with 0xf2 and 0xf3 prefixes.
Needed to support umonitor from D45253.

llvm-svn: 329327
2018-04-05 18:20:14 +00:00
Andrea Di Biagio 9730bb8dd8 Fix the buildbots after r329304.
llvm-svn: 329306
2018-04-05 15:53:31 +00:00
Andrea Di Biagio c74ad502ce [MC][Tablegen] Allow models to describe the retire control unit for llvm-mca.
This patch adds the ability to describe properties of the hardware retire
control unit.

Tablegen class RetireControlUnit has been added for this purpose (see
TargetSchedule.td).

A RetireControlUnit specifies the size of the reorder buffer, as well as the
maximum number of opcodes that can be retired every cycle.

A zero (or negative) value for the reorder buffer size means: "the size is
unknown". If the size is unknown, then llvm-mca defaults it to the value of
field SchedMachineModel::MicroOpBufferSize.  A zero or negative number of
opcodes retired per cycle means: "there is no restriction on the number of
instructions that can be retired every cycle".

Models can optionally specify an instance of RetireControlUnit. There can only
be up-to one RetireControlUnit definition per scheduling model.

Information related to the RCU (RetireControlUnit) is stored in (two new fields
of) MCExtraProcessorInfo.  llvm-mca loads that information when it initializes
the DispatchUnit / RetireControlUnit (see Dispatch.h/Dispatch.cpp).

This patch fixes PR36661.

Differential Revision: https://reviews.llvm.org/D45259

llvm-svn: 329304
2018-04-05 15:41:41 +00:00
Andrea Di Biagio 8fd4be3ecc [MC] Fix spaces between values printed by EmitRegisterFileInfo.
llvm-svn: 329284
2018-04-05 13:59:52 +00:00
Simon Pilgrim 1d793b8ac5 [SchedModel] Complete models shouldn't match against itineraries when they don't use them (PR35639)
For schedule models that don't use itineraries, checkCompleteness still checks that an instruction has a matching itinerary instead of skipping and going straight to matching the InstRWs. That doesn't seem to match what happens in TargetSchedule.cpp

This patch causes problems for a number of models that had been incorrectly flagged as complete.

Differential Revision: https://reviews.llvm.org/D43235

llvm-svn: 329280
2018-04-05 13:11:36 +00:00
Andrea Di Biagio 378d75ac17 [Tablegen] Slightly refactor method SubtargetEmitter::EmitExtraProcessorInfo.
This patch moves most of the logic from EmitExtraProcessorInfo to a couple of
helper functions. No functional change intended.

llvm-svn: 329173
2018-04-04 11:53:13 +00:00
Andrea Di Biagio 9da4d6db33 [MC][Tablegen] Allow the definition of processor register files in the scheduling model for llvm-mca
This patch allows the description of register files in processor scheduling
models. This addresses PR36662.

A new tablegen class named 'RegisterFile' has been added to TargetSchedule.td.
Targets can optionally describe register files for their processors using that
class. In particular, class RegisterFile allows to specify:
 - The total number of physical registers.
 - Which target registers are accessible through the register file.
 - The cost of allocating a register at register renaming stage.

Example (from this patch - see file X86/X86ScheduleBtVer2.td)

  def FpuPRF : RegisterFile<72, [VR64, VR128, VR256], [1, 1, 2]>

Here, FpuPRF describes a register file for MMX/XMM/YMM registers. On Jaguar
(btver2), a YMM register definition consumes 2 physical registers, while MMX/XMM
register definitions only cost 1 physical register.

The syntax allows to specify an empty set of register classes.  An empty set of
register classes means: this register file models all the registers specified by
the Target.  For each register class, users can specify an optional register
cost. By default, register costs default to 1.  A value of 0 for the number of
physical registers means: "this register file has an unbounded number of
physical registers".

This patch is structured in two parts.

* Part 1 - MC/Tablegen *

A first part adds the tablegen definition of RegisterFile, and teaches the
SubtargetEmitter how to emit information related to register files.

Information about register files is accessible through an instance of
MCExtraProcessorInfo.
The idea behind this design is to logically partition the processor description
which is only used by external tools (like llvm-mca) from the processor
information used by the llvm machine schedulers.
I think that this design would make easier for targets to get rid of the extra
processor information if they don't want it.

* Part 2 - llvm-mca related *

The second part of this patch is related to changes to llvm-mca.

The main differences are:
 1) class RegisterFile now needs to take into account the "cost of a register"
when allocating physical registers at register renaming stage.
 2) Point 1. triggered a minor refactoring which lef to the removal of the
"maximum 32 register files" restriction.
 3) The BackendStatistics view has been updated so that we can print out extra
details related to each register file implemented by the processor.

The effect of point 3. is also visible in tests register-files-[1..5].s.

Differential Revision: https://reviews.llvm.org/D44980

llvm-svn: 329067
2018-04-03 13:36:24 +00:00
Craig Topper 9b6a65b9ef [X86] Reduce number of OpPrefix bits in TSFlags to 2. NFCI
TSFlag doesn't need to disambiguate NoPrfx from PS. So shift the encodings so PS is NoPrfx|0x4.

llvm-svn: 329049
2018-04-03 06:37:04 +00:00
Craig Topper 7d522b6eee [X86][TableGen] Add a missing error check to make sure EVEX instructions use one PS/PD/XS/XD prefixes.
llvm-svn: 329048
2018-04-03 06:37:01 +00:00
Craig Topper 95dade523a [TableGen] Use llvm::cast instead of static_cast so that the cast will be checked. NFC
llvm-svn: 329045
2018-04-03 05:10:12 +00:00
Nicolai Haehnle 398c0b6701 TableGen: Support Intrinsic values in SearchableTable
Summary:
We will use this in the AMDGPU backend in a subsequent patch
in the stack to lookup target-specific per-intrinsic information.

The generic CodeGenIntrinsic machinery is used to ensure that,
even though we don't calculate actual enum values here, we do
get the intrinsics in the right order for the binary search
index.

Change-Id: If61cd5587963a4c5a1cc53df1e59c5e4dec1f9dc

Reviewers: arsenm, rampitec, b-sumner

Subscribers: wdng, tpr, llvm-commits

Differential Revision: https://reviews.llvm.org/D44935

llvm-svn: 328937
2018-04-01 17:08:58 +00:00
Nicolai Haehnle 24e3a4d6e9 TableGen: More helpful error messages
Summary: Change-Id: I3c23f6f6597912423762780cd8c5315870412bbe

Reviewers: arsenm, rampitec, b-sumner

Subscribers: wdng, llvm-commits

Differential Revision: https://reviews.llvm.org/D44936

Change-Id: Ie62614a3e2d7774f46e4034478b28f57100a2c92
llvm-svn: 328936
2018-04-01 17:08:49 +00:00
Fangrui Song 956ee79795 Fix a bunch of typoes. NFC
llvm-svn: 328907
2018-03-30 22:22:31 +00:00
David Blaikie 10f71304b7 Add missing dependency (headers are included from MC, so a link dependency could exist easily enough)
llvm-svn: 328743
2018-03-29 00:29:43 +00:00
Simon Pilgrim d044f9c9a7 [SchedModel] Remove instregex entries that don't match any instructions
This patch throws a fatal error if an instregex entry doesn't actually match any instructions. This is part of the work to reduce the compile time impact of increased instregex usage (PR35955), although the x86 models seem to be relatively clean.

All the cases I encountered have now been fixed in trunk and this will ensure they don't get reintroduced.

Differential Revision: https://reviews.llvm.org/D44687

llvm-svn: 328459
2018-03-25 19:20:08 +00:00
Craig Topper 2ed540779d [SchedModel] Remove an unneeded temporary vector.
llvm-svn: 328442
2018-03-24 22:58:03 +00:00
Craig Topper 18cfa2c710 [SchedModel] Use std::move in a couple places to reduce copying
llvm-svn: 328441
2018-03-24 22:58:02 +00:00
Craig Topper 59d13776f7 [SchedModel] Use std::move to replace a vector instead of vector::swap
We don't really care about the old vector value so we don't care to swap it.

llvm-svn: 328440
2018-03-24 22:58:00 +00:00
Craig Topper 9f3293a9d4 [SchedModel] Remove std::vectors that were created with 1 element and then passed to an ArrayRef parameter.
ArrayRef can capture a single element. We don't need a vector for that.

llvm-svn: 328438
2018-03-24 21:57:35 +00:00
Simon Pilgrim 494d07516e [SchedModel] Record::getName() returns StringRef - avoid std::string creation. NFCI.
llvm-svn: 328437
2018-03-24 21:22:32 +00:00
Simon Pilgrim 34d512ec1e [SchedModel] Avoid std::string creation for instregex patterns that don't contain regex metas. NFCI.
llvm-svn: 328436
2018-03-24 21:04:20 +00:00
Craig Topper 097b47a0fc [X86] Add a new disassembler opcode map for 3DNow. Stop treating 3DNow as an attribute.
This reduces the size of llvm-mc by at least 150k since we no longer have to multiply the attribute across 7 tables.

llvm-svn: 328416
2018-03-24 07:48:54 +00:00
Craig Topper c0e1880db9 [X86] Use unique_ptr to simplify memory management. NFC
llvm-svn: 328413
2018-03-24 07:15:47 +00:00
Craig Topper a51ec94d68 [X86] Use X86_INSTR_MRM_MAPPING macro instead of listing all MRM_C0-MRM_FF format encodings. NFC
llvm-svn: 328412
2018-03-24 07:15:46 +00:00
Craig Topper 0bafe239fc [X86] Remove an unnecessary switch around two other switches. NFC
The outer switch only had one valid block so didn't provide any value.

llvm-svn: 328411
2018-03-24 07:15:45 +00:00
Craig Topper e865641aea [X86] Merge the Has3DNow0F0FOpcode TSFlag into the OpMap encoding. NFC
The 3DNow instructions are encoded a little weird, but we can still represent it as an opcode map.

llvm-svn: 328410
2018-03-24 06:04:12 +00:00
David Blaikie b48ba4e6bc Fix layering of CodeGen/TargetOpcodes.def by moving it to Support
It's also used by utils/TableGen so needs to reside somewhere common to
TableGen and CodeGen.

llvm-svn: 328396
2018-03-23 23:58:27 +00:00
David Blaikie 13e77db2df Fix layering of MachineValueType.h by moving it from CodeGen to Support
This is used by llvm tblgen as well as by LLVM Targets, so the only
common place is Support for now. (maybe we need another target for these
sorts of things - but for now I'm at least making them correct & we can
make them better if/when people have strong feelings)

llvm-svn: 328395
2018-03-23 23:58:25 +00:00
David Blaikie ab7f17f4ec Fix layering by moving X86DisassemblerDecoderCommon to Support
This is used from llvm tblgen and the X86Disassembler - the only common
library (apart from TableGen, which probably doesn't make sense to have
as a dependency from a release tool (rather than a use-while-building-llvm
tool) of LLVM)

llvm-svn: 328393
2018-03-23 23:58:20 +00:00
Craig Topper 37eeb32046 [TableGen] Don't capture returned std::vectors by const reference.
The full vector is being returned not a reference. So the reference was just a to a temporary.

llvm-svn: 328275
2018-03-23 00:02:45 +00:00
Craig Topper 40d3b32e12 [X86] Rename VROUNDYPS* and VROUNDYPD* instructions to VROUNDPSY* and VROUNDPDY*. Fix itinerary mistake on all memory forms of VROUNDPD
This makes the Y position consistent with other instructions.

This should have been NFC, but while refactoring the multiclass I noticed that VROUNDPD memory forms were using the register itinerary.

llvm-svn: 328254
2018-03-22 21:55:20 +00:00
Craig Topper 195aaaf541 [TableGen] Use empty emplace_back to add defaulted constructed objects to vectors to avoid using resize(size()+1). NFC
llvm-svn: 328184
2018-03-22 06:15:10 +00:00
Craig Topper 281a19cf7b [TableGen] Add a non-default constructor to CodeGenSchedClass and use it via emplace_back to create new SchedClasses instead of using resize(size+1)
llvm-svn: 328183
2018-03-22 06:15:08 +00:00
Craig Topper 989d94ddde [TableGen] Hoist the code for copying InstRWs from an old scheduling class to a new one out of the loop that assigns instructions to the new class. NFCI
We already know all the of instructions we're processing in the instruction loop belong to no class or all to the same class. So we only have to worry about remapping one class. So hoist it all out and remove the SmallPtrSet that tracked which class we'd already remapped.

I had to introduce new instruction loop inside this code to print an error message, but that only occurs on the error path.

llvm-svn: 328142
2018-03-21 19:52:13 +00:00
Craig Topper f2ab05c432 [TableGen] Remove unnecessary map lookup and shadowing of a variable. NFCI
We already have an OldSCIdx variable in the outer loop here. And we already did the map lookup in the loop that populated ClassInstrs. And the outer OldSCIdx got it from ClassInstrs.

llvm-svn: 328139
2018-03-21 19:30:33 +00:00
Craig Topper 9fbbe5d9ca [TableGen] Use range-based for loops. NFC
llvm-svn: 328138
2018-03-21 19:30:31 +00:00
Craig Topper 06d783765a [TableGen] Use count_if instead of a manual loop. NFC
llvm-svn: 328137
2018-03-21 19:30:30 +00:00
Simon Pilgrim 949437e882 [SchedModel] Use CodeGenSchedClass::getSchedClassIdx helper directly. NFCI.
llvm-svn: 328128
2018-03-21 18:09:34 +00:00
Simon Pilgrim 4cca3b190b [SchedModel] Use CodeGenSchedClass::isKeyEqual instead of duplicating code. NFCI.
llvm-svn: 328126
2018-03-21 17:57:21 +00:00
Craig Topper e26118473e [TableGen] Remove a defaulted function argument that is never called with another value. NFC
llvm-svn: 328075
2018-03-21 05:13:04 +00:00
Craig Topper 6f2cc9b107 [TableGen] Move a function from llvm namespace and make it a static function. NFC
It's only called from one place and is defined just above that use.

llvm-svn: 328074
2018-03-21 05:13:01 +00:00
Craig Topper f19eacfe0b [TableGen] Use SmallMapVector to simplify some code that was trying to keep a vector unique
Summary:
This code previously had a SmallVector of std::pairs containing an unsigned and another SmallVector. The outer vector was using the unsigned effectively as a key to decide which SmallVector to add into. So each time something new needed to be added the out vector needed to be scanned. If it wasn't found a new entry needed to be added to be added. This sounds very much like a map, but the next loop iterates over the outer vector to get a deterministic order.

We can simplify this code greatly if use SmallMapVector instead. This uses more stack space since we now have a vector and a map, but the searching and creating new entries all happens behind the scenes. It should also make the search more efficient though usually there are only a few entries so that doesn't matter much.

We could probably get determinism by just using std::map which would iterate over the unsigned key, but that would generate different output from what we get with the current implementation.

Reviewers: RKSimon, dblaikie

Reviewed By: dblaikie

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D44711

llvm-svn: 328070
2018-03-21 02:48:34 +00:00
Simon Pilgrim 75cc2f9e1f [SchedModel] Simplify InstRegexOp::apply. NFCI.
As discussed on D44687, there was no need for 2 separate for loops for collecting the Regex and then matching against instructions.

llvm-svn: 328052
2018-03-20 22:20:28 +00:00
Craig Topper f611425916 [TableGen] Use range based for loop. NFC
llvm-svn: 328034
2018-03-20 20:24:16 +00:00
Craig Topper 1f57456c78 [TableGen] Use vector::append instead of looping and calling push_back. NFC
Both vectors contain unsigned so we can just use append to do the copying. Not only is this shorter, but it should be able to predict the final size and only grow the vector once if needed.

llvm-svn: 328033
2018-03-20 20:24:14 +00:00
Craig Topper 1970e95575 [TableGen] Use llvm::transform to simplify some loops. NFCI
llvm-svn: 328032
2018-03-20 20:24:12 +00:00
Craig Topper b5ed275025 [TableGen] Pass result of std::unique to vector::erase instead of calculating a size and calling resize.
llvm-svn: 328031
2018-03-20 20:24:10 +00:00
Krzysztof Parzyszek eb0c510ecd [X86] Add phony registers for high halves of regs with low halves
Registers E[A-D]X, E[SD]I, E[BS]P, and EIP have 16-bit subregisters
that cover the low halves of these registers. This change adds artificial
subregisters for the high halves in order to differentiate (in terms of
register units) between the 32- and the low 16-bit registers.

This patch contains parts that aim to preserve the calculated register
pressure. This is in order to preserve the current codegen (minimize the
impact of this patch). The approach of having artificial subregisters
could be used to fix PR23423, but the pressure calculation would need
to be changed.

Differential Revision: https://reviews.llvm.org/D43353

llvm-svn: 328016
2018-03-20 18:46:55 +00:00
Craig Topper e1d6a4df1c [TableGen] When trying to reuse a scheduler class for instructions from an InstRW, make sure we haven't already seen another InstRW containing this instruction on this CPU.
This is similar to the check later when we remap some of the instructions from one class to a new one. But if we reuse the class we don't get to do that check.

So many CPUs have violations of this check that I had to add a flag to the SchedMachineModel to allow it to be disabled. Hopefully we can get those cleaned up quickly and remove this flag.

A lot of the violations are due to overlapping regular expressions, but that's not the only kind of issue it found.

llvm-svn: 327808
2018-03-18 19:56:15 +00:00
Craig Topper 93dd77d2dc [TableGen] Remove unnecessary uses of make_range.
llvm-svn: 327785
2018-03-18 08:38:03 +00:00
Craig Topper 7f31e735c9 [TableGen] Move some variables into for loop declaration. NFC
They aren't needed after the loop.

llvm-svn: 327784
2018-03-18 08:38:02 +00:00
Sylvestre Ledru 543f15b028 Fix some user facing typos
llvm-svn: 327776
2018-03-17 17:30:08 +00:00
Oren Ben Simhon fdd72fd522 [X86] Added support for nocf_check attribute for indirect Branch Tracking
X86 Supports Indirect Branch Tracking (IBT) as part of Control-Flow Enforcement Technology (CET).
IBT instruments ENDBR instructions used to specify valid targets of indirect call / jmp.
The `nocf_check` attribute has two roles in the context of X86 IBT technology:
	1. Appertains to a function - do not add ENDBR instruction at the beginning of the function.
	2. Appertains to a function pointer - do not track the target function of this pointer by adding nocf_check prefix to the indirect-call instruction.

This patch implements `nocf_check` context for Indirect Branch Tracking.
It also auto generates `nocf_check` prefixes before indirect branchs to jump tables that are guarded by range checks.

Differential Revision: https://reviews.llvm.org/D41879

llvm-svn: 327767
2018-03-17 13:29:46 +00:00
Craig Topper bca036b669 [X86] Add all of the MRM_C0-MRM_FF forms to the switch in RecognizableInstr::emitInstructionSpecifier. NFC
Remove the special casing for MRM_F8 by using HANDLE_OPTIONAL.

This should be NFC as the forms that were missing aren't used by any instructions today. They exist in the enum so that we didn't have to put them in one at a time when instructions are added. But looks like we failed here.

llvm-svn: 327298
2018-03-12 17:24:50 +00:00
Andrea Di Biagio 30e94029d8 [MCSchedule] Always generate processor resource names.
With this patch, the tablegen 'SubtargetEmitter' always generates processor
resource names. 
The impact of this patch on the code size of other llvm tools is small.  I have
observed an average increase of 0.03% in code size when doing a release build of
LLVM (on windows, using MSVC) with all the default backends.

This change is done in preparation for the upcoming llvm-mca patch.

llvm-svn: 326993
2018-03-08 10:38:45 +00:00
Heejin Ahn 0de587296e [WebAssembly] Add except_ref as a first-class type
Summary: Add except_ref as a first-class type, according to the [[https://github.com/WebAssembly/exception-handling/blob/master/proposals/Level-1.md | Level 1 exception handling proposal ]].

Reviewers: dschuff

Subscribers: jfb, sbc100, llvm-commits

Differential Revision: https://reviews.llvm.org/D43706

llvm-svn: 326985
2018-03-08 04:05:37 +00:00
Nicolai Haehnle 0409b28af1 TableGen: Use DefInit::getDef() instead of the type's getRecord()
The former simply makes more sense: we want to access the data here in
the backend, not information about the type.

More importantly, removing users of RecordRecTy::getRecord() allows us
more freedom to refactor the frontend.

Change-Id: Iee8905fd22cdb9b11c42ca03246c03d8fe4dd77f
llvm-svn: 326699
2018-03-05 14:01:30 +00:00
Benjamin Kramer b941ababce Shrink various scheduling tables by using narrower types.
16 bits ought to be enough for everyone. This shrinks clang by ~1MB.

llvm-svn: 325941
2018-02-23 19:32:56 +00:00
Geoff Berry f8bf2ec0a8 [MachineOperand][Target] MachineOperand::isRenamable semantics changes
Summary:
Add a target option AllowRegisterRenaming that is used to opt in to
post-register-allocation renaming of registers.  This is set to 0 by
default, which causes the hasExtraSrcRegAllocReq/hasExtraDstRegAllocReq
fields of all opcodes to be set to 1, causing
MachineOperand::isRenamable to always return false.

Set the AllowRegisterRenaming flag to 1 for all in-tree targets that
have lit tests that were effected by enabling COPY forwarding in
MachineCopyPropagation (AArch64, AMDGPU, ARM, Hexagon, Mips, PowerPC,
RISCV, Sparc, SystemZ and X86).

Add some more comments describing the semantics of the
MachineOperand::isRenamable function and how it is set and maintained.

Change isRenamable to check the operand's opcode
hasExtraSrcRegAllocReq/hasExtraDstRegAllocReq bit directly instead of
relying on it being consistently reflected in the IsRenamable bit
setting.

Clear the IsRenamable bit when changing an operand's register value.

Remove target code that was clearing the IsRenamable bit when changing
registers/opcodes now that this is done conservatively by default.

Change setting of hasExtraSrcRegAllocReq in AMDGPU target to be done in
one place covering all opcodes that have constant pipe read limit
restrictions.

Reviewers: qcolombet, MatzeB

Subscribers: aemerson, arsenm, jyknight, mcrosier, sdardis, nhaehnle, javed.absar, tpr, arichardson, kristof.beyls, kbarton, fedor.sergeev, asb, rbar, johnrusso, simoncook, jordy.potman.lists, apazos, sabuasal, niosHD, escha, nemanjai, llvm-commits

Differential Revision: https://reviews.llvm.org/D43042

llvm-svn: 325931
2018-02-23 18:25:08 +00:00
Simon Pilgrim e4d40f9b7d Fix signed/unsigned comparison warning in AsmGenMatcher generated code. NFCI.
llvm-svn: 325428
2018-02-17 12:29:47 +00:00
Aditya Nandakumar b63e763847 [GISel]: Make GlobalISelEmitter rule prioritization compatible with selectionDAG
This patch changes GlobalISelEmitter to rank patterns similar to how the
DAG does it (ie it computes a score for a pattern and adds the added
complexity to it).
This is so that the decision tree for GISelSelector remains compatible
with that of SelectionDAG.

https://reviews.llvm.org/D43270

llvm-svn: 325401
2018-02-16 22:37:15 +00:00
Rafael Auler de9ad4ba84 [X86][3DNOW] Teach decoder about AMD 3DNow! instrs
Summary:
This patch makes the decoder understand old AMD 3DNow!
instructions that have never been properly supported in the X86
disassembler, despite being supported in other subsystems. Hopefully
this should make the X86 decoder more complete with respect to binaries
containing legacy code.

Reviewers: craig.topper

Reviewed By: craig.topper

Subscribers: llvm-commits, maksfb, bruno

Differential Revision: https://reviews.llvm.org/D43311

llvm-svn: 325295
2018-02-15 21:20:31 +00:00
Craig Topper df99baa4df [X86] Teach EVEX->VEX pass to turn VRNDSCALE into VROUND when bits 7:4 of the immediate are 0 and the regular EVEX->VEX checks pass.
Bits 7:4 control the scale part of the operation. If the scale is 0 the behavior is equivalent to VROUND.

Fixes PR36246

llvm-svn: 324985
2018-02-13 04:19:26 +00:00
Clement Courbet 873aa11dd9 [TargetSchedule] Fix r324582.
Increment was using the wrong NumUnits (the one from the ProcResGroup
and not the subunit).

llvm-svn: 324727
2018-02-09 10:28:46 +00:00
David Blaikie e6503d87da Fix missing field initializer warning in TableGen SubtargetEmitter
llvm-svn: 324643
2018-02-08 19:57:05 +00:00
Clement Courbet 39911e2ee6 [TargetSchedule] Expose sub-units of a ProcResGroup in MCProcResourceDesc.
Summary:
Right now using a ProcResource automatically counts as usage of all
super ProcResGroups. All this is done during codegen, so there is no
way for schedulers to get this information at runtime.

This adds the information of which individual ProcRes units are
contained in a ProcResGroup in MCProcResourceDesc.

Reviewers: gchatelet

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D43023

llvm-svn: 324582
2018-02-08 08:46:48 +00:00
Clement Courbet eb4f5d2890 [CodeGenSchedule][NFC] Always emit ProcResourceUnits.
Summary:
Right now only the ProcResourceUnits that are directly referenced by
instructions are emitted. This change emits all of them, so that
analysis passes can use the information.
This has no functional impact. It typically adds a few entries (e.g. 4
for X86/haswell) to the generated ProcRes table.

Reviewers: gchatelet

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D42903

llvm-svn: 324228
2018-02-05 12:23:51 +00:00
Sander de Smalen 5b691a10c0 [TableGen][AsmMatcherEmitter] Fix tied-constraint checking for InstAliases
Summary:
This is a bit of a reimplementation the work done in
https://reviews.llvm.org/D41446, since that patch only really works for
tied operands of instructions, not aliases.

Instead of checking the constraints based on the matched instruction's opcode,
this patch uses the match-info's convert function to check the operand
constraints for that specific instruction/alias.
This is based on the matched operands for the instruction, not the
resulting opcode of the MCInst.

This patch adds the following enum/table to the *GenAsmMatcher.inc file:
  enum {
    Tie0_1_1,
    Tie0_1_2,
    Tie0_1_5,
    ...
  };

  const char TiedAsmOperandTable[][3] = {
    /* Tie0_1_1 */ { 0, 1, 1 },
    /* Tie0_1_2 */ { 0, 1, 2 },
    /* Tie0_1_5 */ { 0, 1, 5 },
    ...
  };

And it is referenced directly in the ConversionTable, like this:
static const uint8_t ConversionTable[CVT_NUM_SIGNATURES][13] = {
  ...
  { CVT_95_addRegOperands, 1,
    CVT_95_addRegOperands, 2,
    CVT_Tied, Tie0_1_5,
    CVT_95_addRegOperands, 6, CVT_Done },
  ...


The Tie0_1_5 (and corresponding table) encodes that:
* Result operand 0 is the operand to copy (which is e.g. done when
  building up the operands to the MCInst in convertToMCInst())
* Asm operands 1 and 5 should be the same operands (which is checked
  in checkAsmTiedOperandConstraints()).

Reviewers: olista01, rengolin, fhahn, craig.topper, echristo, apazos, dsanders

Reviewed By: olista01

Subscribers: llvm-commits

Differential Revision: https://reviews.llvm.org/D42293

llvm-svn: 324196
2018-02-04 16:24:17 +00:00
Daniel Sanders 08464524c3 [ARM][GISel] PR35965 Constrain RegClasses of nested instructions built from Dst Pattern
Summary:
Apparently, we missed on constraining register classes of VReg-operands of all the instructions
built from a destination pattern but the root (top-level) one. The issue exposed itself
while selecting G_FPTOSI for armv7: the corresponding pattern generates VTOSIZS wrapped
into COPY_TO_REGCLASS, so top-level COPY_TO_REGCLASS gets properly constrained,
while nested VTOSIZS (or rather its destination virtual register to be exact) does not.

Fixing this by issuing GIR_ConstrainSelectedInstOperands for every nested GIR_BuildMI.

https://bugs.llvm.org/show_bug.cgi?id=35965
rdar://problem/36886530

Patch by Roman Tereshin

Reviewers: dsanders, qcolombet, rovka, bogner, aditya_nandakumar, volkan

Reviewed By: dsanders, qcolombet, rovka

Subscribers: aemerson, javed.absar, kristof.beyls, llvm-commits

Differential Revision: https://reviews.llvm.org/D42565

llvm-svn: 323692
2018-01-29 21:09:12 +00:00
Clement Courbet 3c299e1b71 [TableGen][NFC]Remove dead variable.
llvm-svn: 323525
2018-01-26 13:21:43 +00:00
Volkan Keles 4f3fa79842 [GlobalISel][TableGen] Fix the statistics for emitted patters
Collected statistics for the number of patterns emitted can be
incorrect because rules can be grouped if OptimizeMatchTable
is enabled. Increase the counter in RuleMatcher::emit(...)
to avoid that.

llvm-svn: 323391
2018-01-25 00:18:52 +00:00
Benjamin Kramer 4890a71f02 [TableGen] Add a way of getting the number of generic opcodes without including modular CodeGen headers.
This is a bit of a hack, but removes a cycle that broke modular builds
of LLVM. Of course the cycle is still there in form of a dependency
on the .def file.

llvm-svn: 323383
2018-01-24 22:35:11 +00:00
Hiroshi Inoue 501931b117 [NFC] fix trivial typos in comments
"the the" -> "the"

llvm-svn: 323302
2018-01-24 05:04:35 +00:00
Benjamin Kramer cbce2f02e9 [TableGen] Optimize the regex search.
llvm::Regex is still the slowest regex engine on earth, running it over
all instructions on X86 takes a while. Extract a prefix and use a binary
search to reduce the search space before we resort to regex matching.

There are a couple of caveats here:
- The generic opcodes are outside of the sorted enum. They're handled in an extra loop.
- If there's a top-level bar we can't use the prefix trick.
- We bail on top-level ?. This could be handled, but it's rare.

This brings the time to generate X86GenInstrInfo.inc from 21s to 4.7s on
my machine.

llvm-svn: 323277
2018-01-23 23:05:04 +00:00
Daniel Sanders 12e6e709e9 [globalisel][tablegen] Honour priority order within nested instructions.
It appears that we haven't been prioritizing rules that contain nested
instructions properly. InstructionOperandMatcher didn't override
isHigherPriorityThan so it never compared the instructions/operands/predicates
inside nested instructions.

Fixes PR35926. Thanks to Diana Picus for the bug report.

llvm-svn: 322754
2018-01-17 20:34:29 +00:00
Volkan Keles f7f2568613 [GlobalISel][TableGen] Add support for SDNodeXForm
Summary:
This patch adds CustomRenderer which renders the matched
operands to the specified instruction.

Targets can enable the matching of SDNodeXForm by adding
a definition that inherits from GICustomOperandRenderer and
GISDNodeXFormEquiv as follows.

def gi_imm8 : GICustomOperandRenderer<"renderImm8”>,
                       GISDNodeXFormEquiv<imm8_xform>;

Custom renderer functions should be of the form:
void render(MachineInstrBuilder &MIB, const MachineInstr &I);

Reviewers: dsanders, ab, rovka

Reviewed By: dsanders

Subscribers: kristof.beyls, javed.absar, llvm-commits, mgrang, qcolombet

Differential Revision: https://reviews.llvm.org/D42012

llvm-svn: 322582
2018-01-16 18:44:05 +00:00
Craig Topper 7a0c601f95 [X86] Revisit the fix I made years ago to make 'xchgl %eax, %eax' not encode using the 0x90 encoding in 64-bit mode.
Prior to this we had a separate instruction and register class that excluded eax to prevent matching the instruction that would encode with 0x90.

This patch changes this to just use an InstAlias to force xchgl %eax, %eax to use XCHG32rr instruction in 64-bit mode. This gets rid of the separate instruction and register class.

llvm-svn: 322532
2018-01-16 06:07:16 +00:00
Sander de Smalen 886510f350 [TableGen][AsmMatcherEmitter] Generate assembler checks for tied operands
Summary:
This extends TableGen's AsmMatcherEmitter with code that generates
a table with tied-operand constraints. The constraints are checked
when parsing the instruction. If an operand is not equal to its tied operand,
the assembler will give an error.

Patch [2/3] in a series to add operand constraint checks for SVE's predicated ADD/SUB.

Reviewers: olista01, rengolin, mcrosier, fhahn, craig.topper, evandro, echristo

Reviewed By: fhahn

Subscribers: javed.absar, llvm-commits

Differential Revision: https://reviews.llvm.org/D41446

llvm-svn: 322166
2018-01-10 10:10:56 +00:00
Aleksandar Beserminji f02ad15ff1 [mips] Improve diagnostics for instruction mapping
This patch improves diagnostic for case when mapped instruction 
does not contain a field listed under RowFields.

Differential Revision: https://reviews.llvm.org/D41778

llvm-svn: 322004
2018-01-08 16:25:40 +00:00
Craig Topper 85657d59a9 [X86] Don't put any EVEX_B instructions in the tablegen generated load folding tables.
EVEX_B means different things for memory and register forms. The instructions should not be considered equivalent.

llvm-svn: 321954
2018-01-07 06:24:28 +00:00
Craig Topper ad89541ae9 [TableGen] Make the ambiguous match debug messages from the AsmMatcherEmitter slightly more useful.
Don't report ambiguous matches on different variants. Print the variant number in the output.

llvm-svn: 321938
2018-01-06 19:20:32 +00:00
Hal Finkel 8b4bdfdbc4 [TableGen] Add support of Intrinsics with multiple returns
This change deals with intrinsics with multiple outputs, for example load
instrinsic with address updated.

DAG selection for Instrinsics could be done either through source code or
tablegen. Handling all intrinsics in source code would introduce a huge chunk
of repetitive code if we have a large number of intrinsic that return multiple
values (see NVPTX as an example). While intrinsic class in tablegen supports
multiple outputs, tablegen only supports Intrinsics with zero or one output on
TreePattern. This appears to be a simple bug in tablegen that is fixed by this
change.

For Intrinsics defined as:

  def int_xxx_load_addr_updated: Intrinsic<[llvm_i32_ty, llvm_ptr_ty], [llvm_ptr_ty, llvm_i32_ty], []>;

Instruction will be defined as:

  def L32_X: Inst<(outs reg:$d1, reg:$d2), (ins reg:$s1, reg:$s2), "ld32_x $d1, $d2, $s2", [(set i32:$d1, i32:$d2, (int_xxx_load_addr_updated i32:$s1, i32:$s2))]>;

Patch by Wenbo Sun, thanks!

Differential Revision: https://reviews.llvm.org/D32888

llvm-svn: 321704
2018-01-03 11:35:09 +00:00
Benjamin Kramer 3a13ed60ba Avoid int to string conversion in Twine or raw_ostream contexts.
Some output changes from uppercase hex to lowercase hex, no other functionality change intended.

llvm-svn: 321526
2017-12-28 16:58:54 +00:00
Krzysztof Parzyszek 2e0f7bd0fe [TableGen] Print more helpful information in case of type contradiction
Dump the failing TreePattern.

llvm-svn: 321282
2017-12-21 17:12:43 +00:00
Aaron Ballman 82e17f58f0 Do not generate an empty switch statement as it causes MSVC to issue diagnostics about switch statements without case or default labels.
llvm-svn: 321217
2017-12-20 20:09:30 +00:00
Matt Arsenault 303327d58b TableGen: Allow setting SDNodeProperties on intrinsics
Allows preserving MachineMemOperands on intrinsics
through selection. For reasons I don't understand, this
is a static property of the pattern and the selector
deliberately goes out of its way to drop if not present.

Intrinsics already inherit from SDPatternOperator allowing
them to be used directly in instruction patterns. SDPatternOperator
has a list of SDNodeProperty, but you currently can't set them on
the intrinsic. Without SDNPMemOperand, when the node is selected
any memory operands are always dropped. Allowing setting this
on the intrinsics avoids needing to introduce another equivalent
target node just to have SDNPMemOperand set.

llvm-svn: 321212
2017-12-20 19:36:28 +00:00
Daniel Sanders 32de8bbd30 [globalisel][tablegen] Allow ImmLeaf predicates to use InstructionSelector members
NFC for currently supported targets. This resolves a problem encountered by
targets such as RISCV that reference `Subtarget` in ImmLeaf predicates.

llvm-svn: 321176
2017-12-20 14:41:51 +00:00
Sander de Smalen cd6be960ce [AArch64][SVE] Re-submit patch series for ZIP1/ZIP2
This patch resubmits the SVE ZIP1/ZIP2 patch series consisting of
of r320992, r320986, r320973, and r320970 by reverting
https://reviews.llvm.org/rL321024.

The issue that caused r321024 has been addressed in https://reviews.llvm.org/rL321158,
so this patch-series should be safe to resubmit.

llvm-svn: 321163
2017-12-20 11:02:42 +00:00
Adrian Prantl 0e6694d111 Silence a bunch of implicit fallthrough warnings
llvm-svn: 321114
2017-12-19 22:05:25 +00:00
Quentin Colombet 63a328c30c [TableGen][GlobalISel] Reset the internal map of RuleMatchers just before the emission
Between the creation of the last InstructionMatcher and the first
emission of the related Rule, we need to clear the internal map of IDs.
We used to do that right after the creation of the main
InstructionMatcher when building the rule and although that worked, this
is fragile because if for some reason some later code decides to create
more InstructionMatcher before the final call to emit, then the IDs
would be completely messed up.

Move that to the beginning of "emit" so that the IDs are guarantee to be
consistent.

NFC.

llvm-svn: 321053
2017-12-19 02:57:23 +00:00
Quentin Colombet eba10cbc88 [TableGen][GlobalISel] Make the arguments of the Instruction and Operand Matchers consistent
Move InsnVarID and OpIdx at the beginning of the list of arguments
for all the constructors of the OperandMatcher subclasses.
This matches what we do for the InstructionMatcher.

NFC.

llvm-svn: 321031
2017-12-18 22:12:13 +00:00
Quentin Colombet 34688b9e38 [TableGen][GlobalISel] Refactor optimizeRules related bit to allow code reuse
In theory, reapplying optimizeRules on each group matchers should give
us a second nesting level on the matching table. In practice, we need
more work to make that happen because all the predicates are actually
not directly available through the predicate matchers list.

NFC.

llvm-svn: 321025
2017-12-18 21:25:53 +00:00
Reid Kleckner 37517a2ddd Revert "[AArch64][SVE] Asm" changes, they broke libjpeg_turbo
This reverts changes r320992, r320986, r320973, and r320970.

r320970 by itself breaks the test case, and the rest depend on it.

Test case will land soon.

llvm-svn: 321024
2017-12-18 20:58:25 +00:00
Quentin Colombet ec76d9c47f [TableGen][GlobalISel] Optimize MatchTable for faster instruction selection
*** Context ***

Prior to this patchw, the table generated for matching instruction was
straight forward but highly inefficient.

Basically, each pattern generates its own set of self contained checks
and actions.
E.g., TableGen generated:
// First pattern
CheckNumOperand 3
CheckOpcode G_ADD
...
Build ADDrr
// Second pattern
CheckNumOperand 3
CheckOpcode G_ADD
...
Build ADDri
// Third pattern
CheckNumOperand 3
CheckOpcode G_SUB
...
Build SUBrr

*** Problem ***

Because of that generation, a *lot* of check were redundant between each
pattern and were checked every single time until we reach the pattern
that matches.
E.g., Taking the previous table, let say we are matching a G_SUB, that
means we were going to check all the rules for G_ADD before looking at
the G_SUB rule. In particular we are going to do:
check 3 operands; PASS
check G_ADD; FAIL
; Next rule
check 3 operands; PASS (but we already knew that!)
check G_ADD; FAIL (well it is still not true)
; Next rule
check 3 operands; PASS (really!!)
check G_SUB; PASS (at last :P)

*** Proposed Solution ***

This patch introduces a concept of group of rules (GroupMatcher) that
share some predicates and only get checked once for the whole group.

This patch only creates groups with one nesting level. Conceptually
there is nothing preventing us for having deeper nest level. However,
the current implementation is not smart enough to share the recording
(aka capturing) of values. That limits its ability to do more sharing.

For the given example the current patch will generate:
// First group
CheckOpcode G_ADD

 // First pattern
 CheckNumOperand 3
 ...
 Build ADDrr
 // Second pattern
 CheckNumOperand 3
 ...
 Build ADDri

// Second group
CheckOpcode G_SUB

 // Third pattern
 CheckNumOperand 3
 ...
 Build SUBrr

But if we allowed several nesting level, it could create a sub group
for the checknumoperand 3.
(We would need to call optimizeRules on the rules within a group.)

*** Result ***

With only one level of nesting, the instruction selection pass is up
to 4x faster. For instance, one instruction now takes 500 checks,
instead of 24k! With more nesting we could get in the tens I believe.

Differential Revision: https://reviews.llvm.org/D39034

rdar://problem/34670699

llvm-svn: 321017
2017-12-18 19:47:41 +00:00
Sander de Smalen 09f56a54d0 [AArch64][SVE] Asm: Improve diagnostics further when +sve is not specified
Summary: Patch [4/4] in a series to add parsing of predicates and properly parse SVE ZIP1/ZIP2 instructions. This patch further improves diagnostic messages for when the SVE feature is not specified.

Reviewers: rengolin, fhahn, olista01, echristo, efriedma

Reviewed By: fhahn

Subscribers: sdardis, aemerson, javed.absar, tschuett, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D40363

llvm-svn: 320992
2017-12-18 16:48:53 +00:00
Sander de Smalen 190979189a [TableGen][AsmMatcherEmitter] Only choose specific diagnostic for enabled instruction
Summary:
When emitting a diagnostic for an invalid operand, a specific diagnostic
should only be reported when the instruction being matched is actually
enabled by the feature flags.

Patch [3/4] in a series to add parsing of predicates and properly parse SVE 
ZIP1/ZIP2 instructions. This patch fixes bogus diagnostic messages for when
the SVE feature is not specified.

Reviewers: rengolin, craig.topper, olista01, sdardis, stoklund

Reviewed By: olista01, sdardis

Subscribers: fhahn, javed.absar, llvm-commits

Differential Revision: https://reviews.llvm.org/D40362

llvm-svn: 320986
2017-12-18 14:34:24 +00:00
Quentin Colombet 893e0f15e2 [TableGen][GlobalISel] Make the different Matcher comparable
This opens refactoring opportunities in the match table now that we can
check that two predicates are the same.

NFC.

llvm-svn: 320890
2017-12-15 23:24:39 +00:00
Quentin Colombet a646ef08e8 [TableGen][GlobalISel] Fix unused variable warning in release mode
Introduced in r320887.

NFC.

llvm-svn: 320889
2017-12-15 23:24:36 +00:00
Quentin Colombet aad20be6ca [TableGen][GlobalISel] Have the predicate directly know which data they are dealing with
Prior to this patch, a predicate wouldn't make sense outside of its
rule. Indeed, it was only during emitting a rule that a predicate would
be made aware of the IDs of the data it is checking. Because of that,
predicates could not be moved around or compared between each other.

NFC.

llvm-svn: 320887
2017-12-15 23:07:42 +00:00
Quentin Colombet 063d798acb [TableGen][GlobalISel] Add a common class for all PredicateMatcher
NFC.

llvm-svn: 320767
2017-12-14 23:44:07 +00:00
Krzysztof Parzyszek 2aaeeb40b3 Add MVT::v128i1, NFC
Hexagon HVX has type v128i8, comparing two vectors of that type will
produce v128i1 types in SelectionDAG.

llvm-svn: 320732
2017-12-14 19:05:21 +00:00
Sander de Smalen 14e36ee5c3 Re-commit: [TableGen] AsmMatcher: Fix bug with reported diagnostic for operand.
Summary:
The generated diagnostic by the AsmMatcher isn't always applicable to the AsmOperand.

This is because the code will only update the diagnostic if it is more 
specific than the previous diagnostic. However, when having validated
operands and 'moved on' to a next operand (for some instruction/alias for
which all previous operands are valid), if the diagnostic is InvalidOperand,
than that should be set as the diagnostic, not the more specific message
about a previous operand for some other instruction/alias candidate.

(Re-committed with an extra whitespace in SVEInstrFormats.td to trigger rebuild 
of AArch64GenAsmMatcher.inc, since the llvm-clang-x86_64-expensive-checks-win
builder does not seem to rebuild AArch64GenAsmMatcher.inc with the
newly built TableGen due to a missing dependency somewhere (see:
http://lists.llvm.org/pipermail/llvm-dev/2017-December/119555.html))

Reviewers: craig.topper, olista01, rengolin, stoklund

Reviewed By: olista01

Subscribers: javed.absar, llvm-commits

Differential Revision: https://reviews.llvm.org/D40011

llvm-svn: 320711
2017-12-14 16:09:48 +00:00
Michael Zolotukhin 4d6b43ca94 Remove redundant includes from utils/TableGen.
llvm-svn: 320632
2017-12-13 21:31:13 +00:00
Craig Topper ac59db2efe [Targets] Don't automatically include the scheduler class enum from *GenInstrInfo.inc with GET_INSTRINFO_ENUM. Make targets request is separately.
Most of the targets don't need the scheduler class enum.

I have an X86 scheduler model change that causes some names in the enum to become about 18000 characters long. This is because using instregex in scheduler models causes the scheduler class to get named with every instruction that matches the regex concatenated together. MSVC has a limit of 4096 characters for an identifier name. Rather than trying to come up with way to reduce the name length, I'm just going to sidestep the problem by not including the enum in X86.

llvm-svn: 320552
2017-12-13 07:26:17 +00:00
Richard Smith 8a3adc3abb Avoid constructing an out-of-range value for an enumeration (which results in UB).
llvm-svn: 320206
2017-12-08 22:32:35 +00:00
Alex Bradbury d590c85753 [TableGen] Give the option of tolerating duplicate register names
A number of architectures re-use the same register names (e.g. for both 32-bit 
FPRs and 64-bit FPRs). They are currently unable to use the tablegen'erated 
MatchRegisterName and MatchRegisterAltName, as tablegen (when built with 
asserts enabled) will fail.

When the AllowDuplicateRegisterNames in AsmParser is set, duplicated register 
names will be tolerated. A backend can then coerce registers to the desired 
register class by (for instance) implementing validateTargetOperandClass.

At least the in-tree Sparc backend could benefit from this, as does RISC-V 
(single and double precision floating point registers).

Differential Revision: https://reviews.llvm.org/D39845

llvm-svn: 320018
2017-12-07 09:51:55 +00:00
Daniel Sanders 3c1c4c0ee0 Revert r319691: [globalisel][tablegen] Split atomic load/store into separate opcode and enable for AArch64.
Some concerns were raised with the direction. Revert while we discuss it and look into an alternative

llvm-svn: 319739
2017-12-05 05:52:07 +00:00
Daniel Sanders 04e4f47e93 [globalisel][tablegen] Split atomic load/store into separate opcode and enable for AArch64.
This patch splits atomics out of the generic G_LOAD/G_STORE and into their own
G_ATOMIC_LOAD/G_ATOMIC_STORE. This is a pragmatic decision rather than a
necessary one. Atomic load/store has little in implementation in common with
non-atomic load/store. They tend to be handled very differently throughout the
backend. It also has the nice side-effect of slightly improving the common-case
performance at ISel since there's no longer a need for an atomicity check in the
matcher table.

All targets have been updated to remove the atomic load/store check from the
G_LOAD/G_STORE path. AArch64 has also been updated to mark
G_ATOMIC_LOAD/G_ATOMIC_STORE legal.

There is one issue with this patch though which also affects the extending loads
and truncating stores. The rules only match when an appropriate G_ANYEXT is
present in the MIR. For example,
  (G_ATOMIC_STORE (G_TRUNC:s16 (G_ANYEXT:s32 (G_ATOMIC_LOAD:s16 X))))
will match but:
  (G_ATOMIC_STORE (G_ATOMIC_LOAD:s16 X))
will not. This shouldn't be a problem at the moment, but as we get better at
eliminating extends/truncates we'll likely start failing to match in some
cases. The current plan is to fix this in a patch that changes the
representation of extending-load/truncating-store to allow the MMO to describe
a different type to the operation.

llvm-svn: 319691
2017-12-04 20:39:32 +00:00
Oliver Stannard 7ab60605f8 Revert r319649 - [Asm, ARM] Add fallback diag for multiple invalid operands
This is causing a failure in the llvm-clang-x86_64-expensive-checks-win
buildbot, and I can't reproduce it locally, so reverting until I can work out
what is wrong.

llvm-svn: 319654
2017-12-04 13:42:22 +00:00
Oliver Stannard 7cd4db94f8 [Asm, ARM] Add fallback diag for multiple invalid operands
This adds a "invalid operands for instruction" diagnostic for
instructions where there is an instruction encoding with the correct
mnemonic and which is available for this target, but where multiple
operands do not match those which were provided. This makes it clear
that there is some combination of operands that is valid for the current
target, which the default diagnostic of "invalid instruction" does not.

Since this is a very general error, we only emit it if we don't have a
more specific error.

Differential revision: https://reviews.llvm.org/D36747

llvm-svn: 319649
2017-12-04 12:02:32 +00:00
Matt Arsenault 2e8be9d126 Fix typo in emitted attribute name
Fixes build when using this attribute combination
on an intrinsic.

llvm-svn: 319625
2017-12-03 00:03:01 +00:00
Daniel Sanders 0c43b3a023 [globalisel][tablegen] Add support for relative AtomicOrderings
No test yet because the relevant rules are blocked on the atomic_load,
and atomic_store nodes.

llvm-svn: 319475
2017-11-30 21:05:59 +00:00
Daniel Sanders f499b2bf1f [globalisel][tablegen] Add support for specific immediates in the match pattern
This enables a few rules such as ARM's uxtb instruction.

llvm-svn: 319457
2017-11-30 18:48:35 +00:00
Daniel Sanders 766646517f [globalisel][tablegen] Add support for importing G_ATOMIC_CMPXCHG, G_ATOMICRMW_* rules from SelectionDAG.
GIM_CheckNonAtomic has been replaced by GIM_CheckAtomicOrdering to allow it to support a wider
range of orderings. This has then been used to import patterns using nodes such
as atomic_cmp_swap, atomic_swap, and atomic_load_*.

llvm-svn: 319232
2017-11-28 22:07:05 +00:00
George Rimar 33894b619b Revert r318822 "[llvm-tblgen] - Stop using std::string in RecordKeeper."
It reported to have problems with memory sanitizers and DBUILD_SHARED_LIBS=ON.

llvm-svn: 318899
2017-11-23 06:52:44 +00:00
George Rimar 860a7b7901 [llvm-tblgen] - Stop using std::string in RecordKeeper.
RecordKeeper::getDef() is a hot place, it shows up in profiling
and it creates std::string instance for each search in RecordMap
though RecordKeeper::RecordMap can use StringRef as a key
instead to avoid that. Patch do that change.

Differential revision: https://reviews.llvm.org/D40170

llvm-svn: 318822
2017-11-22 07:53:48 +00:00
Craig Topper fb0d4cd48c [SelectionDAG] Add a isel matcher op to check the type of node results other than result 0.
I plan to use this to check the type of the mask result of masked gathers in the X86 backend.

llvm-svn: 318820
2017-11-22 07:11:01 +00:00
Evandro Menezes 9dc54e25f0 [TableGen] Improve error reporting
When searching for a resource unit, use the reference location instead of
the definition location in case of an error.

Differential revision: https://reviews.llvm.org/D40263

llvm-svn: 318803
2017-11-21 21:33:52 +00:00
Oliver Stannard 1e73e95f3c [Asm] Improve "too few operands" errors
- We can still emit this error if the actual instruction has two or more
  operands missing compared to the expected one.
- We should only emit this error once per instruction.

Differential revision: https://reviews.llvm.org/D36746

llvm-svn: 318770
2017-11-21 15:16:50 +00:00
Oliver Stannard 6e94331259 [Asm] Finish matching once end of formal and actual lists reached (NFC)
This is NFC, as the matcher would continue looping up to the maximum
number of operands with no effect, but this should improve performance a
bit, and makes the debug trace clearer.

Differential revision: https://reviews.llvm.org/D36744

llvm-svn: 318769
2017-11-21 15:12:05 +00:00
Sander de Smalen 4acd57eb51 Revert r318759 due to make check-all failure on Windows
llvm-svn: 318768
2017-11-21 15:07:43 +00:00
Sander de Smalen f475eed48d [TableGen] AsmMatcher: Fix bug with reported diagnostic for operand.
Summary:
The generated diagnostic by the AsmMatcher isn't always applicable to the AsmOperand.

This is because the code will only update the diagnostic if it is more specific than the previous diagnostic. However, when having validated operands and 'moved on' to a next operand (for some instruction/alias for which all previous operands are valid), if the diagnostic is InvalidOperand, than that should be set as the diagnostic, not the more specific message about a previous operand for some other instruction/alias candidate.

Reviewers: craig.topper, olista01, rengolin, stoklund

Reviewed By: olista01

Subscribers: javed.absar, llvm-commits

Differential Revision: https://reviews.llvm.org/D40011

llvm-svn: 318759
2017-11-21 12:26:06 +00:00
Sander de Smalen 0c5a29b6be [AArch64][TableGen] Skip tied result operands for InstAlias
Summary:
This patch fixes an issue so that the right alias is printed when the instruction has tied operands. It checks the number of operands in the resulting instruction as opposed to the alias, and then skips over tied operands that should not be printed in the alias.

This allows to generate the preferred assembly syntax for the AArch64 'ins' instruction, which should always be displayed as 'mov' according to the ARM Architecture Reference Manual. Several unit tests have changed as a result, but only to reflect the preferred disassembly. Some other InstAlias patterns (movk/bic/orr) needed a slight adjustment to stop them becoming the default and breaking other unit tests.

Please note that the patch is mostly the same as https://reviews.llvm.org/D29219 which was reverted because of an issue found when running TableGen with the Address Sanitizer. That issue has been addressed in this iteration of the patch.


Reviewers: rengolin, stoklund, huntergr, SjoerdMeijer, rovka

Reviewed By: rengolin, SjoerdMeijer

Subscribers: fhahn, aemerson, javed.absar, kristof.beyls, llvm-commits

Differential Revision: https://reviews.llvm.org/D40030

llvm-svn: 318650
2017-11-20 14:36:40 +00:00
Daniel Sanders c54aa9c844 [globalisel][tablegen] Generalize pointer-type inference by introducing ptypeN. NFC
ptypeN is functionally the same as typeN except that it informs the
SelectionDAG importer that an operand should be treated as a pointer even
if it was written as iN. This is important for patterns that use iN instead
of iPTR to represent pointers. E.g.:
  (set GPR64:$dst, (load GPR64:$addr))

Previously, this was handled as a hardcoded special case for the appropriate
operands to G_LOAD and G_STORE.

llvm-svn: 318574
2017-11-18 00:16:44 +00:00
David Blaikie b3bde2ea50 Fix a bunch more layering of CodeGen headers that are in Target
All these headers already depend on CodeGen headers so moving them into
CodeGen fixes the layering (since CodeGen depends on Target, not the
other way around).

llvm-svn: 318490
2017-11-17 01:07:10 +00:00
Daniel Sanders f76f315436 [globalisel][tablegen] Generate rule coverage and use it to identify untested rules
Summary:
This patch adds a LLVM_ENABLE_GISEL_COV which, like LLVM_ENABLE_DAGISEL_COV,
causes TableGen to instrument the generated table to collect rule coverage
information. However, LLVM_ENABLE_GISEL_COV goes a bit further than
LLVM_ENABLE_DAGISEL_COV. The information is written to files
(${CMAKE_BINARY_DIR}/gisel-coverage-* by default). These files can then be
concatenated into ${LLVM_GISEL_COV_PREFIX}-all after which TableGen will
read this information and use it to emit warnings about untested rules.

This technique could also be used by SelectionDAG and can be further
extended to detect hot rules and give them priority over colder rules.

Usage:
* Enable LLVM_ENABLE_GISEL_COV in CMake
* Build the compiler and run some tests
* cat gisel-coverage-[0-9]* > gisel-coverage-all
* Delete lib/Target/*/*GenGlobalISel.inc*
* Build the compiler

Known issues:
* ${LLVM_GISEL_COV_PREFIX}-all must be generated as a manual
  step due to a lack of a portable 'cat' command. It should be the
  concatenation of all ${LLVM_GISEL_COV_PREFIX}-[0-9]* files.
* There's no mechanism to discard coverage information when the ruleset
  changes

Depends on D39742

Reviewers: ab, qcolombet, t.p.northover, aditya_nandakumar, rovka

Reviewed By: rovka

Subscribers: vsk, arsenm, nhaehnle, mgorny, kristof.beyls, javed.absar, igorb, llvm-commits

Differential Revision: https://reviews.llvm.org/D39747

llvm-svn: 318356
2017-11-16 00:46:35 +00:00
Oliver Stannard 174fdef458 [Docs] Add tablegen backend for target opcode documentation
This is a tablegen backend to generate documentation for the opcodes that exist
for each target. For each opcode, it lists the assembly string, the names and
types of all operands, and the flags and predicates that apply to the opcode.

Differential revision: https://reviews.llvm.org/D31025

llvm-svn: 318155
2017-11-14 15:35:15 +00:00
Daniel Sanders 6d9d30a917 [tablegen] Handle atomic predicates for ordering inside tablegen. NFC.
Similar to r315841, GlobalISel and SelectionDAG require different code for the
common atomic predicates due to differences in the representation.
Even without that, differences in the IR (SDNode vs MachineInstr) require
differences in the C++ predicate.

This patch moves the implementation of the common atomic predicates related to
ordering into tablegen so that it can handle these differences.

It's NFC for SelectionDAG since it emits equivalent code and it's NFC for
GlobalISel since the rules involving the relevant predicates are still
rejected by the importer.

llvm-svn: 318102
2017-11-13 23:03:47 +00:00
Daniel Sanders 87d196ca48 [tablegen] Handle atomic predicates for memory type inside tablegen. NFC.
Similar to r315841, GlobalISel and SelectionDAG require different code for the
common atomic predicates due to differences in the representation.
Even without that, differences in the IR (SDNode vs MachineInstr) require
differences in the C++ predicate.

This patch moves the implementation of the common atomic predicates related to
memory type into tablegen so that it can handle these differences.

It's NFC for SelectionDAG since it emits equivalent code and it's NFC for
GlobalISel since the rules involving the relevant predicates are still
rejected by the importer.

llvm-svn: 318095
2017-11-13 22:26:13 +00:00
Daniel Sanders b78ac6e322 [globalisel][tablegen] Add support for extload.
llvm-svn: 318068
2017-11-13 18:30:23 +00:00
Petar Jovanovic bd57b8bf3f fix printing of alias instructions by removing redundant spacing
Some alias instructions are printed with an extra space after the tab
character. Fix this by skipping that space when the tab character is printed
so that the instructions are aligned with the rest of the code.

Patch by Milos Stojanovic.

Differential Revision: https://reviews.llvm.org/D35946

llvm-svn: 318059
2017-11-13 18:00:24 +00:00
Daniel Sanders 7e52367398 [globalisel][tablegen] Import signextload and zeroextload.
Allow a pattern rewriter to be installed in CodeGenDAGPatterns and use it to
correct situations where SelectionDAG and GlobalISel disagree on
representation. For example, it would rewrite:
  (sextload:i32 $ptr)<<unindexedload>><<sextload>><<sextloadi16>
to:
  (sext:i32 (load:i16 $ptr)<<unindexedload>>)

I'd have preferred to replace the fragments and have the expansion happen
naturally as part of PatFrag expansion but the type inferencing system can't
cope with loads of types narrower than those mentioned in register classes.
This is because the SDTCisInt's on the sext constrain both the result and
operand to the 'legal' integer types (where legal is defined as 'a register
class can contain the type') which immediately rules the narrower types out.
Several targets (those with only one legal integer type) would then go on to
crash on the SDTCisOpSmallerThanOp<> when it removes all the possible types
for the result of the extend.

Also, improve isObviouslySafeToFold() slightly to automatically return true for
neighbouring instructions. There can't be any re-ordering problems if
re-ordering isn't happenning. We'll need to improve it further to handle
sign/zero-extending loads when the extend and load aren't immediate neighbours
though.

llvm-svn: 317971
2017-11-11 03:23:44 +00:00
Krzysztof Parzyszek b8c68c67dc Allow separation of declarations and definitions in <Target>ISelDAGToDAG.inc
This patch adds the ability to include the member function declarations
in the instruction selector class separately from the member bodies.

Defining GET_DAGISEL_DECL macro to any value will only include the member
declarations. To include bodies, define GET_DAGISEL_BODY macro to be the
selector class name. Example:

  class FooDAGToDAGISel : public SelectionDAGISel {
    // Pull in declarations only.
    #define GET_DAGISEL_DECL
    #include "FooISelDAGToDAG.inc"
  };

  // Include the function bodies (with names qualified with the provided
  // class name).
  #define GET_DAGISEL_BODY FooDAGToDAGISel
  #include "FooISelDAGToDAG.inc"

When neither of the two macros are defined, the function bodies are emitted
inline (in the same way as before this patch).

Differential Revision: https://reviews.llvm.org/D39596

llvm-svn: 317903
2017-11-10 18:36:04 +00:00
Florian Hahn 603c6455d2 [AArch64][SVE] Asm: Extend EnforceVectorSubVectorTypeIs to distinguish Scalable Vectors
Patch [1/5] in a series to add assembler/disassembler support for AArch64 SVE
unpredicated ADD/SUB instructions.

Patch by Sander De Smalen.

Reviewed by: rengolin

Differential Revision: https://reviews.llvm.org/D39087

llvm-svn: 317564
2017-11-07 10:43:56 +00:00
Aaron Ballman cbaf5a4f50 Move the llvm-tblgen project into the Tablegenning folder on IDEs like Visual Studio rather than leave it in the root directory. NFC.
llvm-svn: 317420
2017-11-04 20:07:16 +00:00
Craig Topper e5d44cefea [X86] Teach EVEX->VEX pass to turn SHUFI32X4/SHUFF32X4/SHUFI64X/SHUFF64X2 into VPERM2F128/VPERM2I128.
This recovers some of the tests that were changed by r317403.

llvm-svn: 317410
2017-11-04 18:10:03 +00:00
Diana Picus d1b618177a [globalisel][tablegen] Skip src child predicates
The GlobalISel TableGen backend didn't check for predicates on the
source children. This caused it to generate code for ARM patterns such
as SMLABB or similar, but without properly checking for the sext_16_node
part of the operands. This in turn meant that we would select SMLABB
instead of MLA for simple sequences such as s32 + s32 * s32, which is
wrong (we want a MLA on the full operands, not just their bottom 16
bits).

This patch forces TableGen to skip patterns with predicates on the src
children, so it doesn't generate code for SMLABB and other similar ARM
instructions at all anymore. AArch64 and X86 are not affected.

Differential Revision: https://reviews.llvm.org/D39554

llvm-svn: 317313
2017-11-03 10:30:19 +00:00
Craig Topper 2fda36a18e [TableGen] Add an extra blank line to DAGISel output file to separate functions.
llvm-svn: 317298
2017-11-03 05:19:34 +00:00
Daniel Sanders 466fe399b8 [globalisel][regbank] Warn about MIR ambiguities when register bank/class names clash.
llvm-svn: 317132
2017-11-01 22:13:05 +00:00
Craig Topper 4e56ba271e [X86] Add custom code to EVEX to VEX pass to turn unmasked 128-bit VPALIGND/Q into VPALIGNR if the extended registers aren't being used.
This will enable us to prefer VALIGND/Q during shuffle lowering in order to get the extended register encoding space when BWI isn't available. But if we end up not using the extended registers we can switch VPALIGNR for the shorter VEX encoding.

Differential Revision: https://reviews.llvm.org/D39401

llvm-svn: 317122
2017-11-01 21:00:59 +00:00
Daniel Sanders 9cbe7c7f93 [globalisel][tablegen] Add support for multi-insn emission
The importer will now accept nested instructions in the result pattern such as
(ADDWrr $a, (SUBWrr $b, $c)). This is only valid when the nested instruction
def's a single vreg and the parent instruction consumes a single vreg where a
nested instruction is specified. The importer will automatically create a vreg
to connect the two using the type information from the pattern. This vreg will
be constrained to the register classes given in the instruction definitions*.

* REG_SEQUENCE is explicitly rejected because of this. The definition doesn't
  constrain to a register class and it therefore needs special handling.

llvm-svn: 317117
2017-11-01 19:57:57 +00:00
NAKAMURA Takumi 1657f2ad99 Fix warnings discovered by rL317076. [-Wunused-private-field]
llvm-svn: 317091
2017-11-01 13:47:55 +00:00
Daniel Sanders 198447a447 [globalisel][tablegen] Stop hard-coding the emitted instruction ID to 0. NFC
The next commit will add support for multi-instruction emission so we need to
start allocating instruction ID's instead of hard-coding them to 0.

llvm-svn: 317057
2017-11-01 00:29:47 +00:00
Daniel Sanders 7438b26317 Re-commit: [globalisel][tablegen] Keep track of the insertion point while adding BuildMIAction's. NFC
Multi-instruction emission needs to ensure the the instructions are generated
a depth-first fashion. For example:
(ADDWrr (SUBWrr a, b), c)
needs to emit the SUBWrr before the ADDWrr. However, our walk over
TreePatternNode's is highly context sensitive which makes it difficult to append
BuildMIActions in the order we want. To fix this, we now keep track of the
insertion point as we add actions. This will allow multi-insn emission to insert
BuildMI's in the correct place.

The previous commit failed on the Ubuntu bots using GCC 4.8. These bots lack the
const_iterator forms of insert() and emplace() that were added in C++11. As a
result I've switched the const_iterators to iterators.

llvm-svn: 317049
2017-10-31 23:03:18 +00:00
Daniel Sanders f69d1b018c Revert r317040: [globalisel][tablegen] Keep track of the insertion point while adding BuildMIAction's. NFC
The same bots fail but I believe I know what the issue is now. These bots are
missing the const_iterator versions of insert/emplace/etc. that were introduced
in C++11.

llvm-svn: 317042
2017-10-31 21:54:52 +00:00
Daniel Sanders 374f71ac90 Re-commit: [globalisel][tablegen] Keep track of the insertion point while adding BuildMIAction's. NFC
Multi-instruction emission needs to ensure the the instructions are generated
a depth-first fashion. For example:
 (ADDWrr (SUBWrr a, b), c)
needs to emit the SUBWrr before the ADDWrr. However, our walk over
TreePatternNode's is highly context sensitive which makes it difficult to append
BuildMIActions in the order we want. To fix this, we now keep track of the
insertion point as we add actions. This will allow multi-insn emission to insert
BuildMI's in the correct place.

The previous commit failed on the Ubuntu bots using GCC 4.8. These bots didn't
like a call to emplace(). I've replaced it with insert() to see if it's a quirk
of the C++11 support.

llvm-svn: 317040
2017-10-31 21:34:53 +00:00
Daniel Sanders f87c142d9e Revert r317029: [globalisel][tablegen] Keep track of the insertion point while adding BuildMIAction's. NFC
The Linux bots don't seem to like this usage of emplace(). Reverting while I look into it.

llvm-svn: 317033
2017-10-31 20:29:28 +00:00
Daniel Sanders e8a4f0d869 [globalisel][tablegen] Keep track of the insertion point while adding BuildMIAction's. NFC
Multi-instruction emission needs to ensure the the instructions are generated
a depth-first fashion. For example:
  (ADDWrr (SUBWrr a, b), c)
needs to emit the SUBWrr before the ADDWrr. However, our walk over
TreePatternNode's is highly context sensitive which makes it difficult to append
BuildMIActions in the order we want. To fix this, we now keep track of the
insertion point as we add actions. This will allow multi-insn emission to insert
BuildMI's in the correct place.

llvm-svn: 317029
2017-10-31 19:54:05 +00:00
Daniel Sanders df258e328a [globalisel][tablegen] Factor out implicit def/use renderers from createAndImportInstructionRenderer(). NFC
Multi-instruction emission will require that we have separate handling for
the defs between the implicitly created temporaries and the rule outputs.
The former require new temporary vregs while the latter should copy existing
operands. Factor out the implicit def/use renderers to minimize the code
duplication when we implement that.

llvm-svn: 317025
2017-10-31 19:09:29 +00:00
Daniel Sanders a7b7526416 [globalisel][tablegen] Add infrastructure to potentially allow BuildMIAction to choose a mutatable instruction. NFC
Prepare for multiple instruction emission by allowing BuildMIAction to
search for a suitable matcher that will support mutation.

This patch deliberately neglects to add matchers aside from the root to
preserve NFC. That said, it should be noted that until we support mutations
other than just the opcode the chances of finding a non-root instruction
for which canMutate() is true, is essentially zero. Furthermore in the
presence of multi-instruction emission the chances of finding any
instruction for which canMutate() is true is also zero. Nevertheless, we
can't continue to require that all BuildMIAction's consider the root of the match
to be recyclable due to the risk of recycling it twice in the same rule.

llvm-svn: 317022
2017-10-31 18:50:24 +00:00
Daniel Sanders 6ea17edd80 [globalisel][tablegen] Allow any comment in DebugCommentAction. NFC
llvm-svn: 317017
2017-10-31 18:07:03 +00:00
Craig Topper 01ebd9b7f7 [TableGen] Use Twine instead of std::string concatenation in two calls to PrintFatalError.
llvm-svn: 316697
2017-10-26 20:49:36 +00:00
Craig Topper 0551556ed2 [AsmParser][TableGen] Add VariantID argument to the generated mnemonic spell check function so it can use the correct table based on variant.
I'm considering implementing the mnemonic spell checker for x86, and that would require the separate intel and att variants.

llvm-svn: 316641
2017-10-26 06:46:41 +00:00
Craig Topper 2a06028c0a [AsmParser][TableGen] Make the generated mnemonic spell checker function a file local static function.
Also only emit in targets that specificially request it. This is required so we don't get an unused static function error.

llvm-svn: 316640
2017-10-26 06:46:40 +00:00
Daniel Sanders ab1d119154 [globalisel][tablegen] Fix future undefined behaviour in r316463.
I missed a dereference of `Matched` that preceeded the new check. Thanks to
Justin Bogner for spotting it.

llvm-svn: 316480
2017-10-24 18:11:54 +00:00
Daniel Sanders 64f745cf20 [globalisel][tablegen] Multi-insn emission requires that BuildMIAction support not being linked to an InstructionMatcher. NFC
When multi-instruction emission is supported, it will no longer be guaranteed
that every BuildMIAction has a corresponding matched instruction. BuildMIAction
should support not having one to cover the case where a rule produces more
instructions than it matched.

llvm-svn: 316463
2017-10-24 17:08:43 +00:00
Craig Topper 194cb74dec [TableGen] Fix some formatting quirks in the subtarget output file.
llvm-svn: 316450
2017-10-24 15:50:55 +00:00
Craig Topper df1285b8f4 [TableGen] Simplify some of the subtarget emission by removing code that avoids printing commas at the end of arrays and enums.
The C++ standard allows for trailing commas. We already do this in many other emitters.

llvm-svn: 316449
2017-10-24 15:50:53 +00:00
Daniel Sanders bd83ad4a87 [globalisel][tablegen] Remove unused InstructionMatcher's. NFC
llvm-svn: 316407
2017-10-24 01:48:34 +00:00
Daniel Sanders d66e0901ae [globalisel][tablegen] Import stores and allow GISel to automatically substitute zero regs like WZR/XZR/$zero.
This patch enables the import of stores. Unfortunately, doing so by itself,
loses an optimization where storing 0 to memory makes use of WZR/XZR.

To mitigate this, this patch also introduces a new feature that allows register
operands to nominate a zero register. When this is done, GlobalISel will
substitute (G_CONSTANT 0) with the nominated register automatically. This
is currently configured to only apply to the stores.

Applying it to GPR32/GPR64 register classes in general will be done after
review see (https://reviews.llvm.org/D39150).

llvm-svn: 316360
2017-10-23 18:19:24 +00:00
Craig Topper e06cc6d9ba [X86] Fix disassembler table generation to prevent instructions tagged with 'PS' being inherited into PD/XS/XD attribute entries.
llvm-svn: 316345
2017-10-23 16:49:26 +00:00
Craig Topper 64cb997ce1 [X86] Update a doxygen comment in the disassembler tablegen code. NFC
llvm-svn: 316309
2017-10-23 03:42:35 +00:00
Craig Topper 326008c615 [X86] Fix disassembly of EVEX rounding control and SAE instructions.
Fixes PR31955.

llvm-svn: 316308
2017-10-23 02:26:24 +00:00
Craig Topper dac20263a1 [X86] More correctly support LIG and WIG for EVEX instructions in the disassembler tables.
This is similar to how we generate the VEX tables.

More fixes are still needed for the instructions that use EVEX.b (broadcast and embedded rounding).

llvm-svn: 316294
2017-10-22 17:22:29 +00:00
Craig Topper e975127db6 [X86] Teach the disassembler that some instructions use VEX.W==0 without a corresponding VEX.W==1 instruction and we shouldn't treat them as if VEX.W is ignored.
Fixes PR11304.

llvm-svn: 316285
2017-10-22 06:18:26 +00:00
Craig Topper ca2382d809 [X86] Fix disassembling of EVEX instructions to stop accidentally decoding the SIB index register as an XMM/YMM/ZMM register.
This introduces a new operand type to encode the whether the index register should be XMM/YMM/ZMM. And new code to fixup the results created by readSIB.

This has the nice effect of removing a bunch of code that hard coded the name of every GATHER and SCATTER instruction to map the index type.

This fixes PR32807.

llvm-svn: 316273
2017-10-21 20:03:20 +00:00
Daniel Sanders 1e4569fdc1 [globalisel][tablegen] Fix small spelling nits. NFC
ComplexRendererFn -> ComplexRendererFns
Corrected a couple lingering references to tied operands that were missed.

llvm-svn: 316237
2017-10-20 20:55:29 +00:00
Reid Kleckner cfdd4a2699 Revert MSVC 2017 build fix and fix it by moving the method that implicitly instantiates addPredicate out of line
llvm-svn: 315932
2017-10-16 20:31:16 +00:00
Reid Kleckner 46e0918f83 Fix the build of GlobalISelEmitter with MSVC 2017 by specializing earlier
MSVC doesn't seem to like implicitly instantiating addPredicate and then
explicitly specializing it later. It causes an internal compiler error.

llvm-svn: 315930
2017-10-16 20:23:16 +00:00
Javed Absar dfd8e2ffd1 [TableGen] Simplify CallingConvEmitter.cpp. NFC.
llvm-svn: 315911
2017-10-16 14:52:26 +00:00
Javed Absar 776bf1de01 [TableGen] Range loopify DAGISelMatcher. NFC.
llvm-svn: 315891
2017-10-16 06:43:54 +00:00
Daniel Sanders ea8711b88e Re-commit r315885: [globalisel][tblgen] Add support for iPTR and implement am_unscaled* and am_indexed*
Summary:
iPTR is a pointer of subtarget-specific size to any address space. Therefore
type checks on this size derive the SizeInBits from a subtarget hook.

At this point, we can import the simplests G_LOAD rules and select load
instructions using them. Further patches will support for the predicates to
enable additional loads as well as the stores.

The previous commit failed on MSVC due to a failure to convert an
initializer_list to a std::vector. Hopefully, MSVC will accept this version.

Depends on D37457

Reviewers: ab, qcolombet, t.p.northover, rovka, aditya_nandakumar

Reviewed By: qcolombet

Subscribers: kristof.beyls, javed.absar, llvm-commits, igorb

Differential Revision: https://reviews.llvm.org/D37458

llvm-svn: 315887
2017-10-16 03:36:29 +00:00
Daniel Sanders ce72d611af Revert r315885: [globalisel][tblgen] Add support for iPTR and implement am_unscaled* and am_indexed*
MSVC doesn't like one of the constructors.

llvm-svn: 315886
2017-10-16 02:15:39 +00:00
Daniel Sanders 6735ea86cd [globalisel][tblgen] Add support for iPTR and implement am_unscaled* and am_indexed*
Summary:
iPTR is a pointer of subtarget-specific size to any address space. Therefore
type checks on this size derive the SizeInBits from a subtarget hook.

At this point, we can import the simplests G_LOAD rules and select load
instructions using them. Further patches will support for the predicates to
enable additional loads as well as the stores.

Depends on D37457

Reviewers: ab, qcolombet, t.p.northover, rovka, aditya_nandakumar

Reviewed By: qcolombet

Subscribers: kristof.beyls, javed.absar, llvm-commits, igorb

Differential Revision: https://reviews.llvm.org/D37458

llvm-svn: 315885
2017-10-16 01:16:35 +00:00
Daniel Sanders a71f454765 [globalisel][tablegen] Implement unindexed load, non-extending load, and MemVT checks
Summary:
This includes some context-sensitivity in the MVT to LLT conversion so that
pointer types are tested correctly.
FIXME: I'm not happy with the way this is done since everything is a
       special-case. I've yet to find a reasonable way to implement it.

select-load.mir fails because <1 x s64> loads in tablegen get priority over s64
loads. This is fixed in the next patch and as such they should be committed
together, I've posted them separately to help with the review.

Depends on D37456

Reviewers: ab, qcolombet, t.p.northover, rovka, aditya_nandakumar

Subscribers: kristof.beyls, javed.absar, llvm-commits, igorb

Differential Revision: https://reviews.llvm.org/D37457

llvm-svn: 315884
2017-10-16 00:56:30 +00:00
Daniel Sanders adbf58dc0b [tablegen] Use hasPredCode()/hasImmCode() instead of getPredCode().empty()/getImmCode().empty(). NFC
These are cheaper ways of testing for the presence of code than generating the C++ code and testing it's empty.

llvm-svn: 315872
2017-10-15 19:01:32 +00:00
Daniel Sanders df39cbae2f Re-commit r315863: [globalisel][tablegen] Import ComplexPattern when used as an operator
Summary:
It's possible for a ComplexPattern to be used as an operator in a match
pattern. This is used by the load/store patterns in AArch64 to name the
suboperands returned by ComplexPattern predicate so that they can be broken
apart and referenced independently in the result pattern.

This patch adds support for this in order to enable the import of load/store
patterns.

Depends on D37445

Hopefully fixed the ambiguous constructor that a large number of bots reported.

Reviewers: ab, qcolombet, t.p.northover, rovka, aditya_nandakumar

Reviewed By: qcolombet

Subscribers: aemerson, javed.absar, igorb, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D37456

llvm-svn: 315869
2017-10-15 18:22:54 +00:00
Daniel Sanders bb082a36d3 Revert r315863: [globalisel][tablegen] Import ComplexPattern when used as an operator
A large number of bots are failing on an ambiguous constructor call.

llvm-svn: 315866
2017-10-15 17:51:07 +00:00
Daniel Sanders b95b867dd8 [globalisel][tablegen] Import ComplexPattern when used as an operator
Summary:
It's possible for a ComplexPattern to be used as an operator in a match
pattern. This is used by the load/store patterns in AArch64 to name the
suboperands returned by ComplexPattern predicate so that they can be broken
apart and referenced independently in the result pattern.

This patch adds support for this in order to enable the import of load/store
patterns.

Depends on D37445

Reviewers: ab, qcolombet, t.p.northover, rovka, aditya_nandakumar

Reviewed By: qcolombet

Subscribers: aemerson, javed.absar, igorb, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D37456

llvm-svn: 315863
2017-10-15 17:03:36 +00:00
Krzysztof Parzyszek 4b3876f7a3 [TableGen] Remove error checks incorrectly failing on non-error conditions
In type inference, an empty type set for a specific hw mode is not an
error. In earlier stages of the design it was, but having to use non-
parameterized types with target intrinsics necessarily led to type
contradictions: since the intrinsics used specific types, they were
only valid for a specific hw mode, and the resulting type set for other
modes ended up empty. To accommodate the existence of such intrinsics
individual type sets were allowed to be empty as long as not all sets
were empty.

llvm-svn: 315858
2017-10-15 15:39:56 +00:00
Aaron Ballman 615eb47035 Reverting r315590; it did not include changes for llvm-tblgen, which is causing link errors for several people.
Error LNK2019 unresolved external symbol "public: void __cdecl `anonymous namespace'::MatchableInfo::dump(void)const " (?dump@MatchableInfo@?A0xf4f1c304@@QEBAXXZ) referenced in function "public: void __cdecl `anonymous namespace'::AsmMatcherEmitter::run(class llvm::raw_ostream &)" (?run@AsmMatcherEmitter@?A0xf4f1c304@@QEAAXAEAVraw_ostream@llvm@@@Z) llvm-tblgen D:\llvm\2017\utils\TableGen\AsmMatcherEmitter.obj 1

llvm-svn: 315854
2017-10-15 14:32:27 +00:00
Daniel Sanders 39690bdf42 [globalisel][tablegen] Map ld and st to G_LOAD and G_STORE. NFC
Summary:
There is an important mismatch between ISD::LOAD and G_LOAD (and likewise for
ISD::STORE and G_STORE). In SelectionDAG, ISD::LOAD is a non-atomic load
and atomic loads are handled by a separate node. However, this is not true of
GlobalISel's G_LOAD. For G_LOAD, the MachineMemOperand indicates the atomicity
of the operation. As a result, this mapping must also add a predicate that
checks for non-atomic MachineMemOperands.

This is NFC since these nodes always have predicates in practice and are
therefore always rejected at the moment.

Depends on D37443

Reviewers: ab, qcolombet, t.p.northover, rovka, aditya_nandakumar

Reviewed By: qcolombet

Subscribers: kristof.beyls, llvm-commits, igorb

Differential Revision: https://reviews.llvm.org/D37445

llvm-svn: 315843
2017-10-15 02:41:12 +00:00
Daniel Sanders 3f267bf769 [tablegen] Handle common load/store predicates inside tablegen. NFC.
Summary:
GlobalISel and SelectionDAG require different code for the common
load/store predicates due to differences in the representation.
For example:
   SelectionDAG: (load<signext,i8>:i32 GPR32:$addr) // The <> denote properties of the SDNode that are not printed in the DAG
   GlobalISel: (G_SEXT:s32 (G_LOAD:s8 GPR32:$addr))
Even without that, differences in the IR (SDNode vs MachineInstr) require
differences in the C++ predicate.

This patch moves the implementation of the common load/store predicates
into tablegen so that it can handle these differences.

It's NFC for SelectionDAG since it emits equivalent code and it's NFC for
GlobalISel since the rules involving the relevant predicates are still
rejected by the importer.

Depends on D36618

Reviewers: ab, qcolombet, t.p.northover, rovka, aditya_nandakumar

Subscribers: llvm-commits, igorb

Differential Revision: https://reviews.llvm.org/D37443

Includes a partial revert of r315826 since this patch makes it necessary for
getPredCode() to return a std::string and getImmCode() should have the same
interface as getPredCode().

llvm-svn: 315841
2017-10-15 02:06:44 +00:00
Simon Pilgrim 6ecae9fc97 [TableGen] Avoid unnecessary std::string creations
Avoid unnecessary std::string creations in the TreePredicateFn getters.

llvm-svn: 315826
2017-10-14 21:27:53 +00:00
Daniel Sanders bfa9e2cae7 [globalisel][tablegen] Simplify named operand/operator lookups and fix a wrong-code bug this revealed.
Summary:
Operand variable lookups are now performed by the RuleMatcher rather than
searching the whole matcher hierarchy for a match. This revealed a wrong-code
bug that currently affects ARM and X86 where patterns that use a variable more
than once in the match pattern will be imported but won't check that the
operands are identical. This can cause the tablegen-erated matcher to
accept matches that should be rejected.

Depends on D36569

Reviewers: ab, t.p.northover, qcolombet, rovka, aditya_nandakumar

Subscribers: aemerson, igorb, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D36618

llvm-svn: 315780
2017-10-14 00:31:58 +00:00
Daniel Sanders 2fed4ffa9e [globalisel][tablegen] Fix a use-after free bug that manifests on non-macOS machines.
llvm-svn: 315765
2017-10-13 21:51:20 +00:00
Daniel Sanders 11300cead8 [globalisel][tablegen] Add support for fpimm and import of APInt/APFloat based ImmLeaf.
Summary:
There's only a tablegen testcase for IntImmLeaf and not a CodeGen one
because the relevant rules are rejected for other reasons at the moment.
On AArch64, it's because there's an SDNodeXForm attached to the operand.
On X86, it's because the rule either emits multiple instructions or has
another predicate using PatFrag which cannot easily be supported at the
same time.

Reviewers: ab, t.p.northover, qcolombet, rovka, aditya_nandakumar

Reviewed By: qcolombet

Subscribers: aemerson, javed.absar, igorb, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D36569

llvm-svn: 315761
2017-10-13 21:28:03 +00:00
Daniel Sanders 649c585710 [aarch64] Support APInt and APFloat in ImmLeaf subclasses and make AArch64 use them.
Summary:
The purpose of this patch is to expose more information about ImmLeaf-like
PatLeaf's so that GlobalISel can learn to import them. Previously, ImmLeaf
could only be used to test int64_t's produced by sign-extending an APInt.
Other tests on immediates had to use the generic PatLeaf and extract the
constant using C++.

With this patch, tablegen will know how to generate predicates for APInt,
and APFloat. This will allow it to 'do the right thing' for both SelectionDAG
and GlobalISel which require different methods of extracting the immediate
from the IR.

This is NFC for SelectionDAG since the new code is equivalent to the
previous code. It's also NFC for FastISel because FastIselShouldIgnore is 1
for the ImmLeaf subclasses. Enabling FastIselShouldIgnore == 0 for these new
subclasses will require a significant re-factor of FastISel.

For GlobalISel, it's currently NFC because the relevant code to import the
affected rules is not yet present. This will be added in a later patch.

Depends on D36086

Reviewers: ab, t.p.northover, qcolombet, rovka, aditya_nandakumar

Reviewed By: qcolombet

Subscribers: bjope, aemerson, rengolin, javed.absar, igorb, llvm-commits, kristof.beyls

Differential Revision: https://reviews.llvm.org/D36534

llvm-svn: 315747
2017-10-13 20:42:18 +00:00
Daniel Sanders 9e0ae7bd8b Revert r315148 [TableGen] Avoid unnecessary std::string creations
I'm about to commit a patch that makes them necessary for getPredCode() and
it would be strange for getPredCode() and getImmCode() to require different
usage.

llvm-svn: 315733
2017-10-13 19:00:01 +00:00
Javed Absar 08fb414b21 [TableGen] : Simplify RegisterInfoEmitter
llvm-svn: 315700
2017-10-13 16:27:57 +00:00
Artem Belevich 786ca6a166 [TableGen] Allow intrinsics to have up to 8 return values.
Differential Revision: https://reviews.llvm.org/D38633

llvm-svn: 315598
2017-10-12 17:40:00 +00:00
Don Hinton 3e0199f7eb [dump] Remove NDEBUG from test to enable dump methods [NFC]
Summary:
Add LLVM_FORCE_ENABLE_DUMP cmake option, and use it along with
LLVM_ENABLE_ASSERTIONS to set LLVM_ENABLE_DUMP.

Remove NDEBUG and only use LLVM_ENABLE_DUMP to enable dump methods.

Move definition of LLVM_ENABLE_DUMP from config.h to llvm-config.h so
it'll be picked up by public headers.

Differential Revision: https://reviews.llvm.org/D38406

llvm-svn: 315590
2017-10-12 16:16:06 +00:00
Oliver Stannard dab5212884 [AsmParser] Suppress compile warning for targets with no register diags
This fixes the "switch statement contains 'default' but no 'case' labels"
warnings in table-generated code introduced in r315295.

llvm-svn: 315571
2017-10-12 09:28:23 +00:00
Zachary Turner 41a9ee98f9 Revert "[ADT] Make Twine's copy constructor private."
This reverts commit 4e4ee1c507e2707bb3c208e1e1b6551c3015cbf5.

This is failing due to some code that isn't built on MSVC
so I didn't catch.  Not immediately obvious how to fix this
at first glance, so I'm reverting for now.

llvm-svn: 315536
2017-10-11 23:54:34 +00:00