maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

1912 lines
65 KiB
C++
Raw Normal View History

AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
//=- AArch64LoadStoreOptimizer.cpp - AArch64 load/store opt. pass -*- C++ -*-=//
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
//
// The LLVM Compiler Infrastructure
//
// This file is distributed under the University of Illinois Open Source
// License. See LICENSE.TXT for details.
//
//===----------------------------------------------------------------------===//
//
// This file contains a pass that performs load / store related peephole
// optimizations. This pass should be run after register allocation.
//
//===----------------------------------------------------------------------===//
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
#include "AArch64InstrInfo.h"
Remove the TargetMachine forwards for TargetSubtargetInfo based information and update all callers. No functional change. llvm-svn: 214781
2014-08-05 05:25:23 +08:00
#include "AArch64Subtarget.h"
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
#include "MCTargetDesc/AArch64AddressingModes.h"
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
#include "llvm/ADT/BitVector.h"
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
#include "llvm/ADT/SmallVector.h"
Run sort_includes.py on the AArch64 backend. No functionality change. llvm-svn: 213938
2014-07-25 19:42:14 +08:00
#include "llvm/ADT/Statistic.h"
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
#include "llvm/CodeGen/MachineBasicBlock.h"
#include "llvm/CodeGen/MachineFunctionPass.h"
#include "llvm/CodeGen/MachineInstr.h"
#include "llvm/CodeGen/MachineInstrBuilder.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Debug.h"
#include "llvm/Support/ErrorHandling.h"
#include "llvm/Support/raw_ostream.h"
Run sort_includes.py on the AArch64 backend. No functionality change. llvm-svn: 213938
2014-07-25 19:42:14 +08:00
#include "llvm/Target/TargetInstrInfo.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Target/TargetRegisterInfo.h"
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
using namespace llvm;
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
#define DEBUG_TYPE "aarch64-ldst-opt"
[Modules] Fix potential ODR violations by sinking the DEBUG_TYPE definition below all of the header #include lines, lib/Target/... edition. llvm-svn: 206842
2014-04-22 10:41:26 +08:00
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
STATISTIC(NumPairCreated, "Number of load/store pair instructions generated");
STATISTIC(NumPostFolded, "Number of post-index updates folded");
STATISTIC(NumPreFolded, "Number of pre-index updates folded");
STATISTIC(NumUnscaledPairCreated,
"Number of load/store from unscaled generated");
[AArch64] Refactoring aarch64-ldst-opt. NCF. Summary : * Rename isSmallTypeLdMerge() to isNarrowLoad(). * Rename NumSmallTypeMerged to NumNarrowTypePromoted. * Use Subtarget defined as a member variable. llvm-svn: 253587
2015-11-20 02:41:27 +08:00
STATISTIC(NumNarrowLoadsPromoted, "Number of narrow loads promoted");
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
STATISTIC(NumZeroStoresPromoted, "Number of narrow zero stores promoted");
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
STATISTIC(NumLoadsFromStoresPromoted, "Number of loads from stores promoted");
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
// The LdStLimit limits how far we search for load/store pairs.
static cl::opt<unsigned> LdStLimit("aarch64-load-store-scan-limit",
[AArch64] clang-format the load/store optimizer. No change in functionality. llvm-svn: 210182
2014-06-04 20:40:35 +08:00
cl::init(20), cl::Hidden);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
// The UpdateLimit limits how far we search for update instructions when we form
// pre-/post-index instructions.
static cl::opt<unsigned> UpdateLimit("aarch64-update-scan-limit", cl::init(100),
cl::Hidden);
[AArch64] Decouple zero store promotion from narrow ld merge. NFC. Summary: This change refactors to decouple the zero store promotion from the narrow ld merge and add a flag (enable-narrow-ld-merge=true) to control the narrow ld merge optimization. Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D19885 llvm-svn: 268744
2016-05-06 23:08:57 +08:00
static cl::opt<bool> EnableNarrowLdMerge("enable-narrow-ld-merge", cl::Hidden,
[AArch64] Disable narrow load merge by default Summary: As this optimization converts two loads into one load with two shift instructions, it could potentially hurt performance if a loop is arithmetic operation intensive. Reviewers: t.p.northover, mcrosier, jmolloy Subscribers: evandro, jmolloy, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D20172 llvm-svn: 270251
2016-05-21 02:45:49 +08:00
cl::init(false),
[AArch64] Decouple zero store promotion from narrow ld merge. NFC. Summary: This change refactors to decouple the zero store promotion from the narrow ld merge and add a flag (enable-narrow-ld-merge=true) to control the narrow ld merge optimization. Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D19885 llvm-svn: 268744
2016-05-06 23:08:57 +08:00
cl::desc("Enable narrow load merge"));
[AArch64] Register (existing) AArch64LoadStoreOpt pass with LLVM pass manager. Summary: Among other things, this allows -print-after-all/-print-before-all to dump IR around this pass. This is the AArch64 version of r243052. llvm-svn: 244041
2015-08-05 21:44:51 +08:00
#define AARCH64_LOAD_STORE_OPT_NAME "AArch64 load / store optimization pass"
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
namespace {
[AArch64] Simplify the passing of arguments. NFC. This is setup for future work planned for the AArch64 Load/Store Opt pass. llvm-svn: 242810
2015-07-22 01:42:04 +08:00
typedef struct LdStPairFlags {
// If a matching instruction is found, MergeForward is set to true if the
// merge is to remove the first instruction and replace the second with
// a pair-wise insn, and false if the reverse is true.
bool MergeForward;
// SExtIdx gives the index of the result of the load pair that must be
// extended. The value of SExtIdx assumes that the paired load produces the
// value in this order: (I, returned iterator), i.e., -1 means no value has
// to be extended, 0 means I, and 1 means the returned iterator.
int SExtIdx;
LdStPairFlags() : MergeForward(false), SExtIdx(-1) {}
void setMergeForward(bool V = true) { MergeForward = V; }
bool getMergeForward() const { return MergeForward; }
void setSExtIdx(int V) { SExtIdx = V; }
int getSExtIdx() const { return SExtIdx; }
} LdStPairFlags;
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
struct AArch64LoadStoreOpt : public MachineFunctionPass {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
static char ID;
[AArch64]Enable the narrow ld promotion only on profitable microarchitectures The benefit from converting narrow loads into a wider load (r251438) could be micro-architecturally dependent, as it assumes that a single load with two bitfield extracts is cheaper than two narrow loads. Currently, this conversion is enabled only in cortex-a57 on which performance benefits were verified. llvm-svn: 252316
2015-11-07 00:27:47 +08:00
AArch64LoadStoreOpt() : MachineFunctionPass(ID) {
[AArch64] Register (existing) AArch64LoadStoreOpt pass with LLVM pass manager. Summary: Among other things, this allows -print-after-all/-print-before-all to dump IR around this pass. This is the AArch64 version of r243052. llvm-svn: 244041
2015-08-05 21:44:51 +08:00
initializeAArch64LoadStoreOptPass(*PassRegistry::getPassRegistry());
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
const AArch64InstrInfo *TII;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
const TargetRegisterInfo *TRI;
[AArch64] Fix halfword load merging for big-endian targets For big-endian targets, when we merge two halfword loads into a word load, the order of the halfwords in the loaded value is reversed compared to little-endian, so the load-store optimiser needs to swap the destination registers. This does not affect merging of two word loads, as we use ldp, which treats the memory as two separate 32-bit words. llvm-svn: 252597
2015-11-10 19:04:18 +08:00
const AArch64Subtarget *Subtarget;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Allocate the modified and used regs only once per function. llvm-svn: 259510
2016-02-02 23:02:30 +08:00
// Track which registers have been modified and used.
BitVector ModifiedRegs, UsedRegs;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Scan the instructions looking for a load/store that can be combined
// with the current instruction into a load/store pair.
// Return the matching instruction if one is found, else MBB->end().
MachineBasicBlock::iterator findMatchingInsn(MachineBasicBlock::iterator I,
[AArch64] Simplify the passing of arguments. NFC. This is setup for future work planned for the AArch64 Load/Store Opt pass. llvm-svn: 242810
2015-07-22 01:42:04 +08:00
LdStPairFlags &Flags,
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
unsigned Limit,
bool FindNarrowMerge);
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
// Scan the instructions looking for a store that writes to the address from
// which the current load instruction reads. Return true if one is found.
bool findMatchingStore(MachineBasicBlock::iterator I, unsigned Limit,
MachineBasicBlock::iterator &StoreI);
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
// Merge the two instructions indicated into a wider instruction.
MachineBasicBlock::iterator
mergeNarrowInsns(MachineBasicBlock::iterator I,
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MachineBasicBlock::iterator MergeMI,
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
const LdStPairFlags &Flags);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Merge the two instructions indicated into a single pair-wise instruction.
MachineBasicBlock::iterator
mergePairedInsns(MachineBasicBlock::iterator I,
[AArch64] Simplify the passing of arguments. NFC. This is setup for future work planned for the AArch64 Load/Store Opt pass. llvm-svn: 242810
2015-07-22 01:42:04 +08:00
MachineBasicBlock::iterator Paired,
Follow up to r242810. NFC. llvm-svn: 242812
2015-07-22 01:47:56 +08:00
const LdStPairFlags &Flags);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
// Promote the load that reads directly from the address stored to.
MachineBasicBlock::iterator
promoteLoadFromStore(MachineBasicBlock::iterator LoadI,
MachineBasicBlock::iterator StoreI);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Scan the instruction list to find a base register update that can
// be combined with the current instruction (a load or store) using
// pre or post indexed addressing with writeback. Scan forwards.
MachineBasicBlock::iterator
[AArch64] Remove unused arguments. NFC. AFAICT, these have been unused since the initial backend import. llvm-svn: 258093
2016-01-19 05:56:40 +08:00
findMatchingUpdateInsnForward(MachineBasicBlock::iterator I,
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
int UnscaledOffset, unsigned Limit);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Scan the instruction list to find a base register update that can
// be combined with the current instruction (a load or store) using
// pre or post indexed addressing with writeback. Scan backwards.
MachineBasicBlock::iterator
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
findMatchingUpdateInsnBackward(MachineBasicBlock::iterator I, unsigned Limit);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// Find an instruction that updates the base register of the ld/st
// instruction.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
bool isMatchingUpdateInsn(MachineInstr &MemMI, MachineInstr &MI,
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
unsigned BaseReg, int Offset);
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
// Merge a pre- or post-index base register update into a ld/st instruction.
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
mergeUpdateInsn(MachineBasicBlock::iterator I,
MachineBasicBlock::iterator Update, bool IsPreIdx);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
// Find and merge foldable ldr/str instructions.
bool tryToMergeLdStInst(MachineBasicBlock::iterator &MBBI);
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
// Find and pair ldr/str instructions.
bool tryToPairLdStInst(MachineBasicBlock::iterator &MBBI);
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
// Find and promote load instructions which read directly from store.
bool tryToPromoteLoadFromStore(MachineBasicBlock::iterator &MBBI);
[AArch64]Enable the narrow ld promotion only on profitable microarchitectures The benefit from converting narrow loads into a wider load (r251438) could be micro-architecturally dependent, as it assumes that a single load with two bitfield extracts is cheaper than two narrow loads. Currently, this conversion is enabled only in cortex-a57 on which performance benefits were verified. llvm-svn: 252316
2015-11-07 00:27:47 +08:00
bool optimizeBlock(MachineBasicBlock &MBB, bool enableNarrowLdOpt);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[C++11] Add 'override' keywords and remove 'virtual'. Additionally add 'final' and leave 'virtual' on some methods that are marked virtual without overriding anything and have no obvious overrides themselves. ARM64 edition llvm-svn: 207509
2014-04-29 15:58:25 +08:00
bool runOnMachineFunction(MachineFunction &Fn) override;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
Add MachineFunctionProperty checks for AllVRegsAllocated for target passes Summary: This adds the same checks that were added in r264593 to all target-specific passes that run after register allocation. Reviewers: qcolombet Subscribers: jyknight, dsanders, llvm-commits Differential Revision: http://reviews.llvm.org/D18525 llvm-svn: 265313
2016-04-05 01:09:25 +08:00
MachineFunctionProperties getRequiredProperties() const override {
return MachineFunctionProperties().set(
MachineFunctionProperties::Property::AllVRegsAllocated);
}
[C++11] Add 'override' keywords and remove 'virtual'. Additionally add 'final' and leave 'virtual' on some methods that are marked virtual without overriding anything and have no obvious overrides themselves. ARM64 edition llvm-svn: 207509
2014-04-29 15:58:25 +08:00
const char *getPassName() const override {
[AArch64] Register (existing) AArch64LoadStoreOpt pass with LLVM pass manager. Summary: Among other things, this allows -print-after-all/-print-before-all to dump IR around this pass. This is the AArch64 version of r243052. llvm-svn: 244041
2015-08-05 21:44:51 +08:00
return AARCH64_LOAD_STORE_OPT_NAME;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
};
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
char AArch64LoadStoreOpt::ID = 0;
Add missing closing namespace comment. llvm-svn: 215402
2014-08-12 06:42:31 +08:00
} // namespace
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Register (existing) AArch64LoadStoreOpt pass with LLVM pass manager. Summary: Among other things, this allows -print-after-all/-print-before-all to dump IR around this pass. This is the AArch64 version of r243052. llvm-svn: 244041
2015-08-05 21:44:51 +08:00
INITIALIZE_PASS(AArch64LoadStoreOpt, "aarch64-ldst-opt",
AARCH64_LOAD_STORE_OPT_NAME, false, false)
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
static unsigned getBitExtrOpcode(MachineInstr &MI) {
switch (MI.getOpcode()) {
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
default:
llvm_unreachable("Unexpected opcode.");
case AArch64::LDRBBui:
case AArch64::LDURBBi:
case AArch64::LDRHHui:
case AArch64::LDURHHi:
return AArch64::UBFMWri;
case AArch64::LDRSBWui:
case AArch64::LDURSBWi:
case AArch64::LDRSHWui:
case AArch64::LDURSHWi:
return AArch64::SBFMWri;
}
}
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
static bool isNarrowStore(unsigned Opc) {
switch (Opc) {
default:
return false;
case AArch64::STRBBui:
case AArch64::STURBBi:
case AArch64::STRHHui:
case AArch64::STURHHi:
return true;
}
}
[AArch64] Refactoring aarch64-ldst-opt. NCF. Summary : * Rename isSmallTypeLdMerge() to isNarrowLoad(). * Rename NumSmallTypeMerged to NumNarrowTypePromoted. * Use Subtarget defined as a member variable. llvm-svn: 253587
2015-11-20 02:41:27 +08:00
static bool isNarrowLoad(unsigned Opc) {
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
switch (Opc) {
default:
return false;
case AArch64::LDRHHui:
case AArch64::LDURHHi:
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
case AArch64::LDRBBui:
case AArch64::LDURBBi:
case AArch64::LDRSHWui:
case AArch64::LDURSHWi:
case AArch64::LDRSBWui:
case AArch64::LDURSBWi:
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
return true;
}
}
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
static bool isNarrowLoad(MachineInstr &MI) {
return isNarrowLoad(MI.getOpcode());
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
}
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
static bool isNarrowLoadOrStore(unsigned Opc) {
return isNarrowLoad(Opc) || isNarrowStore(Opc);
}
[AArch64] Scale offsets by the size of the memory operation. NFC. The immediate in the load/store should be scaled by the size of the memory operation, not the size of the register being loaded/stored. This change gets us one step closer to forming LDPSW instructions. This change also enables pre- and post-indexing for halfword and byte loads and stores. llvm-svn: 248804
2015-09-30 00:07:32 +08:00
// Scaling factor for unscaled load or store.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
static int getMemScale(MachineInstr &MI) {
switch (MI.getOpcode()) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
default:
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
llvm_unreachable("Opcode has unknown scale!");
case AArch64::LDRBBui:
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
case AArch64::LDURBBi:
case AArch64::LDRSBWui:
case AArch64::LDURSBWi:
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::STRBBui:
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
case AArch64::STURBBi:
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
return 1;
case AArch64::LDRHHui:
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
case AArch64::LDURHHi:
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
case AArch64::LDRSHWui:
case AArch64::LDURSHWi:
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::STRHHui:
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
case AArch64::STURHHi:
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
return 2;
[AArch64] Remove some redundant cases. NFC. llvm-svn: 248800
2015-09-29 22:57:10 +08:00
case AArch64::LDRSui:
case AArch64::LDURSi:
case AArch64::LDRSWui:
case AArch64::LDURSWi:
case AArch64::LDRWui:
case AArch64::LDURWi:
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::STRSui:
case AArch64::STURSi:
case AArch64::STRWui:
case AArch64::STURWi:
[AArch64] Scale offsets by the size of the memory operation. NFC. The immediate in the load/store should be scaled by the size of the memory operation, not the size of the register being loaded/stored. This change gets us one step closer to forming LDPSW instructions. This change also enables pre- and post-indexing for halfword and byte loads and stores. llvm-svn: 248804
2015-09-30 00:07:32 +08:00
case AArch64::LDPSi:
[AArch64] Add support for pre- and post-index LDPSWs. llvm-svn: 248825
2015-09-30 04:39:55 +08:00
case AArch64::LDPSWi:
[AArch64] Scale offsets by the size of the memory operation. NFC. The immediate in the load/store should be scaled by the size of the memory operation, not the size of the register being loaded/stored. This change gets us one step closer to forming LDPSW instructions. This change also enables pre- and post-indexing for halfword and byte loads and stores. llvm-svn: 248804
2015-09-30 00:07:32 +08:00
case AArch64::LDPWi:
case AArch64::STPSi:
case AArch64::STPWi:
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return 4;
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::LDRDui:
case AArch64::LDURDi:
[AArch64] Remove some redundant cases. NFC. llvm-svn: 248800
2015-09-29 22:57:10 +08:00
case AArch64::LDRXui:
case AArch64::LDURXi:
case AArch64::STRDui:
case AArch64::STURDi:
case AArch64::STRXui:
case AArch64::STURXi:
[AArch64] Scale offsets by the size of the memory operation. NFC. The immediate in the load/store should be scaled by the size of the memory operation, not the size of the register being loaded/stored. This change gets us one step closer to forming LDPSW instructions. This change also enables pre- and post-indexing for halfword and byte loads and stores. llvm-svn: 248804
2015-09-30 00:07:32 +08:00
case AArch64::LDPDi:
case AArch64::LDPXi:
case AArch64::STPDi:
case AArch64::STPXi:
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return 8;
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::LDRQui:
case AArch64::LDURQi:
[AArch64] Remove some redundant cases. NFC. llvm-svn: 248800
2015-09-29 22:57:10 +08:00
case AArch64::STRQui:
case AArch64::STURQi:
[AArch64] Scale offsets by the size of the memory operation. NFC. The immediate in the load/store should be scaled by the size of the memory operation, not the size of the register being loaded/stored. This change gets us one step closer to forming LDPSW instructions. This change also enables pre- and post-indexing for halfword and byte loads and stores. llvm-svn: 248804
2015-09-30 00:07:32 +08:00
case AArch64::LDPQi:
case AArch64::STPQi:
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return 16;
}
}
[AArch64][LoadStoreOptimizer] Generate LDP + SXTW instead of LD[U]R + LD[U]RSW. Teach the load store optimizer how to sign extend a result of a load pair when it helps creating more pairs. The rational is that loads are more expensive than sign extensions, so if we gather some in one instruction this is better! <rdar://problem/20072968> llvm-svn: 231527
2015-03-07 06:42:10 +08:00
static unsigned getMatchingNonSExtOpcode(unsigned Opc,
bool *IsValidLdStrOpc = nullptr) {
if (IsValidLdStrOpc)
*IsValidLdStrOpc = true;
switch (Opc) {
default:
if (IsValidLdStrOpc)
*IsValidLdStrOpc = false;
return UINT_MAX;
case AArch64::STRDui:
case AArch64::STURDi:
case AArch64::STRQui:
case AArch64::STURQi:
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
case AArch64::STRBBui:
case AArch64::STURBBi:
case AArch64::STRHHui:
case AArch64::STURHHi:
[AArch64][LoadStoreOptimizer] Generate LDP + SXTW instead of LD[U]R + LD[U]RSW. Teach the load store optimizer how to sign extend a result of a load pair when it helps creating more pairs. The rational is that loads are more expensive than sign extensions, so if we gather some in one instruction this is better! <rdar://problem/20072968> llvm-svn: 231527
2015-03-07 06:42:10 +08:00
case AArch64::STRWui:
case AArch64::STURWi:
case AArch64::STRXui:
case AArch64::STURXi:
case AArch64::LDRDui:
case AArch64::LDURDi:
case AArch64::LDRQui:
case AArch64::LDURQi:
case AArch64::LDRWui:
case AArch64::LDURWi:
case AArch64::LDRXui:
case AArch64::LDURXi:
case AArch64::STRSui:
case AArch64::STURSi:
case AArch64::LDRSui:
case AArch64::LDURSi:
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
case AArch64::LDRHHui:
case AArch64::LDURHHi:
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
case AArch64::LDRBBui:
case AArch64::LDURBBi:
[AArch64][LoadStoreOptimizer] Generate LDP + SXTW instead of LD[U]R + LD[U]RSW. Teach the load store optimizer how to sign extend a result of a load pair when it helps creating more pairs. The rational is that loads are more expensive than sign extensions, so if we gather some in one instruction this is better! <rdar://problem/20072968> llvm-svn: 231527
2015-03-07 06:42:10 +08:00
return Opc;
case AArch64::LDRSWui:
return AArch64::LDRWui;
case AArch64::LDURSWi:
return AArch64::LDURWi;
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
case AArch64::LDRSBWui:
return AArch64::LDRBBui;
case AArch64::LDRSHWui:
return AArch64::LDRHHui;
case AArch64::LDURSBWi:
return AArch64::LDURBBi;
case AArch64::LDURSHWi:
return AArch64::LDURHHi;
[AArch64][LoadStoreOptimizer] Generate LDP + SXTW instead of LD[U]R + LD[U]RSW. Teach the load store optimizer how to sign extend a result of a load pair when it helps creating more pairs. The rational is that loads are more expensive than sign extensions, so if we gather some in one instruction this is better! <rdar://problem/20072968> llvm-svn: 231527
2015-03-07 06:42:10 +08:00
}
}
[AArch64] Refactoring aarch64-ldst-opt. NCF. Remove narrow load / store instructions from getMatchingPairOpcode(), and add getMatchingWideOpcode(). llvm-svn: 259914
2016-02-06 04:02:03 +08:00
static unsigned getMatchingWideOpcode(unsigned Opc) {
switch (Opc) {
default:
llvm_unreachable("Opcode has no wide equivalent!");
case AArch64::STRBBui:
return AArch64::STRHHui;
case AArch64::STRHHui:
return AArch64::STRWui;
case AArch64::STURBBi:
return AArch64::STURHHi;
case AArch64::STURHHi:
return AArch64::STURWi;
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
case AArch64::STURWi:
return AArch64::STURXi;
case AArch64::STRWui:
return AArch64::STRXui;
[AArch64] Refactoring aarch64-ldst-opt. NCF. Remove narrow load / store instructions from getMatchingPairOpcode(), and add getMatchingWideOpcode(). llvm-svn: 259914
2016-02-06 04:02:03 +08:00
case AArch64::LDRHHui:
case AArch64::LDRSHWui:
return AArch64::LDRWui;
case AArch64::LDURHHi:
case AArch64::LDURSHWi:
return AArch64::LDURWi;
case AArch64::LDRBBui:
case AArch64::LDRSBWui:
return AArch64::LDRHHui;
case AArch64::LDURBBi:
case AArch64::LDURSBWi:
return AArch64::LDURHHi;
}
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
static unsigned getMatchingPairOpcode(unsigned Opc) {
switch (Opc) {
default:
llvm_unreachable("Opcode has no pairwise equivalent!");
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::STRSui:
case AArch64::STURSi:
return AArch64::STPSi;
case AArch64::STRDui:
case AArch64::STURDi:
return AArch64::STPDi;
case AArch64::STRQui:
case AArch64::STURQi:
return AArch64::STPQi;
case AArch64::STRWui:
case AArch64::STURWi:
return AArch64::STPWi;
case AArch64::STRXui:
case AArch64::STURXi:
return AArch64::STPXi;
case AArch64::LDRSui:
case AArch64::LDURSi:
return AArch64::LDPSi;
case AArch64::LDRDui:
case AArch64::LDURDi:
return AArch64::LDPDi;
case AArch64::LDRQui:
case AArch64::LDURQi:
return AArch64::LDPQi;
case AArch64::LDRWui:
case AArch64::LDURWi:
return AArch64::LDPWi;
case AArch64::LDRXui:
case AArch64::LDURXi:
return AArch64::LDPXi;
[AArch64][LoadStoreOptimizer] Form LDPSW when possible. This patch adds the missing LD[U]RSW variants to the load store optimizer, so that we generate LDPSW when possible. <rdar://problem/19583480> llvm-svn: 226978
2015-01-24 09:25:54 +08:00
case AArch64::LDRSWui:
case AArch64::LDURSWi:
return AArch64::LDPSWi;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
}
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
static unsigned isMatchingStore(MachineInstr &LoadInst,
MachineInstr &StoreInst) {
unsigned LdOpc = LoadInst.getOpcode();
unsigned StOpc = StoreInst.getOpcode();
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
switch (LdOpc) {
default:
llvm_unreachable("Unsupported load instruction!");
case AArch64::LDRBBui:
return StOpc == AArch64::STRBBui || StOpc == AArch64::STRHHui ||
StOpc == AArch64::STRWui || StOpc == AArch64::STRXui;
case AArch64::LDURBBi:
return StOpc == AArch64::STURBBi || StOpc == AArch64::STURHHi ||
StOpc == AArch64::STURWi || StOpc == AArch64::STURXi;
case AArch64::LDRHHui:
return StOpc == AArch64::STRHHui || StOpc == AArch64::STRWui ||
StOpc == AArch64::STRXui;
case AArch64::LDURHHi:
return StOpc == AArch64::STURHHi || StOpc == AArch64::STURWi ||
StOpc == AArch64::STURXi;
case AArch64::LDRWui:
return StOpc == AArch64::STRWui || StOpc == AArch64::STRXui;
case AArch64::LDURWi:
return StOpc == AArch64::STURWi || StOpc == AArch64::STURXi;
case AArch64::LDRXui:
return StOpc == AArch64::STRXui;
case AArch64::LDURXi:
return StOpc == AArch64::STURXi;
}
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
static unsigned getPreIndexedOpcode(unsigned Opc) {
switch (Opc) {
default:
llvm_unreachable("Opcode has no pre-indexed equivalent!");
[AArch64] clang-format the load/store optimizer. No change in functionality. llvm-svn: 210182
2014-06-04 20:40:35 +08:00
case AArch64::STRSui:
return AArch64::STRSpre;
case AArch64::STRDui:
return AArch64::STRDpre;
case AArch64::STRQui:
return AArch64::STRQpre;
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::STRBBui:
return AArch64::STRBBpre;
case AArch64::STRHHui:
return AArch64::STRHHpre;
[AArch64] clang-format the load/store optimizer. No change in functionality. llvm-svn: 210182
2014-06-04 20:40:35 +08:00
case AArch64::STRWui:
return AArch64::STRWpre;
case AArch64::STRXui:
return AArch64::STRXpre;
case AArch64::LDRSui:
return AArch64::LDRSpre;
case AArch64::LDRDui:
return AArch64::LDRDpre;
case AArch64::LDRQui:
return AArch64::LDRQpre;
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::LDRBBui:
return AArch64::LDRBBpre;
case AArch64::LDRHHui:
return AArch64::LDRHHpre;
[AArch64] clang-format the load/store optimizer. No change in functionality. llvm-svn: 210182
2014-06-04 20:40:35 +08:00
case AArch64::LDRWui:
return AArch64::LDRWpre;
case AArch64::LDRXui:
return AArch64::LDRXpre;
[AArch64][LoadStoreOptimizer] Form LDPSW when possible. This patch adds the missing LD[U]RSW variants to the load store optimizer, so that we generate LDPSW when possible. <rdar://problem/19583480> llvm-svn: 226978
2015-01-24 09:25:54 +08:00
case AArch64::LDRSWui:
return AArch64::LDRSWpre;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
case AArch64::LDPSi:
return AArch64::LDPSpre;
[AArch64] Add support for pre- and post-index LDPSWs. llvm-svn: 248825
2015-09-30 04:39:55 +08:00
case AArch64::LDPSWi:
return AArch64::LDPSWpre;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
case AArch64::LDPDi:
return AArch64::LDPDpre;
case AArch64::LDPQi:
return AArch64::LDPQpre;
case AArch64::LDPWi:
return AArch64::LDPWpre;
case AArch64::LDPXi:
return AArch64::LDPXpre;
case AArch64::STPSi:
return AArch64::STPSpre;
case AArch64::STPDi:
return AArch64::STPDpre;
case AArch64::STPQi:
return AArch64::STPQpre;
case AArch64::STPWi:
return AArch64::STPWpre;
case AArch64::STPXi:
return AArch64::STPXpre;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
}
static unsigned getPostIndexedOpcode(unsigned Opc) {
switch (Opc) {
default:
llvm_unreachable("Opcode has no post-indexed wise equivalent!");
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::STRSui:
return AArch64::STRSpost;
case AArch64::STRDui:
return AArch64::STRDpost;
case AArch64::STRQui:
return AArch64::STRQpost;
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::STRBBui:
return AArch64::STRBBpost;
case AArch64::STRHHui:
return AArch64::STRHHpost;
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::STRWui:
return AArch64::STRWpost;
case AArch64::STRXui:
return AArch64::STRXpost;
case AArch64::LDRSui:
return AArch64::LDRSpost;
case AArch64::LDRDui:
return AArch64::LDRDpost;
case AArch64::LDRQui:
return AArch64::LDRQpost;
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::LDRBBui:
return AArch64::LDRBBpost;
case AArch64::LDRHHui:
return AArch64::LDRHHpost;
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::LDRWui:
return AArch64::LDRWpost;
case AArch64::LDRXui:
return AArch64::LDRXpost;
[AArch64][LoadStoreOptimizer] Form LDPSW when possible. This patch adds the missing LD[U]RSW variants to the load store optimizer, so that we generate LDPSW when possible. <rdar://problem/19583480> llvm-svn: 226978
2015-01-24 09:25:54 +08:00
case AArch64::LDRSWui:
return AArch64::LDRSWpost;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
case AArch64::LDPSi:
return AArch64::LDPSpost;
[AArch64] Add support for pre- and post-index LDPSWs. llvm-svn: 248825
2015-09-30 04:39:55 +08:00
case AArch64::LDPSWi:
return AArch64::LDPSWpost;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
case AArch64::LDPDi:
return AArch64::LDPDpost;
case AArch64::LDPQi:
return AArch64::LDPQpost;
case AArch64::LDPWi:
return AArch64::LDPWpost;
case AArch64::LDPXi:
return AArch64::LDPXpost;
case AArch64::STPSi:
return AArch64::STPSpost;
case AArch64::STPDi:
return AArch64::STPDpost;
case AArch64::STPQi:
return AArch64::STPQpost;
case AArch64::STPWi:
return AArch64::STPWpost;
case AArch64::STPXi:
return AArch64::STPXpost;
}
}
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
static bool isPairedLdSt(const MachineInstr &MI) {
switch (MI.getOpcode()) {
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
default:
return false;
case AArch64::LDPSi:
[AArch64] Add support for pre- and post-index LDPSWs. llvm-svn: 248825
2015-09-30 04:39:55 +08:00
case AArch64::LDPSWi:
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
case AArch64::LDPDi:
case AArch64::LDPQi:
case AArch64::LDPWi:
case AArch64::LDPXi:
case AArch64::STPSi:
case AArch64::STPDi:
case AArch64::STPQi:
case AArch64::STPWi:
case AArch64::STPXi:
return true;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
}
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
static const MachineOperand &getLdStRegOp(const MachineInstr &MI,
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
unsigned PairedRegOp = 0) {
assert(PairedRegOp < 2 && "Unexpected register operand idx.");
unsigned Idx = isPairedLdSt(MI) ? PairedRegOp : 0;
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
return MI.getOperand(Idx);
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
}
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
static const MachineOperand &getLdStBaseOp(const MachineInstr &MI) {
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
unsigned Idx = isPairedLdSt(MI) ? 2 : 1;
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
return MI.getOperand(Idx);
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
}
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
static const MachineOperand &getLdStOffsetOp(const MachineInstr &MI) {
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
unsigned Idx = isPairedLdSt(MI) ? 3 : 2;
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
return MI.getOperand(Idx);
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
}
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
static bool isLdOffsetInRangeOfSt(MachineInstr &LoadInst,
MachineInstr &StoreInst,
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
const AArch64InstrInfo *TII) {
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
assert(isMatchingStore(LoadInst, StoreInst) && "Expect only matched ld/st.");
int LoadSize = getMemScale(LoadInst);
int StoreSize = getMemScale(StoreInst);
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
int UnscaledStOffset = TII->isUnscaledLdSt(StoreInst)
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
? getLdStOffsetOp(StoreInst).getImm()
: getLdStOffsetOp(StoreInst).getImm() * StoreSize;
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
int UnscaledLdOffset = TII->isUnscaledLdSt(LoadInst)
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
? getLdStOffsetOp(LoadInst).getImm()
: getLdStOffsetOp(LoadInst).getImm() * LoadSize;
return (UnscaledStOffset <= UnscaledLdOffset) &&
(UnscaledLdOffset + LoadSize <= (UnscaledStOffset + StoreSize));
}
[AArch64] Decouple zero store promotion from narrow ld merge. NFC. Summary: This change refactors to decouple the zero store promotion from the narrow ld merge and add a flag (enable-narrow-ld-merge=true) to control the narrow ld merge optimization. Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D19885 llvm-svn: 268744
2016-05-06 23:08:57 +08:00
static bool isPromotableZeroStoreOpcode(unsigned Opc) {
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
return isNarrowStore(Opc) || Opc == AArch64::STRWui || Opc == AArch64::STURWi;
}
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
static bool isPromotableZeroStoreOpcode(MachineInstr &MI) {
return isPromotableZeroStoreOpcode(MI.getOpcode());
[AArch64] Decouple zero store promotion from narrow ld merge. NFC. Summary: This change refactors to decouple the zero store promotion from the narrow ld merge and add a flag (enable-narrow-ld-merge=true) to control the narrow ld merge optimization. Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D19885 llvm-svn: 268744
2016-05-06 23:08:57 +08:00
}
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
static bool isPromotableZeroStoreInst(MachineInstr &MI) {
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
return (isPromotableZeroStoreOpcode(MI)) &&
getLdStRegOp(MI).getReg() == AArch64::WZR;
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
AArch64LoadStoreOpt::mergeNarrowInsns(MachineBasicBlock::iterator I,
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MachineBasicBlock::iterator MergeMI,
[AArch64] Simplify the passing of arguments. NFC. This is setup for future work planned for the AArch64 Load/Store Opt pass. llvm-svn: 242810
2015-07-22 01:42:04 +08:00
const LdStPairFlags &Flags) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator NextI = I;
++NextI;
// If NextI is the second of the two instructions to be merged, we need
// to skip one further. Either way we merge will invalidate the iterator,
// and we don't need to scan the new instruction, as it's a pairwise
// instruction, which we're not considering for further action anyway.
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
if (NextI == MergeMI)
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
++NextI;
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
unsigned Opc = I->getOpcode();
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
bool IsScaled = !TII->isUnscaledLdSt(Opc);
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
int OffsetStride = IsScaled ? 1 : getMemScale(*I);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Simplify the passing of arguments. NFC. This is setup for future work planned for the AArch64 Load/Store Opt pass. llvm-svn: 242810
2015-07-22 01:42:04 +08:00
bool MergeForward = Flags.getMergeForward();
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Insert our new paired instruction after whichever of the paired
[AArch64] Fix some LLVM Coding Standards violations in the load/store optimizer. Variable names should start with an upper case letter. No change in functionality. llvm-svn: 210181
2014-06-04 20:36:28 +08:00
// instructions MergeForward indicates.
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MachineBasicBlock::iterator InsertionPoint = MergeForward ? MergeMI : I;
[AArch64] Fix some LLVM Coding Standards violations in the load/store optimizer. Variable names should start with an upper case letter. No change in functionality. llvm-svn: 210181
2014-06-04 20:36:28 +08:00
// Also based on MergeForward is from where we copy the base register operand
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// so we get the flags compatible with the input code.
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
const MachineOperand &BaseRegOp =
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MergeForward ? getLdStBaseOp(*MergeMI) : getLdStBaseOp(*I);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Which register is Rt and which is Rt2 depends on the offset order.
MachineInstr *RtMI, *Rt2MI;
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (getLdStOffsetOp(*I).getImm() ==
getLdStOffsetOp(*MergeMI).getImm() + OffsetStride) {
RtMI = &*MergeMI;
Rt2MI = &*I;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
} else {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
RtMI = &*I;
Rt2MI = &*MergeMI;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
int OffsetImm = getLdStOffsetOp(*RtMI).getImm();
[AArch64] Hoist now common logic. NFC. llvm-svn: 260257
2016-02-10 03:17:18 +08:00
// Change the scaled offset from small to large type.
if (IsScaled) {
assert(((OffsetImm & 1) == 0) && "Unexpected offset to merge");
OffsetImm /= 2;
}
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
DebugLoc DL = I->getDebugLoc();
MachineBasicBlock *MBB = I->getParent();
[AArch64] Refactoring aarch64-ldst-opt. NCF. Summary : * Rename isSmallTypeLdMerge() to isNarrowLoad(). * Rename NumSmallTypeMerged to NumNarrowTypePromoted. * Use Subtarget defined as a member variable. llvm-svn: 253587
2015-11-20 02:41:27 +08:00
if (isNarrowLoad(Opc)) {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr *RtNewDest = &*(MergeForward ? I : MergeMI);
[AArch64] Fix halfword load merging for big-endian targets For big-endian targets, when we merge two halfword loads into a word load, the order of the halfwords in the loaded value is reversed compared to little-endian, so the load-store optimiser needs to swap the destination registers. This does not affect merging of two word loads, as we use ldp, which treats the memory as two separate 32-bit words. llvm-svn: 252597
2015-11-10 19:04:18 +08:00
// When merging small (< 32 bit) loads for big-endian targets, the order of
// the component parts gets swapped.
if (!Subtarget->isLittleEndian())
std::swap(RtMI, Rt2MI);
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
// Construct the new load instruction.
MachineInstr *NewMemMI, *BitExtMI1, *BitExtMI2;
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
NewMemMI =
BuildMI(*MBB, InsertionPoint, DL, TII->get(getMatchingWideOpcode(Opc)))
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
.addOperand(getLdStRegOp(*RtNewDest))
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
.addOperand(BaseRegOp)
.addImm(OffsetImm)
.setMemRefs(I->mergeMemRefsWith(*MergeMI));
[AArch64] Fix warnings pointed out by Hal. llvm-svn: 264882
2016-03-31 02:08:51 +08:00
(void)NewMemMI;
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
DEBUG(
dbgs()
<< "Creating the new load and extract. Replacing instructions:\n ");
DEBUG(I->print(dbgs()));
DEBUG(dbgs() << " ");
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
DEBUG(MergeMI->print(dbgs()));
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
DEBUG(dbgs() << " with instructions:\n ");
DEBUG((NewMemMI)->print(dbgs()));
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
int Width = getMemScale(*I) == 1 ? 8 : 16;
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
int LSBLow = 0;
int LSBHigh = Width;
int ImmsLow = LSBLow + Width - 1;
int ImmsHigh = LSBHigh + Width - 1;
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr *ExtDestMI = &*(MergeForward ? MergeMI : I);
[AArch64] Fix halfword load merging for big-endian targets For big-endian targets, when we merge two halfword loads into a word load, the order of the halfwords in the loaded value is reversed compared to little-endian, so the load-store optimiser needs to swap the destination registers. This does not affect merging of two word loads, as we use ldp, which treats the memory as two separate 32-bit words. llvm-svn: 252597
2015-11-10 19:04:18 +08:00
if ((ExtDestMI == Rt2MI) == Subtarget->isLittleEndian()) {
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
// Create the bitfield extract for high bits.
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BitExtMI1 =
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
BuildMI(*MBB, InsertionPoint, DL, TII->get(getBitExtrOpcode(*Rt2MI)))
.addOperand(getLdStRegOp(*Rt2MI))
.addReg(getLdStRegOp(*RtNewDest).getReg())
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
.addImm(LSBHigh)
.addImm(ImmsHigh);
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
// Create the bitfield extract for low bits.
if (RtMI->getOpcode() == getMatchingNonSExtOpcode(RtMI->getOpcode())) {
// For unsigned, prefer to use AND for low bits.
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BitExtMI2 = BuildMI(*MBB, InsertionPoint, DL, TII->get(AArch64::ANDWri))
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
.addOperand(getLdStRegOp(*RtMI))
.addReg(getLdStRegOp(*RtNewDest).getReg())
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
.addImm(ImmsLow);
} else {
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BitExtMI2 =
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
BuildMI(*MBB, InsertionPoint, DL, TII->get(getBitExtrOpcode(*RtMI)))
.addOperand(getLdStRegOp(*RtMI))
.addReg(getLdStRegOp(*RtNewDest).getReg())
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
.addImm(LSBLow)
.addImm(ImmsLow);
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
}
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
} else {
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
// Create the bitfield extract for low bits.
if (RtMI->getOpcode() == getMatchingNonSExtOpcode(RtMI->getOpcode())) {
// For unsigned, prefer to use AND for low bits.
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BitExtMI1 = BuildMI(*MBB, InsertionPoint, DL, TII->get(AArch64::ANDWri))
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
.addOperand(getLdStRegOp(*RtMI))
.addReg(getLdStRegOp(*RtNewDest).getReg())
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
.addImm(ImmsLow);
} else {
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BitExtMI1 =
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
BuildMI(*MBB, InsertionPoint, DL, TII->get(getBitExtrOpcode(*RtMI)))
.addOperand(getLdStRegOp(*RtMI))
.addReg(getLdStRegOp(*RtNewDest).getReg())
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
.addImm(LSBLow)
.addImm(ImmsLow);
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
}
// Create the bitfield extract for high bits.
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BitExtMI2 =
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
BuildMI(*MBB, InsertionPoint, DL, TII->get(getBitExtrOpcode(*Rt2MI)))
.addOperand(getLdStRegOp(*Rt2MI))
.addReg(getLdStRegOp(*RtNewDest).getReg())
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
.addImm(LSBHigh)
.addImm(ImmsHigh);
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
}
[AArch64] Fix warnings pointed out by Hal. llvm-svn: 264882
2016-03-31 02:08:51 +08:00
(void)BitExtMI1;
(void)BitExtMI2;
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
DEBUG(dbgs() << " ");
DEBUG((BitExtMI1)->print(dbgs()));
DEBUG(dbgs() << " ");
DEBUG((BitExtMI2)->print(dbgs()));
DEBUG(dbgs() << "\n");
// Erase the old instructions.
I->eraseFromParent();
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MergeMI->eraseFromParent();
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
return NextI;
}
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
assert(isPromotableZeroStoreInst(*I) && isPromotableZeroStoreInst(*MergeMI) &&
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
"Expected promotable zero store");
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Construct the new instruction.
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
MachineInstrBuilder MIB;
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
MIB = BuildMI(*MBB, InsertionPoint, DL, TII->get(getMatchingWideOpcode(Opc)))
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
.addReg(isNarrowStore(Opc) ? AArch64::WZR : AArch64::XZR)
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
.addOperand(BaseRegOp)
.addImm(OffsetImm)
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
.setMemRefs(I->mergeMemRefsWith(*MergeMI));
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
(void)MIB;
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
DEBUG(dbgs() << "Creating wider load/store. Replacing instructions:\n ");
DEBUG(I->print(dbgs()));
DEBUG(dbgs() << " ");
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
DEBUG(MergeMI->print(dbgs()));
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
DEBUG(dbgs() << " with instruction:\n ");
DEBUG(((MachineInstr *)MIB)->print(dbgs()));
DEBUG(dbgs() << "\n");
// Erase the old instructions.
I->eraseFromParent();
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MergeMI->eraseFromParent();
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
return NextI;
}
MachineBasicBlock::iterator
AArch64LoadStoreOpt::mergePairedInsns(MachineBasicBlock::iterator I,
MachineBasicBlock::iterator Paired,
const LdStPairFlags &Flags) {
MachineBasicBlock::iterator NextI = I;
++NextI;
// If NextI is the second of the two instructions to be merged, we need
// to skip one further. Either way we merge will invalidate the iterator,
// and we don't need to scan the new instruction, as it's a pairwise
// instruction, which we're not considering for further action anyway.
if (NextI == Paired)
++NextI;
int SExtIdx = Flags.getSExtIdx();
unsigned Opc =
SExtIdx == -1 ? I->getOpcode() : getMatchingNonSExtOpcode(I->getOpcode());
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
bool IsUnscaled = TII->isUnscaledLdSt(Opc);
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
int OffsetStride = IsUnscaled ? getMemScale(*I) : 1;
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
bool MergeForward = Flags.getMergeForward();
// Insert our new paired instruction after whichever of the paired
// instructions MergeForward indicates.
MachineBasicBlock::iterator InsertionPoint = MergeForward ? Paired : I;
// Also based on MergeForward is from where we copy the base register operand
// so we get the flags compatible with the input code.
const MachineOperand &BaseRegOp =
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MergeForward ? getLdStBaseOp(*Paired) : getLdStBaseOp(*I);
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
int Offset = getLdStOffsetOp(*I).getImm();
int PairedOffset = getLdStOffsetOp(*Paired).getImm();
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
bool PairedIsUnscaled = TII->isUnscaledLdSt(Paired->getOpcode());
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
if (IsUnscaled != PairedIsUnscaled) {
// We're trying to pair instructions that differ in how they are scaled. If
// I is scaled then scale the offset of Paired accordingly. Otherwise, do
// the opposite (i.e., make Paired's offset unscaled).
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
int MemSize = getMemScale(*Paired);
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
if (PairedIsUnscaled) {
// If the unscaled offset isn't a multiple of the MemSize, we can't
// pair the operations together.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
assert(!(PairedOffset % getMemScale(*Paired)) &&
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
"Offset should be a multiple of the stride!");
PairedOffset /= MemSize;
} else {
PairedOffset *= MemSize;
}
}
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
// Which register is Rt and which is Rt2 depends on the offset order.
MachineInstr *RtMI, *Rt2MI;
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
if (Offset == PairedOffset + OffsetStride) {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
RtMI = &*Paired;
Rt2MI = &*I;
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
// Here we swapped the assumption made for SExtIdx.
// I.e., we turn ldp I, Paired into ldp Paired, I.
// Update the index accordingly.
if (SExtIdx != -1)
SExtIdx = (SExtIdx + 1) % 2;
} else {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
RtMI = &*I;
Rt2MI = &*Paired;
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
}
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
int OffsetImm = getLdStOffsetOp(*RtMI).getImm();
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
// Scale the immediate offset, if necessary.
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
if (TII->isUnscaledLdSt(RtMI->getOpcode())) {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
assert(!(OffsetImm % getMemScale(*RtMI)) &&
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
"Unscaled offset cannot be scaled.");
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
OffsetImm /= getMemScale(*RtMI);
[AArch64] Add an assert to ensure we don't scale an offset that can't be scaled. llvm-svn: 260272
2016-02-10 04:18:07 +08:00
}
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
// Construct the new instruction.
MachineInstrBuilder MIB;
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
DebugLoc DL = I->getDebugLoc();
MachineBasicBlock *MBB = I->getParent();
MIB = BuildMI(*MBB, InsertionPoint, DL, TII->get(getMatchingPairOpcode(Opc)))
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
.addOperand(getLdStRegOp(*RtMI))
.addOperand(getLdStRegOp(*Rt2MI))
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
.addOperand(BaseRegOp)
[AArch64] Add MMOs to unscaled pairs. Test to be committed in follow up commit, per discussion in D17097. http://reviews.llvm.org/D17097 llvm-svn: 262942
2016-03-09 01:16:38 +08:00
.addImm(OffsetImm)
.setMemRefs(I->mergeMemRefsWith(*Paired));
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
(void)MIB;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
DEBUG(dbgs() << "Creating pair load/store. Replacing instructions:\n ");
DEBUG(I->print(dbgs()));
DEBUG(dbgs() << " ");
DEBUG(Paired->print(dbgs()));
DEBUG(dbgs() << " with instruction:\n ");
[AArch64][LoadStoreOptimizer] Generate LDP + SXTW instead of LD[U]R + LD[U]RSW. Teach the load store optimizer how to sign extend a result of a load pair when it helps creating more pairs. The rational is that loads are more expensive than sign extensions, so if we gather some in one instruction this is better! <rdar://problem/20072968> llvm-svn: 231527
2015-03-07 06:42:10 +08:00
if (SExtIdx != -1) {
// Generate the sign extension for the proper result of the ldp.
// I.e., with X1, that would be:
// %W1<def> = KILL %W1, %X1<imp-def>
// %X1<def> = SBFMXri %X1<kill>, 0, 31
MachineOperand &DstMO = MIB->getOperand(SExtIdx);
// Right now, DstMO has the extended register, since it comes from an
// extended opcode.
unsigned DstRegX = DstMO.getReg();
// Get the W variant of that register.
unsigned DstRegW = TRI->getSubReg(DstRegX, AArch64::sub_32);
// Update the result of LDP to use the W instead of the X variant.
DstMO.setReg(DstRegW);
DEBUG(((MachineInstr *)MIB)->print(dbgs()));
DEBUG(dbgs() << "\n");
// Make the machine verifier happy by providing a definition for
// the X register.
// Insert this definition right after the generated LDP, i.e., before
// InsertionPoint.
MachineInstrBuilder MIBKill =
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BuildMI(*MBB, InsertionPoint, DL, TII->get(TargetOpcode::KILL), DstRegW)
[AArch64][LoadStoreOptimizer] Generate LDP + SXTW instead of LD[U]R + LD[U]RSW. Teach the load store optimizer how to sign extend a result of a load pair when it helps creating more pairs. The rational is that loads are more expensive than sign extensions, so if we gather some in one instruction this is better! <rdar://problem/20072968> llvm-svn: 231527
2015-03-07 06:42:10 +08:00
.addReg(DstRegW)
.addReg(DstRegX, RegState::Define);
MIBKill->getOperand(2).setImplicit();
// Create the sign extension.
MachineInstrBuilder MIBSXTW =
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BuildMI(*MBB, InsertionPoint, DL, TII->get(AArch64::SBFMXri), DstRegX)
[AArch64][LoadStoreOptimizer] Generate LDP + SXTW instead of LD[U]R + LD[U]RSW. Teach the load store optimizer how to sign extend a result of a load pair when it helps creating more pairs. The rational is that loads are more expensive than sign extensions, so if we gather some in one instruction this is better! <rdar://problem/20072968> llvm-svn: 231527
2015-03-07 06:42:10 +08:00
.addReg(DstRegX)
.addImm(0)
.addImm(31);
(void)MIBSXTW;
DEBUG(dbgs() << " Extend operand:\n ");
DEBUG(((MachineInstr *)MIBSXTW)->print(dbgs()));
} else {
DEBUG(((MachineInstr *)MIB)->print(dbgs()));
}
[AArch64] Simplify. NFC. llvm-svn: 260273
2016-02-10 04:27:45 +08:00
DEBUG(dbgs() << "\n");
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Erase the old instructions.
I->eraseFromParent();
Paired->eraseFromParent();
return NextI;
}
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
MachineBasicBlock::iterator
AArch64LoadStoreOpt::promoteLoadFromStore(MachineBasicBlock::iterator LoadI,
MachineBasicBlock::iterator StoreI) {
MachineBasicBlock::iterator NextI = LoadI;
++NextI;
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
int LoadSize = getMemScale(*LoadI);
int StoreSize = getMemScale(*StoreI);
unsigned LdRt = getLdStRegOp(*LoadI).getReg();
unsigned StRt = getLdStRegOp(*StoreI).getReg();
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
bool IsStoreXReg = TRI->getRegClass(AArch64::GPR64RegClassID)->contains(StRt);
assert((IsStoreXReg ||
TRI->getRegClass(AArch64::GPR32RegClassID)->contains(StRt)) &&
"Unexpected RegClass");
MachineInstr *BitExtMI;
if (LoadSize == StoreSize && (LoadSize == 4 || LoadSize == 8)) {
// Remove the load, if the destination register of the loads is the same
// register for stored value.
if (StRt == LdRt && LoadSize == 8) {
DEBUG(dbgs() << "Remove load instruction:\n ");
DEBUG(LoadI->print(dbgs()));
DEBUG(dbgs() << "\n");
LoadI->eraseFromParent();
return NextI;
}
// Replace the load with a mov if the load and store are in the same size.
BitExtMI =
BuildMI(*LoadI->getParent(), LoadI, LoadI->getDebugLoc(),
TII->get(IsStoreXReg ? AArch64::ORRXrs : AArch64::ORRWrs), LdRt)
.addReg(IsStoreXReg ? AArch64::XZR : AArch64::WZR)
.addReg(StRt)
.addImm(AArch64_AM::getShifterImm(AArch64_AM::LSL, 0));
} else {
// FIXME: Currently we disable this transformation in big-endian targets as
// performance and correctness are verified only in little-endian.
if (!Subtarget->isLittleEndian())
return NextI;
CodeGen: Use MachineInstr& in TargetInstrInfo, NFC This is mostly a mechanical change to make TargetInstrInfo API take MachineInstr& (instead of MachineInstr* or MachineBasicBlock::iterator) when the argument is expected to be a valid MachineInstr. This is a general API improvement. Although it would be possible to do this one function at a time, that would demand a quadratic amount of churn since many of these functions call each other. Instead I've done everything as a block and just updated what was necessary. This is mostly mechanical fixes: adding and removing `*` and `&` operators. The only non-mechanical change is to split ARMBaseInstrInfo::getOperandLatencyImpl out from ARMBaseInstrInfo::getOperandLatency. Previously, the latter took a `MachineInstr*` which it updated to the instruction bundle leader; now, the latter calls the former either with the same `MachineInstr&` or the bundle leader. As a side effect, this removes a bunch of MachineInstr* to MachineBasicBlock::iterator implicit conversions, a necessary step toward fixing PR26753. Note: I updated WebAssembly, Lanai, and AVR (despite being off-by-default) since it turned out to be easy. I couldn't run tests for AVR since llc doesn't link with it turned on. llvm-svn: 274189
2016-06-30 08:01:54 +08:00
bool IsUnscaled = TII->isUnscaledLdSt(*LoadI);
assert(IsUnscaled == TII->isUnscaledLdSt(*StoreI) &&
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
"Unsupported ld/st match");
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
assert(LoadSize <= StoreSize && "Invalid load size");
int UnscaledLdOffset = IsUnscaled
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
? getLdStOffsetOp(*LoadI).getImm()
: getLdStOffsetOp(*LoadI).getImm() * LoadSize;
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
int UnscaledStOffset = IsUnscaled
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
? getLdStOffsetOp(*StoreI).getImm()
: getLdStOffsetOp(*StoreI).getImm() * StoreSize;
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
int Width = LoadSize * 8;
int Immr = 8 * (UnscaledLdOffset - UnscaledStOffset);
int Imms = Immr + Width - 1;
unsigned DestReg = IsStoreXReg
? TRI->getMatchingSuperReg(LdRt, AArch64::sub_32,
&AArch64::GPR64RegClass)
: LdRt;
assert((UnscaledLdOffset >= UnscaledStOffset &&
(UnscaledLdOffset + LoadSize) <= UnscaledStOffset + StoreSize) &&
"Invalid offset");
Immr = 8 * (UnscaledLdOffset - UnscaledStOffset);
Imms = Immr + Width - 1;
if (UnscaledLdOffset == UnscaledStOffset) {
uint32_t AndMaskEncoded = ((IsStoreXReg ? 1 : 0) << 12) // N
| ((Immr) << 6) // immr
| ((Imms) << 0) // imms
;
BitExtMI =
BuildMI(*LoadI->getParent(), LoadI, LoadI->getDebugLoc(),
TII->get(IsStoreXReg ? AArch64::ANDXri : AArch64::ANDWri),
DestReg)
.addReg(StRt)
.addImm(AndMaskEncoded);
} else {
BitExtMI =
BuildMI(*LoadI->getParent(), LoadI, LoadI->getDebugLoc(),
TII->get(IsStoreXReg ? AArch64::UBFMXri : AArch64::UBFMWri),
DestReg)
.addReg(StRt)
.addImm(Immr)
.addImm(Imms);
}
}
[AArch64] Fix warnings pointed out by Hal. llvm-svn: 264882
2016-03-31 02:08:51 +08:00
(void)BitExtMI;
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
DEBUG(dbgs() << "Promoting load by replacing :\n ");
DEBUG(StoreI->print(dbgs()));
DEBUG(dbgs() << " ");
DEBUG(LoadI->print(dbgs()));
DEBUG(dbgs() << " with instructions:\n ");
DEBUG(StoreI->print(dbgs()));
DEBUG(dbgs() << " ");
DEBUG((BitExtMI)->print(dbgs()));
DEBUG(dbgs() << "\n");
// Erase the old instructions.
LoadI->eraseFromParent();
return NextI;
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
/// trackRegDefsUses - Remember what registers the specified instruction uses
/// and modifies.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
static void trackRegDefsUses(const MachineInstr &MI, BitVector &ModifiedRegs,
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
BitVector &UsedRegs,
const TargetRegisterInfo *TRI) {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
for (const MachineOperand &MO : MI.operands()) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
if (MO.isRegMask())
ModifiedRegs.setBitsNotInMask(MO.getRegMask());
if (!MO.isReg())
continue;
unsigned Reg = MO.getReg();
[AArch64] AArch64LoadStoreOptimizer: fix bug in pre-inc check iterator Summary: Fix case where a pre-inc/dec load/store would not be formed if the add/sub that forms the inc/dec part of the operation was the first instruction in the block being examined. Reviewers: mcrosier, jmolloy, t.p.northover, junbuml Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16785 llvm-svn: 260275
2016-02-10 04:47:21 +08:00
if (!Reg)
continue;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
if (MO.isDef()) {
for (MCRegAliasIterator AI(Reg, TRI, true); AI.isValid(); ++AI)
ModifiedRegs.set(*AI);
} else {
assert(MO.isUse() && "Reg operand not a def and not a use?!?");
for (MCRegAliasIterator AI(Reg, TRI, true); AI.isValid(); ++AI)
UsedRegs.set(*AI);
}
}
}
static bool inBoundsForPair(bool IsUnscaled, int Offset, int OffsetStride) {
[AArch64] Simplify the logic for computing in bounds offset. NFC. llvm-svn: 245307
2015-08-19 00:20:03 +08:00
// Convert the byte-offset used by unscaled into an "element" offset used
// by the scaled pair load/store instructions.
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
if (IsUnscaled) {
// If the byte-offset isn't a multiple of the stride, there's no point
// trying to match it.
if (Offset % OffsetStride)
return false;
[AArch64] Simplify the logic for computing in bounds offset. NFC. llvm-svn: 245307
2015-08-19 00:20:03 +08:00
Offset /= OffsetStride;
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
}
[AArch64] Simplify the logic for computing in bounds offset. NFC. llvm-svn: 245307
2015-08-19 00:20:03 +08:00
return Offset <= 63 && Offset >= -64;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
// Do alignment, specialized to power of 2 and for signed ints,
// avoiding having to do a C-style cast from uint_64t to int when
Update to use new name alignTo(). llvm-svn: 257804
2016-01-15 05:06:47 +08:00
// using alignTo from include/llvm/Support/MathExtras.h.
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// FIXME: Move this function to include/MathExtras.h?
static int alignTo(int Num, int PowOf2) {
return (Num + PowOf2 - 1) & ~(PowOf2 - 1);
}
CodeGen: Use MachineInstr& in TargetInstrInfo, NFC This is mostly a mechanical change to make TargetInstrInfo API take MachineInstr& (instead of MachineInstr* or MachineBasicBlock::iterator) when the argument is expected to be a valid MachineInstr. This is a general API improvement. Although it would be possible to do this one function at a time, that would demand a quadratic amount of churn since many of these functions call each other. Instead I've done everything as a block and just updated what was necessary. This is mostly mechanical fixes: adding and removing `*` and `&` operators. The only non-mechanical change is to split ARMBaseInstrInfo::getOperandLatencyImpl out from ARMBaseInstrInfo::getOperandLatency. Previously, the latter took a `MachineInstr*` which it updated to the instruction bundle leader; now, the latter calls the former either with the same `MachineInstr&` or the bundle leader. As a side effect, this removes a bunch of MachineInstr* to MachineBasicBlock::iterator implicit conversions, a necessary step toward fixing PR26753. Note: I updated WebAssembly, Lanai, and AVR (despite being off-by-default) since it turned out to be easy. I couldn't run tests for AVR since llc doesn't link with it turned on. llvm-svn: 274189
2016-06-30 08:01:54 +08:00
static bool mayAlias(MachineInstr &MIa, MachineInstr &MIb,
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
const AArch64InstrInfo *TII) {
// One of the instructions must modify memory.
CodeGen: Use MachineInstr& in TargetInstrInfo, NFC This is mostly a mechanical change to make TargetInstrInfo API take MachineInstr& (instead of MachineInstr* or MachineBasicBlock::iterator) when the argument is expected to be a valid MachineInstr. This is a general API improvement. Although it would be possible to do this one function at a time, that would demand a quadratic amount of churn since many of these functions call each other. Instead I've done everything as a block and just updated what was necessary. This is mostly mechanical fixes: adding and removing `*` and `&` operators. The only non-mechanical change is to split ARMBaseInstrInfo::getOperandLatencyImpl out from ARMBaseInstrInfo::getOperandLatency. Previously, the latter took a `MachineInstr*` which it updated to the instruction bundle leader; now, the latter calls the former either with the same `MachineInstr&` or the bundle leader. As a side effect, this removes a bunch of MachineInstr* to MachineBasicBlock::iterator implicit conversions, a necessary step toward fixing PR26753. Note: I updated WebAssembly, Lanai, and AVR (despite being off-by-default) since it turned out to be easy. I couldn't run tests for AVR since llc doesn't link with it turned on. llvm-svn: 274189
2016-06-30 08:01:54 +08:00
if (!MIa.mayStore() && !MIb.mayStore())
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
return false;
// Both instructions must be memory operations.
CodeGen: Use MachineInstr& in TargetInstrInfo, NFC This is mostly a mechanical change to make TargetInstrInfo API take MachineInstr& (instead of MachineInstr* or MachineBasicBlock::iterator) when the argument is expected to be a valid MachineInstr. This is a general API improvement. Although it would be possible to do this one function at a time, that would demand a quadratic amount of churn since many of these functions call each other. Instead I've done everything as a block and just updated what was necessary. This is mostly mechanical fixes: adding and removing `*` and `&` operators. The only non-mechanical change is to split ARMBaseInstrInfo::getOperandLatencyImpl out from ARMBaseInstrInfo::getOperandLatency. Previously, the latter took a `MachineInstr*` which it updated to the instruction bundle leader; now, the latter calls the former either with the same `MachineInstr&` or the bundle leader. As a side effect, this removes a bunch of MachineInstr* to MachineBasicBlock::iterator implicit conversions, a necessary step toward fixing PR26753. Note: I updated WebAssembly, Lanai, and AVR (despite being off-by-default) since it turned out to be easy. I couldn't run tests for AVR since llc doesn't link with it turned on. llvm-svn: 274189
2016-06-30 08:01:54 +08:00
if (!MIa.mayLoadOrStore() && !MIb.mayLoadOrStore())
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
return false;
return !TII->areMemAccessesTriviallyDisjoint(MIa, MIb);
}
CodeGen: Use MachineInstr& in TargetInstrInfo, NFC This is mostly a mechanical change to make TargetInstrInfo API take MachineInstr& (instead of MachineInstr* or MachineBasicBlock::iterator) when the argument is expected to be a valid MachineInstr. This is a general API improvement. Although it would be possible to do this one function at a time, that would demand a quadratic amount of churn since many of these functions call each other. Instead I've done everything as a block and just updated what was necessary. This is mostly mechanical fixes: adding and removing `*` and `&` operators. The only non-mechanical change is to split ARMBaseInstrInfo::getOperandLatencyImpl out from ARMBaseInstrInfo::getOperandLatency. Previously, the latter took a `MachineInstr*` which it updated to the instruction bundle leader; now, the latter calls the former either with the same `MachineInstr&` or the bundle leader. As a side effect, this removes a bunch of MachineInstr* to MachineBasicBlock::iterator implicit conversions, a necessary step toward fixing PR26753. Note: I updated WebAssembly, Lanai, and AVR (despite being off-by-default) since it turned out to be easy. I couldn't run tests for AVR since llc doesn't link with it turned on. llvm-svn: 274189
2016-06-30 08:01:54 +08:00
static bool mayAlias(MachineInstr &MIa,
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
SmallVectorImpl<MachineInstr *> &MemInsns,
const AArch64InstrInfo *TII) {
CodeGen: Use MachineInstr& in TargetInstrInfo, NFC This is mostly a mechanical change to make TargetInstrInfo API take MachineInstr& (instead of MachineInstr* or MachineBasicBlock::iterator) when the argument is expected to be a valid MachineInstr. This is a general API improvement. Although it would be possible to do this one function at a time, that would demand a quadratic amount of churn since many of these functions call each other. Instead I've done everything as a block and just updated what was necessary. This is mostly mechanical fixes: adding and removing `*` and `&` operators. The only non-mechanical change is to split ARMBaseInstrInfo::getOperandLatencyImpl out from ARMBaseInstrInfo::getOperandLatency. Previously, the latter took a `MachineInstr*` which it updated to the instruction bundle leader; now, the latter calls the former either with the same `MachineInstr&` or the bundle leader. As a side effect, this removes a bunch of MachineInstr* to MachineBasicBlock::iterator implicit conversions, a necessary step toward fixing PR26753. Note: I updated WebAssembly, Lanai, and AVR (despite being off-by-default) since it turned out to be easy. I couldn't run tests for AVR since llc doesn't link with it turned on. llvm-svn: 274189
2016-06-30 08:01:54 +08:00
for (MachineInstr *MIb : MemInsns)
if (mayAlias(MIa, *MIb, TII))
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
return true;
return false;
}
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
bool AArch64LoadStoreOpt::findMatchingStore(
MachineBasicBlock::iterator I, unsigned Limit,
MachineBasicBlock::iterator &StoreI) {
[AArch64] Refactoring findMatchingStore() in aarch64-ldst-opt; NFC Summary: This change makes findMatchingStore() follow the same coding style introduced in r260275. Reviewers: gberry, junbuml Subscribers: aemerson, rengolin, haicheng, bmakam, mssimpso Differential Revision: http://reviews.llvm.org/D17083 llvm-svn: 260534
2016-02-12 00:18:24 +08:00
MachineBasicBlock::iterator B = I->getParent()->begin();
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
MachineBasicBlock::iterator MBBI = I;
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr &LoadMI = *I;
[AArch64] Rename variable to improve readability. NFC. llvm-svn: 260228
2016-02-09 23:59:57 +08:00
unsigned BaseReg = getLdStBaseOp(LoadMI).getReg();
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
[AArch64] Refactoring findMatchingStore() in aarch64-ldst-opt; NFC Summary: This change makes findMatchingStore() follow the same coding style introduced in r260275. Reviewers: gberry, junbuml Subscribers: aemerson, rengolin, haicheng, bmakam, mssimpso Differential Revision: http://reviews.llvm.org/D17083 llvm-svn: 260534
2016-02-12 00:18:24 +08:00
// If the load is the first instruction in the block, there's obviously
// not any matching store.
if (MBBI == B)
return false;
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
// Track which registers have been modified and used between the first insn
// and the second insn.
[AArch64] Allocate the modified and used regs only once per function. llvm-svn: 259510
2016-02-02 23:02:30 +08:00
ModifiedRegs.reset();
UsedRegs.reset();
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
[AArch64] Refactoring findMatchingStore() in aarch64-ldst-opt; NFC Summary: This change makes findMatchingStore() follow the same coding style introduced in r260275. Reviewers: gberry, junbuml Subscribers: aemerson, rengolin, haicheng, bmakam, mssimpso Differential Revision: http://reviews.llvm.org/D17083 llvm-svn: 260534
2016-02-12 00:18:24 +08:00
unsigned Count = 0;
do {
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
--MBBI;
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr &MI = *MBBI;
[AArch64] Refactoring findMatchingStore() in aarch64-ldst-opt; NFC Summary: This change makes findMatchingStore() follow the same coding style introduced in r260275. Reviewers: gberry, junbuml Subscribers: aemerson, rengolin, haicheng, bmakam, mssimpso Differential Revision: http://reviews.llvm.org/D17083 llvm-svn: 260534
2016-02-12 00:18:24 +08:00
[AArch64] Load/store opt: Don't count transient instructions towards search limits. Summary: This change also changes findMatchingInsn and findMatchingUpdateInsnForward to take DBG_VALUE opcodes into account when tracking register defs and uses, which could potentially inhibit these optimizations in the presence of debug information. Reviewers: mcrosier Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: https://reviews.llvm.org/D22582 llvm-svn: 276293
2016-07-21 23:20:25 +08:00
// Don't count transient instructions towards the search limit since there
// may be different numbers of them if e.g. debug information is present.
if (!MI.isTransient())
[AArch64] Refactoring findMatchingStore() in aarch64-ldst-opt; NFC Summary: This change makes findMatchingStore() follow the same coding style introduced in r260275. Reviewers: gberry, junbuml Subscribers: aemerson, rengolin, haicheng, bmakam, mssimpso Differential Revision: http://reviews.llvm.org/D17083 llvm-svn: 260534
2016-02-12 00:18:24 +08:00
++Count;
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
// If the load instruction reads directly from the address to which the
// store instruction writes and the stored value is not modified, we can
// promote the load. Since we do not handle stores with pre-/post-index,
// it's unnecessary to check if BaseReg is modified by the store itself.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (MI.mayStore() && isMatchingStore(LoadMI, MI) &&
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
BaseReg == getLdStBaseOp(MI).getReg() &&
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
isLdOffsetInRangeOfSt(LoadMI, MI, TII) &&
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
!ModifiedRegs[getLdStRegOp(MI).getReg()]) {
StoreI = MBBI;
return true;
}
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (MI.isCall())
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
return false;
// Update modified / uses register lists.
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
// Otherwise, if the base register is modified, we have no match, so
// return early.
if (ModifiedRegs[BaseReg])
return false;
// If we encounter a store aliased with the load, return early.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (MI.mayStore() && mayAlias(LoadMI, MI, TII))
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
return false;
[AArch64] Refactoring findMatchingStore() in aarch64-ldst-opt; NFC Summary: This change makes findMatchingStore() follow the same coding style introduced in r260275. Reviewers: gberry, junbuml Subscribers: aemerson, rengolin, haicheng, bmakam, mssimpso Differential Revision: http://reviews.llvm.org/D17083 llvm-svn: 260534
2016-02-12 00:18:24 +08:00
} while (MBBI != B && Count < Limit);
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
return false;
}
[AArch64] Refactor a check earlier. NFC. llvm-svn: 272429
2016-06-11 04:47:14 +08:00
// Returns true if FirstMI and MI are candidates for merging or pairing.
// Otherwise, returns false.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
static bool areCandidatesToMergeOrPair(MachineInstr &FirstMI, MachineInstr &MI,
[AArch64] Refactor a check earlier. NFC. llvm-svn: 272429
2016-06-11 04:47:14 +08:00
LdStPairFlags &Flags,
const AArch64InstrInfo *TII) {
// If this is volatile or if pairing is suppressed, not a candidate.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (MI.hasOrderedMemoryRef() || TII->isLdStPairSuppressed(MI))
[AArch64] Refactor a check earlier. NFC. llvm-svn: 272429
2016-06-11 04:47:14 +08:00
return false;
// We should have already checked FirstMI for pair suppression and volatility.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
assert(!FirstMI.hasOrderedMemoryRef() &&
!TII->isLdStPairSuppressed(FirstMI) &&
[AArch64] Refactor a check earlier. NFC. llvm-svn: 272429
2016-06-11 04:47:14 +08:00
"FirstMI shouldn't get here if either of these checks are true.");
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
unsigned OpcA = FirstMI.getOpcode();
unsigned OpcB = MI.getOpcode();
[AArch64] Refactor a check earlier. NFC. llvm-svn: 272429
2016-06-11 04:47:14 +08:00
[AArch64] Refactor is logic into a helper function. NFC. llvm-svn: 260419
2016-02-11 03:45:48 +08:00
// Opcodes match: nothing more to check.
if (OpcA == OpcB)
return true;
// Try to match a sign-extended load/store with a zero-extended load/store.
bool IsValidLdStrOpc, PairIsValidLdStrOpc;
unsigned NonSExtOpc = getMatchingNonSExtOpcode(OpcA, &IsValidLdStrOpc);
assert(IsValidLdStrOpc &&
"Given Opc should be a Load or Store with an immediate");
// OpcA will be the first instruction in the pair.
if (NonSExtOpc == getMatchingNonSExtOpcode(OpcB, &PairIsValidLdStrOpc)) {
Flags.setSExtIdx(NonSExtOpc == (unsigned)OpcA ? 1 : 0);
return true;
}
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
// If the second instruction isn't even a load/store, bail out.
if (!PairIsValidLdStrOpc)
return false;
// FIXME: We don't support merging narrow loads/stores with mixed
// scaled/unscaled offsets.
if (isNarrowLoadOrStore(OpcA) || isNarrowLoadOrStore(OpcB))
return false;
// Try to match an unscaled load/store with a scaled load/store.
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
return TII->isUnscaledLdSt(OpcA) != TII->isUnscaledLdSt(OpcB) &&
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
getMatchingPairOpcode(OpcA) == getMatchingPairOpcode(OpcB);
// FIXME: Can we also match a mixed sext/zext unscaled/scaled pair?
[AArch64] Refactor is logic into a helper function. NFC. llvm-svn: 260419
2016-02-11 03:45:48 +08:00
}
[AArch64] Update comment to match reality. NFC. llvm-svn: 260406
2016-02-11 02:49:28 +08:00
/// Scan the instructions looking for a load/store that can be combined with the
/// current instruction into a wider equivalent or a load/store pair.
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
AArch64LoadStoreOpt::findMatchingInsn(MachineBasicBlock::iterator I,
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
LdStPairFlags &Flags, unsigned Limit,
bool FindNarrowMerge) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator E = I->getParent()->end();
MachineBasicBlock::iterator MBBI = I;
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr &FirstMI = *I;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
++MBBI;
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
bool MayLoad = FirstMI.mayLoad();
bool IsUnscaled = TII->isUnscaledLdSt(FirstMI);
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
unsigned Reg = getLdStRegOp(FirstMI).getReg();
unsigned BaseReg = getLdStBaseOp(FirstMI).getReg();
int Offset = getLdStOffsetOp(FirstMI).getImm();
[AArch64] Deprecate a command-line option used for testing. Support for pairing unscaled loads and stores has been enabled since the original ARM64 port. This feature is no longer experimental, AFAICT. llvm-svn: 249049
2015-10-02 02:17:12 +08:00
int OffsetStride = IsUnscaled ? getMemScale(FirstMI) : 1;
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
bool IsPromotableZeroStore = isPromotableZeroStoreInst(FirstMI);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Track which registers have been modified and used between the first insn
// (inclusive) and the second insn.
[AArch64] Allocate the modified and used regs only once per function. llvm-svn: 259510
2016-02-02 23:02:30 +08:00
ModifiedRegs.reset();
UsedRegs.reset();
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
// Remember any instructions that read/write memory between FirstMI and MI.
SmallVector<MachineInstr *, 4> MemInsns;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
for (unsigned Count = 0; MBBI != E && Count < Limit; ++MBBI) {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr &MI = *MBBI;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Load/store opt: Don't count transient instructions towards search limits. Summary: This change also changes findMatchingInsn and findMatchingUpdateInsnForward to take DBG_VALUE opcodes into account when tracking register defs and uses, which could potentially inhibit these optimizations in the presence of debug information. Reviewers: mcrosier Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: https://reviews.llvm.org/D22582 llvm-svn: 276293
2016-07-21 23:20:25 +08:00
// Don't count transient instructions towards the search limit since there
// may be different numbers of them if e.g. debug information is present.
if (!MI.isTransient())
++Count;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
Revert "[AArch64] Improve load/store optimizer to handle LDUR + LDR." This reverts commit r259790. tramp3d-v4 is still having problems. llvm-svn: 259795
2016-02-05 00:01:40 +08:00
Flags.setSExtIdx(-1);
[AArch64] Refactor a check earlier. NFC. llvm-svn: 272429
2016-06-11 04:47:14 +08:00
if (areCandidatesToMergeOrPair(FirstMI, MI, Flags, TII) &&
[AArch64] Refactor is logic into a helper function. NFC. llvm-svn: 260419
2016-02-11 03:45:48 +08:00
getLdStOffsetOp(MI).isImm()) {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
assert(MI.mayLoadOrStore() && "Expected memory operation.");
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// If we've found another instruction with the same opcode, check to see
// if the base and offset are compatible with our starting instruction.
// These instructions all have scaled immediate operands, so we just
// check for +1/-1. Make sure to check the new instruction offset is
// actually an immediate and not a symbolic reference destined for
// a relocation.
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
unsigned MIBaseReg = getLdStBaseOp(MI).getReg();
int MIOffset = getLdStOffsetOp(MI).getImm();
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
bool MIIsUnscaled = TII->isUnscaledLdSt(MI);
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
if (IsUnscaled != MIIsUnscaled) {
// We're trying to pair instructions that differ in how they are scaled.
// If FirstMI is scaled then scale the offset of MI accordingly.
// Otherwise, do the opposite (i.e., make MI's offset unscaled).
int MemSize = getMemScale(MI);
if (MIIsUnscaled) {
// If the unscaled offset isn't a multiple of the MemSize, we can't
// pair the operations together: bail and keep looking.
if (MIOffset % MemSize)
continue;
MIOffset /= MemSize;
} else {
MIOffset *= MemSize;
}
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
if (BaseReg == MIBaseReg && ((Offset == MIOffset + OffsetStride) ||
(Offset + OffsetStride == MIOffset))) {
int MinOffset = Offset < MIOffset ? Offset : MIOffset;
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
if (FindNarrowMerge) {
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
// If the alignment requirements of the scaled wide load/store
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
// instruction can't express the offset of the scaled narrow input,
// bail and keep looking. For promotable zero stores, allow only when
// the stored value is the same (i.e., WZR).
if ((!IsUnscaled && alignTo(MinOffset, 2) != MinOffset) ||
(IsPromotableZeroStore && Reg != getLdStRegOp(MI).getReg())) {
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MemInsns.push_back(&MI);
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
continue;
}
} else {
[AArch64] Move comments closer to relevant check. NFC. llvm-svn: 272430
2016-06-11 04:49:18 +08:00
// Pairwise instructions have a 7-bit signed offset field. Single
// insns have a 12-bit unsigned offset field. If the resultant
// immediate offset of merging these instructions is out of range for
// a pairwise instruction, bail and keep looking.
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
if (!inBoundsForPair(IsUnscaled, MinOffset, OffsetStride)) {
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MemInsns.push_back(&MI);
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
continue;
}
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
// If the alignment requirements of the paired (scaled) instruction
// can't express the offset of the unscaled input, bail and keep
// looking.
if (IsUnscaled && (alignTo(MinOffset, OffsetStride) != MinOffset)) {
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MemInsns.push_back(&MI);
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
continue;
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
// If the destination register of the loads is the same register, bail
// and keep looking. A load-pair instruction with both destination
// registers the same is UNPREDICTABLE and will result in an exception.
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
if (MayLoad && Reg == getLdStRegOp(MI).getReg()) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MemInsns.push_back(&MI);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
continue;
}
// If the Rt of the second instruction was not modified or used between
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
// the two instructions and none of the instructions between the second
// and first alias with the second, we can combine the second into the
// first.
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
if (!ModifiedRegs[getLdStRegOp(MI).getReg()] &&
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
!(MI.mayLoad() && UsedRegs[getLdStRegOp(MI).getReg()]) &&
!mayAlias(MI, MemInsns, TII)) {
[AArch64] Simplify the passing of arguments. NFC. This is setup for future work planned for the AArch64 Load/Store Opt pass. llvm-svn: 242810
2015-07-22 01:42:04 +08:00
Flags.setMergeForward(false);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return MBBI;
}
// Likewise, if the Rt of the first instruction is not modified or used
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
// between the two instructions and none of the instructions between the
// first and the second alias with the first, we can combine the first
// into the second.
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
if (!ModifiedRegs[getLdStRegOp(FirstMI).getReg()] &&
[AArch64] Reuse MayLoad. NFC. llvm-svn: 246767
2015-09-03 22:19:43 +08:00
!(MayLoad && UsedRegs[getLdStRegOp(FirstMI).getReg()]) &&
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
!mayAlias(FirstMI, MemInsns, TII)) {
[AArch64] Simplify the passing of arguments. NFC. This is setup for future work planned for the AArch64 Load/Store Opt pass. llvm-svn: 242810
2015-07-22 01:42:04 +08:00
Flags.setMergeForward(true);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return MBBI;
}
// Unable to combine these instructions due to interference in between.
// Keep looking.
}
}
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
// If the instruction wasn't a matching load or store. Stop searching if we
// encounter a call instruction that might modify memory.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (MI.isCall())
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return E;
// Update modified / uses register lists.
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
// Otherwise, if the base register is modified, we have no match, so
// return early.
if (ModifiedRegs[BaseReg])
return E;
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
// Update list of instructions that read/write memory.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (MI.mayLoadOrStore())
MemInsns.push_back(&MI);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
return E;
}
MachineBasicBlock::iterator
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
AArch64LoadStoreOpt::mergeUpdateInsn(MachineBasicBlock::iterator I,
MachineBasicBlock::iterator Update,
bool IsPreIdx) {
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
assert((Update->getOpcode() == AArch64::ADDXri ||
Update->getOpcode() == AArch64::SUBXri) &&
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
"Unexpected base register update instruction to merge!");
MachineBasicBlock::iterator NextI = I;
// Return the instruction following the merged instruction, which is
// the instruction following our unmerged load. Unless that's the add/sub
// instruction we're merging, in which case it's the one after that.
if (++NextI == Update)
++NextI;
int Value = Update->getOperand(2).getImm();
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
assert(AArch64_AM::getShiftValue(Update->getOperand(3).getImm()) == 0 &&
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
"Can't merge 1 << 12 offset into pre-/post-indexed load / store");
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
if (Update->getOpcode() == AArch64::SUBXri)
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
Value = -Value;
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
unsigned NewOpc = IsPreIdx ? getPreIndexedOpcode(I->getOpcode())
: getPostIndexedOpcode(I->getOpcode());
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
MachineInstrBuilder MIB;
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (!isPairedLdSt(*I)) {
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// Non-paired instruction.
MIB = BuildMI(*I->getParent(), I, I->getDebugLoc(), TII->get(NewOpc))
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
.addOperand(getLdStRegOp(*Update))
.addOperand(getLdStRegOp(*I))
.addOperand(getLdStBaseOp(*I))
[AArch64] Set MMOs on pre- and post-index instructions. Without the MMOs the MI scheduler is unable to reason about the dependencies of these instructions. llvm-svn: 259052
2016-01-28 23:38:24 +08:00
.addImm(Value)
.setMemRefs(I->memoperands_begin(), I->memoperands_end());
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
} else {
// Paired instruction.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
int Scale = getMemScale(*I);
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
MIB = BuildMI(*I->getParent(), I, I->getDebugLoc(), TII->get(NewOpc))
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
.addOperand(getLdStRegOp(*Update))
.addOperand(getLdStRegOp(*I, 0))
.addOperand(getLdStRegOp(*I, 1))
.addOperand(getLdStBaseOp(*I))
[AArch64] Set MMOs on pre- and post-index instructions. Without the MMOs the MI scheduler is unable to reason about the dependencies of these instructions. llvm-svn: 259052
2016-01-28 23:38:24 +08:00
.addImm(Value / Scale)
.setMemRefs(I->memoperands_begin(), I->memoperands_end());
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
(void)MIB;
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
if (IsPreIdx)
DEBUG(dbgs() << "Creating pre-indexed load/store.");
else
DEBUG(dbgs() << "Creating post-indexed load/store.");
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
DEBUG(dbgs() << " Replacing instructions:\n ");
DEBUG(I->print(dbgs()));
DEBUG(dbgs() << " ");
DEBUG(Update->print(dbgs()));
DEBUG(dbgs() << " with instruction:\n ");
DEBUG(((MachineInstr *)MIB)->print(dbgs()));
DEBUG(dbgs() << "\n");
// Erase the old instructions for the block.
I->eraseFromParent();
Update->eraseFromParent();
return NextI;
}
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
bool AArch64LoadStoreOpt::isMatchingUpdateInsn(MachineInstr &MemMI,
MachineInstr &MI,
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
unsigned BaseReg, int Offset) {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
switch (MI.getOpcode()) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
default:
break;
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::SUBXri:
case AArch64::ADDXri:
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Make sure it's a vanilla immediate operand, not a relocation or
// anything else we can't handle.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (!MI.getOperand(2).isImm())
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
break;
// Watch out for 1 << 12 shifted value.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (AArch64_AM::getShiftValue(MI.getOperand(3).getImm()))
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
break;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// The update instruction source and destination register must be the
// same as the load/store base register.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (MI.getOperand(0).getReg() != BaseReg ||
MI.getOperand(1).getReg() != BaseReg)
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
break;
bool IsPairedInsn = isPairedLdSt(MemMI);
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
int UpdateOffset = MI.getOperand(2).getImm();
[AArch64LoadStoreOpt] Handle offsets correctly for post-indexed paired loads. Trunk would try to create something like "stp x9, x8, [x0], #512", which isn't actually a valid instruction. Differential revision: https://reviews.llvm.org/D23368 llvm-svn: 278559
2016-08-13 04:28:02 +08:00
if (MI.getOpcode() == AArch64::SUBXri)
UpdateOffset = -UpdateOffset;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// For non-paired load/store instructions, the immediate must fit in a
// signed 9-bit integer.
if (!IsPairedInsn && (UpdateOffset > 255 || UpdateOffset < -256))
break;
// For paired load/store instructions, the immediate must be a multiple of
// the scaling factor. The scaled offset must also fit into a signed 7-bit
// integer.
if (IsPairedInsn) {
[AArch64] Scale offsets by the size of the memory operation. NFC. The immediate in the load/store should be scaled by the size of the memory operation, not the size of the register being loaded/stored. This change gets us one step closer to forming LDPSW instructions. This change also enables pre- and post-indexing for halfword and byte loads and stores. llvm-svn: 248804
2015-09-30 00:07:32 +08:00
int Scale = getMemScale(MemMI);
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
if (UpdateOffset % Scale != 0)
break;
int ScaledOffset = UpdateOffset / Scale;
[AArch64LoadStoreOpt] Handle offsets correctly for post-indexed paired loads. Trunk would try to create something like "stp x9, x8, [x0], #512", which isn't actually a valid instruction. Differential revision: https://reviews.llvm.org/D23368 llvm-svn: 278559
2016-08-13 04:28:02 +08:00
if (ScaledOffset > 63 || ScaledOffset < -64)
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
break;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// If we have a non-zero Offset, we check that it matches the amount
// we're adding to the register.
[AArch64LoadStoreOpt] Handle offsets correctly for post-indexed paired loads. Trunk would try to create something like "stp x9, x8, [x0], #512", which isn't actually a valid instruction. Differential revision: https://reviews.llvm.org/D23368 llvm-svn: 278559
2016-08-13 04:28:02 +08:00
if (!Offset || Offset == UpdateOffset)
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
return true;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
break;
}
return false;
}
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
MachineBasicBlock::iterator AArch64LoadStoreOpt::findMatchingUpdateInsnForward(
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
MachineBasicBlock::iterator I, int UnscaledOffset, unsigned Limit) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator E = I->getParent()->end();
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr &MemMI = *I;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator MBBI = I;
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
unsigned BaseReg = getLdStBaseOp(MemMI).getReg();
[AArch64] Rename variable to improve readability. NFC. llvm-svn: 249008
2015-10-01 21:33:31 +08:00
int MIUnscaledOffset = getLdStOffsetOp(MemMI).getImm() * getMemScale(MemMI);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Hoist commonly failing check. NFC. llvm-svn: 249011
2015-10-01 21:43:05 +08:00
// Scan forward looking for post-index opportunities. Updating instructions
// can't be formed if the memory instruction doesn't have the offset we're
// looking for.
if (MIUnscaledOffset != UnscaledOffset)
return E;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// If the base register overlaps a destination register, we can't
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// merge the update.
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
bool IsPairedInsn = isPairedLdSt(MemMI);
for (unsigned i = 0, e = IsPairedInsn ? 2 : 1; i != e; ++i) {
unsigned DestReg = getLdStRegOp(MemMI, i).getReg();
if (DestReg == BaseReg || TRI->isSubRegister(BaseReg, DestReg))
return E;
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Track which registers have been modified and used between the first insn
// (inclusive) and the second insn.
[AArch64] Allocate the modified and used regs only once per function. llvm-svn: 259510
2016-02-02 23:02:30 +08:00
ModifiedRegs.reset();
UsedRegs.reset();
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
++MBBI;
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
for (unsigned Count = 0; MBBI != E && Count < Limit; ++MBBI) {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr &MI = *MBBI;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Load/store opt: Don't count transient instructions towards search limits. Summary: This change also changes findMatchingInsn and findMatchingUpdateInsnForward to take DBG_VALUE opcodes into account when tracking register defs and uses, which could potentially inhibit these optimizations in the presence of debug information. Reviewers: mcrosier Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: https://reviews.llvm.org/D22582 llvm-svn: 276293
2016-07-21 23:20:25 +08:00
// Don't count transient instructions towards the search limit since there
// may be different numbers of them if e.g. debug information is present.
if (!MI.isTransient())
++Count;
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// If we found a match, return it.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (isMatchingUpdateInsn(*I, MI, BaseReg, UnscaledOffset))
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return MBBI;
// Update the status of what the instruction clobbered and used.
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
// Otherwise, if the base register is used or modified, we have no match, so
// return early.
if (ModifiedRegs[BaseReg] || UsedRegs[BaseReg])
return E;
}
return E;
}
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
MachineBasicBlock::iterator AArch64LoadStoreOpt::findMatchingUpdateInsnBackward(
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
MachineBasicBlock::iterator I, unsigned Limit) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator B = I->getParent()->begin();
MachineBasicBlock::iterator E = I->getParent()->end();
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr &MemMI = *I;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator MBBI = I;
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
unsigned BaseReg = getLdStBaseOp(MemMI).getReg();
int Offset = getLdStOffsetOp(MemMI).getImm();
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// If the load/store is the first instruction in the block, there's obviously
// not any matching update. Ditto if the memory offset isn't zero.
if (MBBI == B || Offset != 0)
return E;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// If the base register overlaps a destination register, we can't
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// merge the update.
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
bool IsPairedInsn = isPairedLdSt(MemMI);
for (unsigned i = 0, e = IsPairedInsn ? 2 : 1; i != e; ++i) {
unsigned DestReg = getLdStRegOp(MemMI, i).getReg();
if (DestReg == BaseReg || TRI->isSubRegister(BaseReg, DestReg))
return E;
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Track which registers have been modified and used between the first insn
// (inclusive) and the second insn.
[AArch64] Allocate the modified and used regs only once per function. llvm-svn: 259510
2016-02-02 23:02:30 +08:00
ModifiedRegs.reset();
UsedRegs.reset();
[AArch64] AArch64LoadStoreOptimizer: fix bug in pre-inc check iterator Summary: Fix case where a pre-inc/dec load/store would not be formed if the add/sub that forms the inc/dec part of the operation was the first instruction in the block being examined. Reviewers: mcrosier, jmolloy, t.p.northover, junbuml Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16785 llvm-svn: 260275
2016-02-10 04:47:21 +08:00
unsigned Count = 0;
do {
--MBBI;
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr &MI = *MBBI;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Load/store opt: Don't count transient instructions towards search limits. Summary: This change also changes findMatchingInsn and findMatchingUpdateInsnForward to take DBG_VALUE opcodes into account when tracking register defs and uses, which could potentially inhibit these optimizations in the presence of debug information. Reviewers: mcrosier Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: https://reviews.llvm.org/D22582 llvm-svn: 276293
2016-07-21 23:20:25 +08:00
// Don't count transient instructions towards the search limit since there
// may be different numbers of them if e.g. debug information is present.
if (!MI.isTransient())
[AArch64] AArch64LoadStoreOptimizer: fix bug in pre-inc check iterator Summary: Fix case where a pre-inc/dec load/store would not be formed if the add/sub that forms the inc/dec part of the operation was the first instruction in the block being examined. Reviewers: mcrosier, jmolloy, t.p.northover, junbuml Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16785 llvm-svn: 260275
2016-02-10 04:47:21 +08:00
++Count;
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// If we found a match, return it.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (isMatchingUpdateInsn(*I, MI, BaseReg, Offset))
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return MBBI;
// Update the status of what the instruction clobbered and used.
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
// Otherwise, if the base register is used or modified, we have no match, so
// return early.
if (ModifiedRegs[BaseReg] || UsedRegs[BaseReg])
return E;
[AArch64] AArch64LoadStoreOptimizer: fix bug in pre-inc check iterator Summary: Fix case where a pre-inc/dec load/store would not be formed if the add/sub that forms the inc/dec part of the operation was the first instruction in the block being examined. Reviewers: mcrosier, jmolloy, t.p.northover, junbuml Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16785 llvm-svn: 260275
2016-02-10 04:47:21 +08:00
} while (MBBI != B && Count < Limit);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return E;
}
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
bool AArch64LoadStoreOpt::tryToPromoteLoadFromStore(
MachineBasicBlock::iterator &MBBI) {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr &MI = *MBBI;
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
// If this is a volatile load, don't mess with it.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (MI.hasOrderedMemoryRef())
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
return false;
// Make sure this is a reg+imm.
// FIXME: It is possible to extend it to handle reg+reg cases.
if (!getLdStOffsetOp(MI).isImm())
return false;
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
// Look backward up to LdStLimit instructions.
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
MachineBasicBlock::iterator StoreI;
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
if (findMatchingStore(MBBI, LdStLimit, StoreI)) {
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
++NumLoadsFromStoresPromoted;
// Promote the load. Keeping the iterator straight is a
// pain, so we let the merge routine tell us what the next instruction
// is after it's done mucking about.
MBBI = promoteLoadFromStore(MBBI, StoreI);
return true;
}
return false;
}
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
// Find narrow loads that can be converted into a single wider load with
// bitfield extract instructions. Also merge adjacent zero stores into a wider
// store.
bool AArch64LoadStoreOpt::tryToMergeLdStInst(
MachineBasicBlock::iterator &MBBI) {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
assert((isNarrowLoad(*MBBI) || isPromotableZeroStoreOpcode(*MBBI)) &&
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
"Expected narrow op.");
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr &MI = *MBBI;
MachineBasicBlock::iterator E = MI.getParent()->end();
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (!TII->isCandidateToMergeOrPair(MI))
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
return false;
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
// For promotable zero stores, the stored value should be WZR.
if (isPromotableZeroStoreOpcode(MI) &&
getLdStRegOp(MI).getReg() != AArch64::WZR)
[AArch64] This check is specific to merging instructions. NFC. llvm-svn: 260283
2016-02-10 05:20:12 +08:00
return false;
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
// Look ahead up to LdStLimit instructions for a mergable instruction.
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
LdStPairFlags Flags;
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
MachineBasicBlock::iterator MergeMI =
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
findMatchingInsn(MBBI, Flags, LdStLimit, /* FindNarrowMerge = */ true);
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
if (MergeMI != E) {
[AArch64] Refactoring aarch64-ldst-opt. NCF. Summary : * Rename isSmallTypeLdMerge() to isNarrowLoad(). * Rename NumSmallTypeMerged to NumNarrowTypePromoted. * Use Subtarget defined as a member variable. llvm-svn: 253587
2015-11-20 02:41:27 +08:00
if (isNarrowLoad(MI)) {
++NumNarrowLoadsPromoted;
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
} else if (isPromotableZeroStoreInst(MI)) {
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
++NumZeroStoresPromoted;
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
}
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
// Keeping the iterator straight is a pain, so we let the merge routine tell
// us what the next instruction is after it's done mucking about.
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MBBI = mergeNarrowInsns(MBBI, MergeMI, Flags);
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
return true;
}
return false;
}
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
// Find loads and stores that can be merged into a single load or store pair
// instruction.
bool AArch64LoadStoreOpt::tryToPairLdStInst(MachineBasicBlock::iterator &MBBI) {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr &MI = *MBBI;
MachineBasicBlock::iterator E = MI.getParent()->end();
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (!TII->isCandidateToMergeOrPair(MI))
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
return false;
[AArch64] This bit of logic is specific to pairing. NFC. llvm-svn: 260383
2016-02-10 23:52:46 +08:00
// Early exit if the offset is not possible to match. (6 bits of positive
// range, plus allow an extra one in case we find a later insn that matches
// with Offset-1)
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
bool IsUnscaled = TII->isUnscaledLdSt(MI);
[AArch64] This bit of logic is specific to pairing. NFC. llvm-svn: 260383
2016-02-10 23:52:46 +08:00
int Offset = getLdStOffsetOp(MI).getImm();
int OffsetStride = IsUnscaled ? getMemScale(MI) : 1;
if (!inBoundsForPair(IsUnscaled, Offset, OffsetStride))
return false;
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
// Look ahead up to LdStLimit instructions for a pairable instruction.
LdStPairFlags Flags;
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
MachineBasicBlock::iterator Paired =
findMatchingInsn(MBBI, Flags, LdStLimit, /* FindNarrowMerge = */ false);
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
if (Paired != E) {
++NumPairCreated;
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
if (TII->isUnscaledLdSt(MI))
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
++NumUnscaledPairCreated;
// Keeping the iterator straight is a pain, so we let the merge routine tell
// us what the next instruction is after it's done mucking about.
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
MBBI = mergePairedInsns(MBBI, Paired, Flags);
return true;
}
return false;
}
[AArch64]Enable the narrow ld promotion only on profitable microarchitectures The benefit from converting narrow loads into a wider load (r251438) could be micro-architecturally dependent, as it assumes that a single load with two bitfield extracts is cheaper than two narrow loads. Currently, this conversion is enabled only in cortex-a57 on which performance benefits were verified. llvm-svn: 252316
2015-11-07 00:27:47 +08:00
bool AArch64LoadStoreOpt::optimizeBlock(MachineBasicBlock &MBB,
bool enableNarrowLdOpt) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
bool Modified = false;
Move comments a bit closer to associated code. NFC. llvm-svn: 259411
2016-02-02 05:38:31 +08:00
// Four tranformations to do here:
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
// 1) Find loads that directly read from stores and promote them by
// replacing with mov instructions. If the store is wider than the load,
// the load will be replaced with a bitfield extract.
// e.g.,
// str w1, [x0, #4]
// ldrh w2, [x0, #6]
// ; becomes
// str w1, [x0, #4]
Untabify. llvm-svn: 273129
2016-06-20 08:37:41 +08:00
// lsr w2, w1, #16
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
for (MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
MBBI != E;) {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr &MI = *MBBI;
switch (MI.getOpcode()) {
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
default:
// Just move on to the next instruction.
++MBBI;
break;
// Scaled instructions.
case AArch64::LDRBBui:
case AArch64::LDRHHui:
case AArch64::LDRWui:
case AArch64::LDRXui:
// Unscaled instructions.
case AArch64::LDURBBi:
case AArch64::LDURHHi:
case AArch64::LDURWi:
case AArch64::LDURXi: {
if (tryToPromoteLoadFromStore(MBBI)) {
Modified = true;
break;
}
++MBBI;
break;
}
}
}
Move comments a bit closer to associated code. NFC. llvm-svn: 259411
2016-02-02 05:38:31 +08:00
// 2) Find narrow loads that can be converted into a single wider load
// with bitfield extract instructions.
// e.g.,
// ldrh w0, [x2]
// ldrh w1, [x2, #2]
// ; becomes
// ldr w0, [x2]
// ubfx w1, w0, #16, #16
// and w0, w0, #ffff
[AArch64] Refactoring aarch64-ldst-opt. NCF. Remove narrow load / store instructions from getMatchingPairOpcode(), and add getMatchingWideOpcode(). llvm-svn: 259914
2016-02-06 04:02:03 +08:00
//
// Also merge adjacent zero stores into a wider store.
// e.g.,
// strh wzr, [x0]
// strh wzr, [x0, #2]
// ; becomes
// str wzr, [x0]
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
for (MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
[AArch64]Enable the narrow ld promotion only on profitable microarchitectures The benefit from converting narrow loads into a wider load (r251438) could be micro-architecturally dependent, as it assumes that a single load with two bitfield extracts is cheaper than two narrow loads. Currently, this conversion is enabled only in cortex-a57 on which performance benefits were verified. llvm-svn: 252316
2015-11-07 00:27:47 +08:00
enableNarrowLdOpt && MBBI != E;) {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr &MI = *MBBI;
unsigned Opc = MI.getOpcode();
[AArch64] Decouple zero store promotion from narrow ld merge. NFC. Summary: This change refactors to decouple the zero store promotion from the narrow ld merge and add a flag (enable-narrow-ld-merge=true) to control the narrow ld merge optimization. Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D19885 llvm-svn: 268744
2016-05-06 23:08:57 +08:00
if (isPromotableZeroStoreOpcode(Opc) ||
(EnableNarrowLdMerge && isNarrowLoad(Opc))) {
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
if (tryToMergeLdStInst(MBBI)) {
Modified = true;
[AArch64] Decouple zero store promotion from narrow ld merge. NFC. Summary: This change refactors to decouple the zero store promotion from the narrow ld merge and add a flag (enable-narrow-ld-merge=true) to control the narrow ld merge optimization. Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D19885 llvm-svn: 268744
2016-05-06 23:08:57 +08:00
} else
++MBBI;
} else
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
++MBBI;
}
[AArch64] Decouple zero store promotion from narrow ld merge. NFC. Summary: This change refactors to decouple the zero store promotion from the narrow ld merge and add a flag (enable-narrow-ld-merge=true) to control the narrow ld merge optimization. Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D19885 llvm-svn: 268744
2016-05-06 23:08:57 +08:00
Move comments a bit closer to associated code. NFC. llvm-svn: 259411
2016-02-02 05:38:31 +08:00
// 3) Find loads and stores that can be merged into a single load or store
// pair instruction.
// e.g.,
// ldr x0, [x2]
// ldr x1, [x2, #8]
// ; becomes
// ldp x0, x1, [x2]
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
for (MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
MBBI != E;) {
[AArch64] Re-factor code shared by AArch64LoadStoreOpt and AArch64InstrInfo. This re-factoring could cause the following slight changes in generated code, though none were observed during testing: - MachineScheduler could decide not to cluster some loads/stores if there are other load/stores with non-pairable opcodes that have the same base register and offset as a pairable set of load/stores. One case of different MachineScheduler pairing did show up in my testing, but it wasn't due to this issue, but due BaseMemOpClusterMutation::clusterNeighboringMemOps() being unstable w.r.t. the order it considers memory operations. See PR28942. - The ImplicitNullChecks optimization could be done for more load/store opcodes. This optimization isn't done for C/C++ code, so it didn't show up in my testing. Reviewers: mcrosier, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: https://reviews.llvm.org/D23365 llvm-svn: 278515
2016-08-12 23:26:00 +08:00
if (TII->isPairableLdStInst(*MBBI) && tryToPairLdStInst(MBBI))
Modified = true;
else
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
++MBBI;
}
Move comments a bit closer to associated code. NFC. llvm-svn: 259411
2016-02-02 05:38:31 +08:00
// 4) Find base register updates that can be merged into the load or store
// as a base-reg writeback.
// e.g.,
// ldr x0, [x2]
// add x2, x2, #4
// ; becomes
// ldr x0, [x2], #4
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
for (MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
MBBI != E;) {
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
MachineInstr &MI = *MBBI;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Do update merging. It's simpler to keep this separate from the above
Move comments a bit closer to associated code. NFC. llvm-svn: 259411
2016-02-02 05:38:31 +08:00
// switchs, though not strictly necessary.
AArch64: Avoid implicit iterator conversions, NFC Avoid implicit conversions from MachineInstrBundleInstr to MachineInstr* in the AArch64 backend, mainly by preferring MachineInstr& over MachineInstr* when a pointer isn't nullable. llvm-svn: 274924
2016-07-09 04:29:42 +08:00
unsigned Opc = MI.getOpcode();
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
switch (Opc) {
default:
// Just move on to the next instruction.
++MBBI;
break;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// Scaled instructions.
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::STRSui:
case AArch64::STRDui:
case AArch64::STRQui:
case AArch64::STRXui:
case AArch64::STRWui:
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::STRHHui:
case AArch64::STRBBui:
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::LDRSui:
case AArch64::LDRDui:
case AArch64::LDRQui:
case AArch64::LDRXui:
case AArch64::LDRWui:
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::LDRHHui:
case AArch64::LDRBBui:
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// Unscaled instructions.
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::STURSi:
case AArch64::STURDi:
case AArch64::STURQi:
case AArch64::STURWi:
case AArch64::STURXi:
case AArch64::LDURSi:
case AArch64::LDURDi:
case AArch64::LDURQi:
case AArch64::LDURWi:
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
case AArch64::LDURXi:
// Paired instructions.
case AArch64::LDPSi:
[AArch64] Add support for pre- and post-index LDPSWs. llvm-svn: 248825
2015-09-30 04:39:55 +08:00
case AArch64::LDPSWi:
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
case AArch64::LDPDi:
case AArch64::LDPQi:
case AArch64::LDPWi:
case AArch64::LDPXi:
case AArch64::STPSi:
case AArch64::STPDi:
case AArch64::STPQi:
case AArch64::STPWi:
case AArch64::STPXi: {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Make sure this is a reg+imm (as opposed to an address reloc).
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
if (!getLdStOffsetOp(MI).isImm()) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
++MBBI;
break;
}
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// Look forward to try to form a post-index instruction. For example,
// ldr x0, [x20]
// add x20, x20, #32
// merged into:
// ldr x0, [x20], #32
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator Update =
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
findMatchingUpdateInsnForward(MBBI, 0, UpdateLimit);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
if (Update != E) {
// Merge the update into the ld/st.
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
MBBI = mergeUpdateInsn(MBBI, Update, /*IsPreIdx=*/false);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
Modified = true;
++NumPostFolded;
break;
}
// Don't know how to handle pre/post-index versions, so move to the next
// instruction.
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
if (TII->isUnscaledLdSt(Opc)) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
++MBBI;
break;
}
// Look back to try to find a pre-index instruction. For example,
// add x0, x0, #8
// ldr x1, [x0]
// merged into:
// ldr x1, [x0, #8]!
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
Update = findMatchingUpdateInsnBackward(MBBI, UpdateLimit);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
if (Update != E) {
// Merge the update into the ld/st.
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
MBBI = mergeUpdateInsn(MBBI, Update, /*IsPreIdx=*/true);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
Modified = true;
++NumPreFolded;
break;
}
[AArch64] Update comment to reflect reality. llvm-svn: 249007
2015-10-01 21:09:44 +08:00
// The immediate in the load/store is scaled by the size of the memory
// operation. The immediate in the add we're looking for,
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// however, is not, so adjust here.
[AArch64] Rename variable to improve readability. NFC. llvm-svn: 249008
2015-10-01 21:33:31 +08:00
int UnscaledOffset = getLdStOffsetOp(MI).getImm() * getMemScale(MI);
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Look forward to try to find a post-index instruction. For example,
// ldr x1, [x0, #64]
// add x0, x0, #64
// merged into:
[ARM64] Fix wrong comment in load/store optimization pass. ldr x1, [x0, #64] add x0, x0, #64 -> ldr x1, [x0], #64 is not a valid transformation, the correct transformation (and what the code actually does) is: ldr x1, [x0, #64] add x0, x0, #64 -> ldr x1, [x0, #64]! llvm-svn: 208998
2014-05-17 00:50:13 +08:00
// ldr x1, [x0, #64]!
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
Update = findMatchingUpdateInsnForward(MBBI, UnscaledOffset, UpdateLimit);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
if (Update != E) {
// Merge the update into the ld/st.
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
MBBI = mergeUpdateInsn(MBBI, Update, /*IsPreIdx=*/true);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
Modified = true;
++NumPreFolded;
break;
}
// Nothing found. Just move to the next instruction.
++MBBI;
break;
}
}
}
return Modified;
}
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
bool AArch64LoadStoreOpt::runOnMachineFunction(MachineFunction &Fn) {
Add optimization bisect opt-in calls for AArch64 passes Differential Revision: http://reviews.llvm.org/D19394 llvm-svn: 267479
2016-04-26 05:58:52 +08:00
if (skipFunction(*Fn.getFunction()))
return false;
[AArch64] Fix halfword load merging for big-endian targets For big-endian targets, when we merge two halfword loads into a word load, the order of the halfwords in the loaded value is reversed compared to little-endian, so the load-store optimiser needs to swap the destination registers. This does not affect merging of two word loads, as we use ldp, which treats the memory as two separate 32-bit words. llvm-svn: 252597
2015-11-10 19:04:18 +08:00
Subtarget = &static_cast<const AArch64Subtarget &>(Fn.getSubtarget());
TII = static_cast<const AArch64InstrInfo *>(Subtarget->getInstrInfo());
TRI = Subtarget->getRegisterInfo();
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Allocate the modified and used regs only once per function. llvm-svn: 259510
2016-02-02 23:02:30 +08:00
// Resize the modified and used register bitfield trackers. We do this once
// per function and then clear the bitfield each time we optimize a load or
// store.
ModifiedRegs.resize(TRI->getNumRegs());
UsedRegs.resize(TRI->getNumRegs());
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
bool Modified = false;
AArch64: Do not test for CPUs, use SubtargetFeatures Testing for specific CPUs has a number of problems, better use subtarget features: - When some tweak is added for a specific CPU it is often desirable for the next version of that CPU as well, yet we often forget to add it. - It is hard to keep track of checks scattered around the target code; Declaring all target specifics together with the CPU in the tablegen file is a clear representation. - Subtarget features can be tweaked from the command line. To discourage people from using CPU checks in the future I removed the isCortexXX(), isCyclone(), ... functions. I added an getProcFamily() function for exceptional circumstances but made it clear in the comment that usage is discouraged. Reformat feature list in AArch64.td to have 1 feature per line in alphabetical order to simplify merging and sorting for out of tree tweaks. No functional change intended. Differential Revision: http://reviews.llvm.org/D20762 llvm-svn: 271555
2016-06-03 02:03:53 +08:00
bool enableNarrowLdOpt =
Subtarget->mergeNarrowLoads() && !Subtarget->requiresStrictAlign();
[C++11,ARM64] Range based for loops in load/store pair optimizer. No functional change intended. llvm-svn: 205444
2014-04-03 02:00:53 +08:00
for (auto &MBB : Fn)
[AArch64]Enable the narrow ld promotion only on profitable microarchitectures The benefit from converting narrow loads into a wider load (r251438) could be micro-architecturally dependent, as it assumes that a single load with two bitfield extracts is cheaper than two narrow loads. Currently, this conversion is enabled only in cortex-a57 on which performance benefits were verified. llvm-svn: 252316
2015-11-07 00:27:47 +08:00
Modified |= optimizeBlock(MBB, enableNarrowLdOpt);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return Modified;
}
// FIXME: Do we need/want a pre-alloc pass like ARM has to try to keep
// loads and stores near one another?
[AArch64] Add a FIXME about invalid KILL markers after the ld/st opt pass. llvm-svn: 260264
2016-02-10 03:42:19 +08:00
// FIXME: When pairing store instructions it's very possible for this pass to
// hoist a store with a KILL marker above another use (without a KILL marker).
// The resulting IR is invalid, but nothing uses the KILL markers after this
// pass, so it's never caused a problem in practice.
Update comment. NFC. llvm-svn: 244038
2015-08-05 20:40:13 +08:00
/// createAArch64LoadStoreOptimizationPass - returns an instance of the
/// load / store optimization pass.
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
FunctionPass *llvm::createAArch64LoadStoreOptimizationPass() {
return new AArch64LoadStoreOpt();
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}