maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/llvm/lib/Target/AArch64/AArch64LoadStoreOptimizer.cpp

1963 lines
66 KiB
C++
Raw Normal View History

AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
//=- AArch64LoadStoreOptimizer.cpp - AArch64 load/store opt. pass -*- C++ -*-=//
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
//
// The LLVM Compiler Infrastructure
//
// This file is distributed under the University of Illinois Open Source
// License. See LICENSE.TXT for details.
//
//===----------------------------------------------------------------------===//
//
// This file contains a pass that performs load / store related peephole
// optimizations. This pass should be run after register allocation.
//
//===----------------------------------------------------------------------===//
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
#include "AArch64InstrInfo.h"
Remove the TargetMachine forwards for TargetSubtargetInfo based information and update all callers. No functional change. llvm-svn: 214781
2014-08-05 05:25:23 +08:00
#include "AArch64Subtarget.h"
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
#include "MCTargetDesc/AArch64AddressingModes.h"
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
#include "llvm/ADT/BitVector.h"
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
#include "llvm/ADT/SmallVector.h"
Run sort_includes.py on the AArch64 backend. No functionality change. llvm-svn: 213938
2014-07-25 19:42:14 +08:00
#include "llvm/ADT/Statistic.h"
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
#include "llvm/CodeGen/MachineBasicBlock.h"
#include "llvm/CodeGen/MachineFunctionPass.h"
#include "llvm/CodeGen/MachineInstr.h"
#include "llvm/CodeGen/MachineInstrBuilder.h"
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Debug.h"
#include "llvm/Support/ErrorHandling.h"
#include "llvm/Support/raw_ostream.h"
Run sort_includes.py on the AArch64 backend. No functionality change. llvm-svn: 213938
2014-07-25 19:42:14 +08:00
#include "llvm/Target/TargetInstrInfo.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Target/TargetRegisterInfo.h"
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
using namespace llvm;
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
#define DEBUG_TYPE "aarch64-ldst-opt"
[Modules] Fix potential ODR violations by sinking the DEBUG_TYPE definition below all of the header #include lines, lib/Target/... edition. llvm-svn: 206842
2014-04-22 10:41:26 +08:00
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
STATISTIC(NumPairCreated, "Number of load/store pair instructions generated");
STATISTIC(NumPostFolded, "Number of post-index updates folded");
STATISTIC(NumPreFolded, "Number of pre-index updates folded");
STATISTIC(NumUnscaledPairCreated,
"Number of load/store from unscaled generated");
[AArch64] Refactoring aarch64-ldst-opt. NCF. Summary : * Rename isSmallTypeLdMerge() to isNarrowLoad(). * Rename NumSmallTypeMerged to NumNarrowTypePromoted. * Use Subtarget defined as a member variable. llvm-svn: 253587
2015-11-20 02:41:27 +08:00
STATISTIC(NumNarrowLoadsPromoted, "Number of narrow loads promoted");
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
STATISTIC(NumZeroStoresPromoted, "Number of narrow zero stores promoted");
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
STATISTIC(NumLoadsFromStoresPromoted, "Number of loads from stores promoted");
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
// The LdStLimit limits how far we search for load/store pairs.
static cl::opt<unsigned> LdStLimit("aarch64-load-store-scan-limit",
[AArch64] clang-format the load/store optimizer. No change in functionality. llvm-svn: 210182
2014-06-04 20:40:35 +08:00
cl::init(20), cl::Hidden);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
// The UpdateLimit limits how far we search for update instructions when we form
// pre-/post-index instructions.
static cl::opt<unsigned> UpdateLimit("aarch64-update-scan-limit", cl::init(100),
cl::Hidden);
[AArch64] Register (existing) AArch64LoadStoreOpt pass with LLVM pass manager. Summary: Among other things, this allows -print-after-all/-print-before-all to dump IR around this pass. This is the AArch64 version of r243052. llvm-svn: 244041
2015-08-05 21:44:51 +08:00
namespace llvm {
void initializeAArch64LoadStoreOptPass(PassRegistry &);
}
#define AARCH64_LOAD_STORE_OPT_NAME "AArch64 load / store optimization pass"
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
namespace {
[AArch64] Simplify the passing of arguments. NFC. This is setup for future work planned for the AArch64 Load/Store Opt pass. llvm-svn: 242810
2015-07-22 01:42:04 +08:00
typedef struct LdStPairFlags {
// If a matching instruction is found, MergeForward is set to true if the
// merge is to remove the first instruction and replace the second with
// a pair-wise insn, and false if the reverse is true.
bool MergeForward;
// SExtIdx gives the index of the result of the load pair that must be
// extended. The value of SExtIdx assumes that the paired load produces the
// value in this order: (I, returned iterator), i.e., -1 means no value has
// to be extended, 0 means I, and 1 means the returned iterator.
int SExtIdx;
LdStPairFlags() : MergeForward(false), SExtIdx(-1) {}
void setMergeForward(bool V = true) { MergeForward = V; }
bool getMergeForward() const { return MergeForward; }
void setSExtIdx(int V) { SExtIdx = V; }
int getSExtIdx() const { return SExtIdx; }
} LdStPairFlags;
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
struct AArch64LoadStoreOpt : public MachineFunctionPass {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
static char ID;
[AArch64]Enable the narrow ld promotion only on profitable microarchitectures The benefit from converting narrow loads into a wider load (r251438) could be micro-architecturally dependent, as it assumes that a single load with two bitfield extracts is cheaper than two narrow loads. Currently, this conversion is enabled only in cortex-a57 on which performance benefits were verified. llvm-svn: 252316
2015-11-07 00:27:47 +08:00
AArch64LoadStoreOpt() : MachineFunctionPass(ID) {
[AArch64] Register (existing) AArch64LoadStoreOpt pass with LLVM pass manager. Summary: Among other things, this allows -print-after-all/-print-before-all to dump IR around this pass. This is the AArch64 version of r243052. llvm-svn: 244041
2015-08-05 21:44:51 +08:00
initializeAArch64LoadStoreOptPass(*PassRegistry::getPassRegistry());
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
const AArch64InstrInfo *TII;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
const TargetRegisterInfo *TRI;
[AArch64] Fix halfword load merging for big-endian targets For big-endian targets, when we merge two halfword loads into a word load, the order of the halfwords in the loaded value is reversed compared to little-endian, so the load-store optimiser needs to swap the destination registers. This does not affect merging of two word loads, as we use ldp, which treats the memory as two separate 32-bit words. llvm-svn: 252597
2015-11-10 19:04:18 +08:00
const AArch64Subtarget *Subtarget;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Allocate the modified and used regs only once per function. llvm-svn: 259510
2016-02-02 23:02:30 +08:00
// Track which registers have been modified and used.
BitVector ModifiedRegs, UsedRegs;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Scan the instructions looking for a load/store that can be combined
// with the current instruction into a load/store pair.
// Return the matching instruction if one is found, else MBB->end().
MachineBasicBlock::iterator findMatchingInsn(MachineBasicBlock::iterator I,
[AArch64] Simplify the passing of arguments. NFC. This is setup for future work planned for the AArch64 Load/Store Opt pass. llvm-svn: 242810
2015-07-22 01:42:04 +08:00
LdStPairFlags &Flags,
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
unsigned Limit,
bool FindNarrowMerge);
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
// Scan the instructions looking for a store that writes to the address from
// which the current load instruction reads. Return true if one is found.
bool findMatchingStore(MachineBasicBlock::iterator I, unsigned Limit,
MachineBasicBlock::iterator &StoreI);
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
// Merge the two instructions indicated into a wider instruction.
MachineBasicBlock::iterator
mergeNarrowInsns(MachineBasicBlock::iterator I,
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MachineBasicBlock::iterator MergeMI,
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
const LdStPairFlags &Flags);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Merge the two instructions indicated into a single pair-wise instruction.
MachineBasicBlock::iterator
mergePairedInsns(MachineBasicBlock::iterator I,
[AArch64] Simplify the passing of arguments. NFC. This is setup for future work planned for the AArch64 Load/Store Opt pass. llvm-svn: 242810
2015-07-22 01:42:04 +08:00
MachineBasicBlock::iterator Paired,
Follow up to r242810. NFC. llvm-svn: 242812
2015-07-22 01:47:56 +08:00
const LdStPairFlags &Flags);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
// Promote the load that reads directly from the address stored to.
MachineBasicBlock::iterator
promoteLoadFromStore(MachineBasicBlock::iterator LoadI,
MachineBasicBlock::iterator StoreI);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Scan the instruction list to find a base register update that can
// be combined with the current instruction (a load or store) using
// pre or post indexed addressing with writeback. Scan forwards.
MachineBasicBlock::iterator
[AArch64] Remove unused arguments. NFC. AFAICT, these have been unused since the initial backend import. llvm-svn: 258093
2016-01-19 05:56:40 +08:00
findMatchingUpdateInsnForward(MachineBasicBlock::iterator I,
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
int UnscaledOffset, unsigned Limit);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Scan the instruction list to find a base register update that can
// be combined with the current instruction (a load or store) using
// pre or post indexed addressing with writeback. Scan backwards.
MachineBasicBlock::iterator
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
findMatchingUpdateInsnBackward(MachineBasicBlock::iterator I, unsigned Limit);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// Find an instruction that updates the base register of the ld/st
// instruction.
bool isMatchingUpdateInsn(MachineInstr *MemMI, MachineInstr *MI,
unsigned BaseReg, int Offset);
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
// Merge a pre- or post-index base register update into a ld/st instruction.
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
mergeUpdateInsn(MachineBasicBlock::iterator I,
MachineBasicBlock::iterator Update, bool IsPreIdx);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
// Find and merge foldable ldr/str instructions.
bool tryToMergeLdStInst(MachineBasicBlock::iterator &MBBI);
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
// Find and pair ldr/str instructions.
bool tryToPairLdStInst(MachineBasicBlock::iterator &MBBI);
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
// Find and promote load instructions which read directly from store.
bool tryToPromoteLoadFromStore(MachineBasicBlock::iterator &MBBI);
[AArch64]Enable the narrow ld promotion only on profitable microarchitectures The benefit from converting narrow loads into a wider load (r251438) could be micro-architecturally dependent, as it assumes that a single load with two bitfield extracts is cheaper than two narrow loads. Currently, this conversion is enabled only in cortex-a57 on which performance benefits were verified. llvm-svn: 252316
2015-11-07 00:27:47 +08:00
// Check if converting two narrow loads into a single wider load with
// bitfield extracts could be enabled.
bool enableNarrowLdMerge(MachineFunction &Fn);
bool optimizeBlock(MachineBasicBlock &MBB, bool enableNarrowLdOpt);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[C++11] Add 'override' keywords and remove 'virtual'. Additionally add 'final' and leave 'virtual' on some methods that are marked virtual without overriding anything and have no obvious overrides themselves. ARM64 edition llvm-svn: 207509
2014-04-29 15:58:25 +08:00
bool runOnMachineFunction(MachineFunction &Fn) override;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[C++11] Add 'override' keywords and remove 'virtual'. Additionally add 'final' and leave 'virtual' on some methods that are marked virtual without overriding anything and have no obvious overrides themselves. ARM64 edition llvm-svn: 207509
2014-04-29 15:58:25 +08:00
const char *getPassName() const override {
[AArch64] Register (existing) AArch64LoadStoreOpt pass with LLVM pass manager. Summary: Among other things, this allows -print-after-all/-print-before-all to dump IR around this pass. This is the AArch64 version of r243052. llvm-svn: 244041
2015-08-05 21:44:51 +08:00
return AARCH64_LOAD_STORE_OPT_NAME;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
};
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
char AArch64LoadStoreOpt::ID = 0;
Add missing closing namespace comment. llvm-svn: 215402
2014-08-12 06:42:31 +08:00
} // namespace
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Register (existing) AArch64LoadStoreOpt pass with LLVM pass manager. Summary: Among other things, this allows -print-after-all/-print-before-all to dump IR around this pass. This is the AArch64 version of r243052. llvm-svn: 244041
2015-08-05 21:44:51 +08:00
INITIALIZE_PASS(AArch64LoadStoreOpt, "aarch64-ldst-opt",
AARCH64_LOAD_STORE_OPT_NAME, false, false)
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
static unsigned getBitExtrOpcode(MachineInstr *MI) {
switch (MI->getOpcode()) {
default:
llvm_unreachable("Unexpected opcode.");
case AArch64::LDRBBui:
case AArch64::LDURBBi:
case AArch64::LDRHHui:
case AArch64::LDURHHi:
return AArch64::UBFMWri;
case AArch64::LDRSBWui:
case AArch64::LDURSBWi:
case AArch64::LDRSHWui:
case AArch64::LDURSHWi:
return AArch64::SBFMWri;
}
}
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
static bool isNarrowStore(unsigned Opc) {
switch (Opc) {
default:
return false;
case AArch64::STRBBui:
case AArch64::STURBBi:
case AArch64::STRHHui:
case AArch64::STURHHi:
return true;
}
}
[AArch64] Refactoring aarch64-ldst-opt. NCF. Summary : * Rename isSmallTypeLdMerge() to isNarrowLoad(). * Rename NumSmallTypeMerged to NumNarrowTypePromoted. * Use Subtarget defined as a member variable. llvm-svn: 253587
2015-11-20 02:41:27 +08:00
static bool isNarrowLoad(unsigned Opc) {
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
switch (Opc) {
default:
return false;
case AArch64::LDRHHui:
case AArch64::LDURHHi:
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
case AArch64::LDRBBui:
case AArch64::LDURBBi:
case AArch64::LDRSHWui:
case AArch64::LDURSHWi:
case AArch64::LDRSBWui:
case AArch64::LDURSBWi:
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
return true;
}
}
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
[AArch64] Refactoring aarch64-ldst-opt. NCF. Summary : * Rename isSmallTypeLdMerge() to isNarrowLoad(). * Rename NumSmallTypeMerged to NumNarrowTypePromoted. * Use Subtarget defined as a member variable. llvm-svn: 253587
2015-11-20 02:41:27 +08:00
static bool isNarrowLoad(MachineInstr *MI) {
return isNarrowLoad(MI->getOpcode());
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
}
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
static bool isNarrowLoadOrStore(unsigned Opc) {
return isNarrowLoad(Opc) || isNarrowStore(Opc);
}
[AArch64] Scale offsets by the size of the memory operation. NFC. The immediate in the load/store should be scaled by the size of the memory operation, not the size of the register being loaded/stored. This change gets us one step closer to forming LDPSW instructions. This change also enables pre- and post-indexing for halfword and byte loads and stores. llvm-svn: 248804
2015-09-30 00:07:32 +08:00
// Scaling factor for unscaled load or store.
static int getMemScale(MachineInstr *MI) {
[AArch64] Use a static function and other minor cleanup for readability. NFC. llvm-svn: 244233
2015-08-07 01:37:18 +08:00
switch (MI->getOpcode()) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
default:
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
llvm_unreachable("Opcode has unknown scale!");
case AArch64::LDRBBui:
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
case AArch64::LDURBBi:
case AArch64::LDRSBWui:
case AArch64::LDURSBWi:
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::STRBBui:
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
case AArch64::STURBBi:
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
return 1;
case AArch64::LDRHHui:
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
case AArch64::LDURHHi:
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
case AArch64::LDRSHWui:
case AArch64::LDURSHWi:
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::STRHHui:
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
case AArch64::STURHHi:
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
return 2;
[AArch64] Remove some redundant cases. NFC. llvm-svn: 248800
2015-09-29 22:57:10 +08:00
case AArch64::LDRSui:
case AArch64::LDURSi:
case AArch64::LDRSWui:
case AArch64::LDURSWi:
case AArch64::LDRWui:
case AArch64::LDURWi:
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::STRSui:
case AArch64::STURSi:
case AArch64::STRWui:
case AArch64::STURWi:
[AArch64] Scale offsets by the size of the memory operation. NFC. The immediate in the load/store should be scaled by the size of the memory operation, not the size of the register being loaded/stored. This change gets us one step closer to forming LDPSW instructions. This change also enables pre- and post-indexing for halfword and byte loads and stores. llvm-svn: 248804
2015-09-30 00:07:32 +08:00
case AArch64::LDPSi:
[AArch64] Add support for pre- and post-index LDPSWs. llvm-svn: 248825
2015-09-30 04:39:55 +08:00
case AArch64::LDPSWi:
[AArch64] Scale offsets by the size of the memory operation. NFC. The immediate in the load/store should be scaled by the size of the memory operation, not the size of the register being loaded/stored. This change gets us one step closer to forming LDPSW instructions. This change also enables pre- and post-indexing for halfword and byte loads and stores. llvm-svn: 248804
2015-09-30 00:07:32 +08:00
case AArch64::LDPWi:
case AArch64::STPSi:
case AArch64::STPWi:
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return 4;
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::LDRDui:
case AArch64::LDURDi:
[AArch64] Remove some redundant cases. NFC. llvm-svn: 248800
2015-09-29 22:57:10 +08:00
case AArch64::LDRXui:
case AArch64::LDURXi:
case AArch64::STRDui:
case AArch64::STURDi:
case AArch64::STRXui:
case AArch64::STURXi:
[AArch64] Scale offsets by the size of the memory operation. NFC. The immediate in the load/store should be scaled by the size of the memory operation, not the size of the register being loaded/stored. This change gets us one step closer to forming LDPSW instructions. This change also enables pre- and post-indexing for halfword and byte loads and stores. llvm-svn: 248804
2015-09-30 00:07:32 +08:00
case AArch64::LDPDi:
case AArch64::LDPXi:
case AArch64::STPDi:
case AArch64::STPXi:
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return 8;
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::LDRQui:
case AArch64::LDURQi:
[AArch64] Remove some redundant cases. NFC. llvm-svn: 248800
2015-09-29 22:57:10 +08:00
case AArch64::STRQui:
case AArch64::STURQi:
[AArch64] Scale offsets by the size of the memory operation. NFC. The immediate in the load/store should be scaled by the size of the memory operation, not the size of the register being loaded/stored. This change gets us one step closer to forming LDPSW instructions. This change also enables pre- and post-indexing for halfword and byte loads and stores. llvm-svn: 248804
2015-09-30 00:07:32 +08:00
case AArch64::LDPQi:
case AArch64::STPQi:
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return 16;
}
}
[AArch64][LoadStoreOptimizer] Generate LDP + SXTW instead of LD[U]R + LD[U]RSW. Teach the load store optimizer how to sign extend a result of a load pair when it helps creating more pairs. The rational is that loads are more expensive than sign extensions, so if we gather some in one instruction this is better! <rdar://problem/20072968> llvm-svn: 231527
2015-03-07 06:42:10 +08:00
static unsigned getMatchingNonSExtOpcode(unsigned Opc,
bool *IsValidLdStrOpc = nullptr) {
if (IsValidLdStrOpc)
*IsValidLdStrOpc = true;
switch (Opc) {
default:
if (IsValidLdStrOpc)
*IsValidLdStrOpc = false;
return UINT_MAX;
case AArch64::STRDui:
case AArch64::STURDi:
case AArch64::STRQui:
case AArch64::STURQi:
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
case AArch64::STRBBui:
case AArch64::STURBBi:
case AArch64::STRHHui:
case AArch64::STURHHi:
[AArch64][LoadStoreOptimizer] Generate LDP + SXTW instead of LD[U]R + LD[U]RSW. Teach the load store optimizer how to sign extend a result of a load pair when it helps creating more pairs. The rational is that loads are more expensive than sign extensions, so if we gather some in one instruction this is better! <rdar://problem/20072968> llvm-svn: 231527
2015-03-07 06:42:10 +08:00
case AArch64::STRWui:
case AArch64::STURWi:
case AArch64::STRXui:
case AArch64::STURXi:
case AArch64::LDRDui:
case AArch64::LDURDi:
case AArch64::LDRQui:
case AArch64::LDURQi:
case AArch64::LDRWui:
case AArch64::LDURWi:
case AArch64::LDRXui:
case AArch64::LDURXi:
case AArch64::STRSui:
case AArch64::STURSi:
case AArch64::LDRSui:
case AArch64::LDURSi:
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
case AArch64::LDRHHui:
case AArch64::LDURHHi:
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
case AArch64::LDRBBui:
case AArch64::LDURBBi:
[AArch64][LoadStoreOptimizer] Generate LDP + SXTW instead of LD[U]R + LD[U]RSW. Teach the load store optimizer how to sign extend a result of a load pair when it helps creating more pairs. The rational is that loads are more expensive than sign extensions, so if we gather some in one instruction this is better! <rdar://problem/20072968> llvm-svn: 231527
2015-03-07 06:42:10 +08:00
return Opc;
case AArch64::LDRSWui:
return AArch64::LDRWui;
case AArch64::LDURSWi:
return AArch64::LDURWi;
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
case AArch64::LDRSBWui:
return AArch64::LDRBBui;
case AArch64::LDRSHWui:
return AArch64::LDRHHui;
case AArch64::LDURSBWi:
return AArch64::LDURBBi;
case AArch64::LDURSHWi:
return AArch64::LDURHHi;
[AArch64][LoadStoreOptimizer] Generate LDP + SXTW instead of LD[U]R + LD[U]RSW. Teach the load store optimizer how to sign extend a result of a load pair when it helps creating more pairs. The rational is that loads are more expensive than sign extensions, so if we gather some in one instruction this is better! <rdar://problem/20072968> llvm-svn: 231527
2015-03-07 06:42:10 +08:00
}
}
[AArch64] Refactoring aarch64-ldst-opt. NCF. Remove narrow load / store instructions from getMatchingPairOpcode(), and add getMatchingWideOpcode(). llvm-svn: 259914
2016-02-06 04:02:03 +08:00
static unsigned getMatchingWideOpcode(unsigned Opc) {
switch (Opc) {
default:
llvm_unreachable("Opcode has no wide equivalent!");
case AArch64::STRBBui:
return AArch64::STRHHui;
case AArch64::STRHHui:
return AArch64::STRWui;
case AArch64::STURBBi:
return AArch64::STURHHi;
case AArch64::STURHHi:
return AArch64::STURWi;
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
case AArch64::STURWi:
return AArch64::STURXi;
case AArch64::STRWui:
return AArch64::STRXui;
[AArch64] Refactoring aarch64-ldst-opt. NCF. Remove narrow load / store instructions from getMatchingPairOpcode(), and add getMatchingWideOpcode(). llvm-svn: 259914
2016-02-06 04:02:03 +08:00
case AArch64::LDRHHui:
case AArch64::LDRSHWui:
return AArch64::LDRWui;
case AArch64::LDURHHi:
case AArch64::LDURSHWi:
return AArch64::LDURWi;
case AArch64::LDRBBui:
case AArch64::LDRSBWui:
return AArch64::LDRHHui;
case AArch64::LDURBBi:
case AArch64::LDURSBWi:
return AArch64::LDURHHi;
}
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
static unsigned getMatchingPairOpcode(unsigned Opc) {
switch (Opc) {
default:
llvm_unreachable("Opcode has no pairwise equivalent!");
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::STRSui:
case AArch64::STURSi:
return AArch64::STPSi;
case AArch64::STRDui:
case AArch64::STURDi:
return AArch64::STPDi;
case AArch64::STRQui:
case AArch64::STURQi:
return AArch64::STPQi;
case AArch64::STRWui:
case AArch64::STURWi:
return AArch64::STPWi;
case AArch64::STRXui:
case AArch64::STURXi:
return AArch64::STPXi;
case AArch64::LDRSui:
case AArch64::LDURSi:
return AArch64::LDPSi;
case AArch64::LDRDui:
case AArch64::LDURDi:
return AArch64::LDPDi;
case AArch64::LDRQui:
case AArch64::LDURQi:
return AArch64::LDPQi;
case AArch64::LDRWui:
case AArch64::LDURWi:
return AArch64::LDPWi;
case AArch64::LDRXui:
case AArch64::LDURXi:
return AArch64::LDPXi;
[AArch64][LoadStoreOptimizer] Form LDPSW when possible. This patch adds the missing LD[U]RSW variants to the load store optimizer, so that we generate LDPSW when possible. <rdar://problem/19583480> llvm-svn: 226978
2015-01-24 09:25:54 +08:00
case AArch64::LDRSWui:
case AArch64::LDURSWi:
return AArch64::LDPSWi;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
}
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
static unsigned isMatchingStore(MachineInstr *LoadInst,
MachineInstr *StoreInst) {
unsigned LdOpc = LoadInst->getOpcode();
unsigned StOpc = StoreInst->getOpcode();
switch (LdOpc) {
default:
llvm_unreachable("Unsupported load instruction!");
case AArch64::LDRBBui:
return StOpc == AArch64::STRBBui || StOpc == AArch64::STRHHui ||
StOpc == AArch64::STRWui || StOpc == AArch64::STRXui;
case AArch64::LDURBBi:
return StOpc == AArch64::STURBBi || StOpc == AArch64::STURHHi ||
StOpc == AArch64::STURWi || StOpc == AArch64::STURXi;
case AArch64::LDRHHui:
return StOpc == AArch64::STRHHui || StOpc == AArch64::STRWui ||
StOpc == AArch64::STRXui;
case AArch64::LDURHHi:
return StOpc == AArch64::STURHHi || StOpc == AArch64::STURWi ||
StOpc == AArch64::STURXi;
case AArch64::LDRWui:
return StOpc == AArch64::STRWui || StOpc == AArch64::STRXui;
case AArch64::LDURWi:
return StOpc == AArch64::STURWi || StOpc == AArch64::STURXi;
case AArch64::LDRXui:
return StOpc == AArch64::STRXui;
case AArch64::LDURXi:
return StOpc == AArch64::STURXi;
}
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
static unsigned getPreIndexedOpcode(unsigned Opc) {
switch (Opc) {
default:
llvm_unreachable("Opcode has no pre-indexed equivalent!");
[AArch64] clang-format the load/store optimizer. No change in functionality. llvm-svn: 210182
2014-06-04 20:40:35 +08:00
case AArch64::STRSui:
return AArch64::STRSpre;
case AArch64::STRDui:
return AArch64::STRDpre;
case AArch64::STRQui:
return AArch64::STRQpre;
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::STRBBui:
return AArch64::STRBBpre;
case AArch64::STRHHui:
return AArch64::STRHHpre;
[AArch64] clang-format the load/store optimizer. No change in functionality. llvm-svn: 210182
2014-06-04 20:40:35 +08:00
case AArch64::STRWui:
return AArch64::STRWpre;
case AArch64::STRXui:
return AArch64::STRXpre;
case AArch64::LDRSui:
return AArch64::LDRSpre;
case AArch64::LDRDui:
return AArch64::LDRDpre;
case AArch64::LDRQui:
return AArch64::LDRQpre;
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::LDRBBui:
return AArch64::LDRBBpre;
case AArch64::LDRHHui:
return AArch64::LDRHHpre;
[AArch64] clang-format the load/store optimizer. No change in functionality. llvm-svn: 210182
2014-06-04 20:40:35 +08:00
case AArch64::LDRWui:
return AArch64::LDRWpre;
case AArch64::LDRXui:
return AArch64::LDRXpre;
[AArch64][LoadStoreOptimizer] Form LDPSW when possible. This patch adds the missing LD[U]RSW variants to the load store optimizer, so that we generate LDPSW when possible. <rdar://problem/19583480> llvm-svn: 226978
2015-01-24 09:25:54 +08:00
case AArch64::LDRSWui:
return AArch64::LDRSWpre;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
case AArch64::LDPSi:
return AArch64::LDPSpre;
[AArch64] Add support for pre- and post-index LDPSWs. llvm-svn: 248825
2015-09-30 04:39:55 +08:00
case AArch64::LDPSWi:
return AArch64::LDPSWpre;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
case AArch64::LDPDi:
return AArch64::LDPDpre;
case AArch64::LDPQi:
return AArch64::LDPQpre;
case AArch64::LDPWi:
return AArch64::LDPWpre;
case AArch64::LDPXi:
return AArch64::LDPXpre;
case AArch64::STPSi:
return AArch64::STPSpre;
case AArch64::STPDi:
return AArch64::STPDpre;
case AArch64::STPQi:
return AArch64::STPQpre;
case AArch64::STPWi:
return AArch64::STPWpre;
case AArch64::STPXi:
return AArch64::STPXpre;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
}
static unsigned getPostIndexedOpcode(unsigned Opc) {
switch (Opc) {
default:
llvm_unreachable("Opcode has no post-indexed wise equivalent!");
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::STRSui:
return AArch64::STRSpost;
case AArch64::STRDui:
return AArch64::STRDpost;
case AArch64::STRQui:
return AArch64::STRQpost;
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::STRBBui:
return AArch64::STRBBpost;
case AArch64::STRHHui:
return AArch64::STRHHpost;
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::STRWui:
return AArch64::STRWpost;
case AArch64::STRXui:
return AArch64::STRXpost;
case AArch64::LDRSui:
return AArch64::LDRSpost;
case AArch64::LDRDui:
return AArch64::LDRDpost;
case AArch64::LDRQui:
return AArch64::LDRQpost;
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::LDRBBui:
return AArch64::LDRBBpost;
case AArch64::LDRHHui:
return AArch64::LDRHHpost;
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::LDRWui:
return AArch64::LDRWpost;
case AArch64::LDRXui:
return AArch64::LDRXpost;
[AArch64][LoadStoreOptimizer] Form LDPSW when possible. This patch adds the missing LD[U]RSW variants to the load store optimizer, so that we generate LDPSW when possible. <rdar://problem/19583480> llvm-svn: 226978
2015-01-24 09:25:54 +08:00
case AArch64::LDRSWui:
return AArch64::LDRSWpost;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
case AArch64::LDPSi:
return AArch64::LDPSpost;
[AArch64] Add support for pre- and post-index LDPSWs. llvm-svn: 248825
2015-09-30 04:39:55 +08:00
case AArch64::LDPSWi:
return AArch64::LDPSWpost;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
case AArch64::LDPDi:
return AArch64::LDPDpost;
case AArch64::LDPQi:
return AArch64::LDPQpost;
case AArch64::LDPWi:
return AArch64::LDPWpost;
case AArch64::LDPXi:
return AArch64::LDPXpost;
case AArch64::STPSi:
return AArch64::STPSpost;
case AArch64::STPDi:
return AArch64::STPDpost;
case AArch64::STPQi:
return AArch64::STPQpost;
case AArch64::STPWi:
return AArch64::STPWpost;
case AArch64::STPXi:
return AArch64::STPXpost;
}
}
static bool isPairedLdSt(const MachineInstr *MI) {
switch (MI->getOpcode()) {
default:
return false;
case AArch64::LDPSi:
[AArch64] Add support for pre- and post-index LDPSWs. llvm-svn: 248825
2015-09-30 04:39:55 +08:00
case AArch64::LDPSWi:
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
case AArch64::LDPDi:
case AArch64::LDPQi:
case AArch64::LDPWi:
case AArch64::LDPXi:
case AArch64::STPSi:
case AArch64::STPDi:
case AArch64::STPQi:
case AArch64::STPWi:
case AArch64::STPXi:
return true;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
}
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
static const MachineOperand &getLdStRegOp(const MachineInstr *MI,
unsigned PairedRegOp = 0) {
assert(PairedRegOp < 2 && "Unexpected register operand idx.");
unsigned Idx = isPairedLdSt(MI) ? PairedRegOp : 0;
return MI->getOperand(Idx);
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
}
static const MachineOperand &getLdStBaseOp(const MachineInstr *MI) {
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
unsigned Idx = isPairedLdSt(MI) ? 2 : 1;
return MI->getOperand(Idx);
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
}
static const MachineOperand &getLdStOffsetOp(const MachineInstr *MI) {
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
unsigned Idx = isPairedLdSt(MI) ? 3 : 2;
return MI->getOperand(Idx);
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
}
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
static bool isLdOffsetInRangeOfSt(MachineInstr *LoadInst,
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
MachineInstr *StoreInst,
const AArch64InstrInfo *TII) {
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
assert(isMatchingStore(LoadInst, StoreInst) && "Expect only matched ld/st.");
int LoadSize = getMemScale(LoadInst);
int StoreSize = getMemScale(StoreInst);
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
int UnscaledStOffset = TII->isUnscaledLdSt(StoreInst)
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
? getLdStOffsetOp(StoreInst).getImm()
: getLdStOffsetOp(StoreInst).getImm() * StoreSize;
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
int UnscaledLdOffset = TII->isUnscaledLdSt(LoadInst)
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
? getLdStOffsetOp(LoadInst).getImm()
: getLdStOffsetOp(LoadInst).getImm() * LoadSize;
return (UnscaledStOffset <= UnscaledLdOffset) &&
(UnscaledLdOffset + LoadSize <= (UnscaledStOffset + StoreSize));
}
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
static bool isPromotableZeroStoreOpcode(MachineInstr *MI) {
unsigned Opc = MI->getOpcode();
return isNarrowStore(Opc) || Opc == AArch64::STRWui || Opc == AArch64::STURWi;
}
static bool isPromotableZeroStoreInst(MachineInstr *MI) {
return (isPromotableZeroStoreOpcode(MI)) &&
getLdStRegOp(MI).getReg() == AArch64::WZR;
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
AArch64LoadStoreOpt::mergeNarrowInsns(MachineBasicBlock::iterator I,
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MachineBasicBlock::iterator MergeMI,
[AArch64] Simplify the passing of arguments. NFC. This is setup for future work planned for the AArch64 Load/Store Opt pass. llvm-svn: 242810
2015-07-22 01:42:04 +08:00
const LdStPairFlags &Flags) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator NextI = I;
++NextI;
// If NextI is the second of the two instructions to be merged, we need
// to skip one further. Either way we merge will invalidate the iterator,
// and we don't need to scan the new instruction, as it's a pairwise
// instruction, which we're not considering for further action anyway.
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
if (NextI == MergeMI)
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
++NextI;
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
unsigned Opc = I->getOpcode();
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
bool IsScaled = !TII->isUnscaledLdSt(Opc);
[AArch64] Hoist now common logic. NFC. llvm-svn: 260257
2016-02-10 03:17:18 +08:00
int OffsetStride = IsScaled ? 1 : getMemScale(I);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Simplify the passing of arguments. NFC. This is setup for future work planned for the AArch64 Load/Store Opt pass. llvm-svn: 242810
2015-07-22 01:42:04 +08:00
bool MergeForward = Flags.getMergeForward();
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Insert our new paired instruction after whichever of the paired
[AArch64] Fix some LLVM Coding Standards violations in the load/store optimizer. Variable names should start with an upper case letter. No change in functionality. llvm-svn: 210181
2014-06-04 20:36:28 +08:00
// instructions MergeForward indicates.
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MachineBasicBlock::iterator InsertionPoint = MergeForward ? MergeMI : I;
[AArch64] Fix some LLVM Coding Standards violations in the load/store optimizer. Variable names should start with an upper case letter. No change in functionality. llvm-svn: 210181
2014-06-04 20:36:28 +08:00
// Also based on MergeForward is from where we copy the base register operand
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// so we get the flags compatible with the input code.
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
const MachineOperand &BaseRegOp =
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MergeForward ? getLdStBaseOp(MergeMI) : getLdStBaseOp(I);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Which register is Rt and which is Rt2 depends on the offset order.
MachineInstr *RtMI, *Rt2MI;
Revert "[AArch64] Improve load/store optimizer to handle LDUR + LDR (take 3)." This reverts commit r259812 as it broke AArch64 self-hosting. llvm-svn: 259881
2016-02-05 20:14:30 +08:00
if (getLdStOffsetOp(I).getImm() ==
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
getLdStOffsetOp(MergeMI).getImm() + OffsetStride) {
RtMI = MergeMI;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
Rt2MI = I;
} else {
RtMI = I;
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
Rt2MI = MergeMI;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
Revert "[AArch64]Merge halfword loads into a 32-bit load" This reverts commit r250719. This introduced a codegen fault in SPEC2000.gcc, when compiled for Cortex-A53. llvm-svn: 251108
2015-10-23 18:41:38 +08:00
int OffsetImm = getLdStOffsetOp(RtMI).getImm();
[AArch64] Hoist now common logic. NFC. llvm-svn: 260257
2016-02-10 03:17:18 +08:00
// Change the scaled offset from small to large type.
if (IsScaled) {
assert(((OffsetImm & 1) == 0) && "Unexpected offset to merge");
OffsetImm /= 2;
}
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
DebugLoc DL = I->getDebugLoc();
MachineBasicBlock *MBB = I->getParent();
[AArch64] Refactoring aarch64-ldst-opt. NCF. Summary : * Rename isSmallTypeLdMerge() to isNarrowLoad(). * Rename NumSmallTypeMerged to NumNarrowTypePromoted. * Use Subtarget defined as a member variable. llvm-svn: 253587
2015-11-20 02:41:27 +08:00
if (isNarrowLoad(Opc)) {
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MachineInstr *RtNewDest = MergeForward ? I : MergeMI;
[AArch64] Fix halfword load merging for big-endian targets For big-endian targets, when we merge two halfword loads into a word load, the order of the halfwords in the loaded value is reversed compared to little-endian, so the load-store optimiser needs to swap the destination registers. This does not affect merging of two word loads, as we use ldp, which treats the memory as two separate 32-bit words. llvm-svn: 252597
2015-11-10 19:04:18 +08:00
// When merging small (< 32 bit) loads for big-endian targets, the order of
// the component parts gets swapped.
if (!Subtarget->isLittleEndian())
std::swap(RtMI, Rt2MI);
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
// Construct the new load instruction.
MachineInstr *NewMemMI, *BitExtMI1, *BitExtMI2;
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
NewMemMI =
BuildMI(*MBB, InsertionPoint, DL, TII->get(getMatchingWideOpcode(Opc)))
.addOperand(getLdStRegOp(RtNewDest))
.addOperand(BaseRegOp)
.addImm(OffsetImm)
.setMemRefs(I->mergeMemRefsWith(*MergeMI));
[AArch64] Fix warnings pointed out by Hal. llvm-svn: 264882
2016-03-31 02:08:51 +08:00
(void)NewMemMI;
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
DEBUG(
dbgs()
<< "Creating the new load and extract. Replacing instructions:\n ");
DEBUG(I->print(dbgs()));
DEBUG(dbgs() << " ");
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
DEBUG(MergeMI->print(dbgs()));
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
DEBUG(dbgs() << " with instructions:\n ");
DEBUG((NewMemMI)->print(dbgs()));
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
int Width = getMemScale(I) == 1 ? 8 : 16;
int LSBLow = 0;
int LSBHigh = Width;
int ImmsLow = LSBLow + Width - 1;
int ImmsHigh = LSBHigh + Width - 1;
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MachineInstr *ExtDestMI = MergeForward ? MergeMI : I;
[AArch64] Fix halfword load merging for big-endian targets For big-endian targets, when we merge two halfword loads into a word load, the order of the halfwords in the loaded value is reversed compared to little-endian, so the load-store optimiser needs to swap the destination registers. This does not affect merging of two word loads, as we use ldp, which treats the memory as two separate 32-bit words. llvm-svn: 252597
2015-11-10 19:04:18 +08:00
if ((ExtDestMI == Rt2MI) == Subtarget->isLittleEndian()) {
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
// Create the bitfield extract for high bits.
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BitExtMI1 =
BuildMI(*MBB, InsertionPoint, DL, TII->get(getBitExtrOpcode(Rt2MI)))
.addOperand(getLdStRegOp(Rt2MI))
.addReg(getLdStRegOp(RtNewDest).getReg())
.addImm(LSBHigh)
.addImm(ImmsHigh);
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
// Create the bitfield extract for low bits.
if (RtMI->getOpcode() == getMatchingNonSExtOpcode(RtMI->getOpcode())) {
// For unsigned, prefer to use AND for low bits.
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BitExtMI2 = BuildMI(*MBB, InsertionPoint, DL, TII->get(AArch64::ANDWri))
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
.addOperand(getLdStRegOp(RtMI))
.addReg(getLdStRegOp(RtNewDest).getReg())
.addImm(ImmsLow);
} else {
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BitExtMI2 =
BuildMI(*MBB, InsertionPoint, DL, TII->get(getBitExtrOpcode(RtMI)))
.addOperand(getLdStRegOp(RtMI))
.addReg(getLdStRegOp(RtNewDest).getReg())
.addImm(LSBLow)
.addImm(ImmsLow);
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
}
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
} else {
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
// Create the bitfield extract for low bits.
if (RtMI->getOpcode() == getMatchingNonSExtOpcode(RtMI->getOpcode())) {
// For unsigned, prefer to use AND for low bits.
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BitExtMI1 = BuildMI(*MBB, InsertionPoint, DL, TII->get(AArch64::ANDWri))
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
.addOperand(getLdStRegOp(RtMI))
.addReg(getLdStRegOp(RtNewDest).getReg())
.addImm(ImmsLow);
} else {
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BitExtMI1 =
BuildMI(*MBB, InsertionPoint, DL, TII->get(getBitExtrOpcode(RtMI)))
.addOperand(getLdStRegOp(RtMI))
.addReg(getLdStRegOp(RtNewDest).getReg())
.addImm(LSBLow)
.addImm(ImmsLow);
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
}
// Create the bitfield extract for high bits.
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BitExtMI2 =
BuildMI(*MBB, InsertionPoint, DL, TII->get(getBitExtrOpcode(Rt2MI)))
.addOperand(getLdStRegOp(Rt2MI))
.addReg(getLdStRegOp(RtNewDest).getReg())
.addImm(LSBHigh)
.addImm(ImmsHigh);
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
}
[AArch64] Fix warnings pointed out by Hal. llvm-svn: 264882
2016-03-31 02:08:51 +08:00
(void)BitExtMI1;
(void)BitExtMI2;
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
DEBUG(dbgs() << " ");
DEBUG((BitExtMI1)->print(dbgs()));
DEBUG(dbgs() << " ");
DEBUG((BitExtMI2)->print(dbgs()));
DEBUG(dbgs() << "\n");
// Erase the old instructions.
I->eraseFromParent();
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MergeMI->eraseFromParent();
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
return NextI;
}
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
assert(isPromotableZeroStoreInst(I) && isPromotableZeroStoreInst(MergeMI) &&
"Expected promotable zero store");
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Construct the new instruction.
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
MachineInstrBuilder MIB;
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
MIB = BuildMI(*MBB, InsertionPoint, DL, TII->get(getMatchingWideOpcode(Opc)))
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
.addReg(isNarrowStore(Opc) ? AArch64::WZR : AArch64::XZR)
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
.addOperand(BaseRegOp)
.addImm(OffsetImm)
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
.setMemRefs(I->mergeMemRefsWith(*MergeMI));
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
(void)MIB;
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
DEBUG(dbgs() << "Creating wider load/store. Replacing instructions:\n ");
DEBUG(I->print(dbgs()));
DEBUG(dbgs() << " ");
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
DEBUG(MergeMI->print(dbgs()));
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
DEBUG(dbgs() << " with instruction:\n ");
DEBUG(((MachineInstr *)MIB)->print(dbgs()));
DEBUG(dbgs() << "\n");
// Erase the old instructions.
I->eraseFromParent();
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MergeMI->eraseFromParent();
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
return NextI;
}
MachineBasicBlock::iterator
AArch64LoadStoreOpt::mergePairedInsns(MachineBasicBlock::iterator I,
MachineBasicBlock::iterator Paired,
const LdStPairFlags &Flags) {
MachineBasicBlock::iterator NextI = I;
++NextI;
// If NextI is the second of the two instructions to be merged, we need
// to skip one further. Either way we merge will invalidate the iterator,
// and we don't need to scan the new instruction, as it's a pairwise
// instruction, which we're not considering for further action anyway.
if (NextI == Paired)
++NextI;
int SExtIdx = Flags.getSExtIdx();
unsigned Opc =
SExtIdx == -1 ? I->getOpcode() : getMatchingNonSExtOpcode(I->getOpcode());
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
bool IsUnscaled = TII->isUnscaledLdSt(Opc);
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
int OffsetStride = IsUnscaled ? getMemScale(I) : 1;
bool MergeForward = Flags.getMergeForward();
// Insert our new paired instruction after whichever of the paired
// instructions MergeForward indicates.
MachineBasicBlock::iterator InsertionPoint = MergeForward ? Paired : I;
// Also based on MergeForward is from where we copy the base register operand
// so we get the flags compatible with the input code.
const MachineOperand &BaseRegOp =
MergeForward ? getLdStBaseOp(Paired) : getLdStBaseOp(I);
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
int Offset = getLdStOffsetOp(I).getImm();
int PairedOffset = getLdStOffsetOp(Paired).getImm();
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
bool PairedIsUnscaled = TII->isUnscaledLdSt(Paired->getOpcode());
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
if (IsUnscaled != PairedIsUnscaled) {
// We're trying to pair instructions that differ in how they are scaled. If
// I is scaled then scale the offset of Paired accordingly. Otherwise, do
// the opposite (i.e., make Paired's offset unscaled).
int MemSize = getMemScale(Paired);
if (PairedIsUnscaled) {
// If the unscaled offset isn't a multiple of the MemSize, we can't
// pair the operations together.
assert(!(PairedOffset % getMemScale(Paired)) &&
"Offset should be a multiple of the stride!");
PairedOffset /= MemSize;
} else {
PairedOffset *= MemSize;
}
}
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
// Which register is Rt and which is Rt2 depends on the offset order.
MachineInstr *RtMI, *Rt2MI;
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
if (Offset == PairedOffset + OffsetStride) {
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
RtMI = Paired;
Rt2MI = I;
// Here we swapped the assumption made for SExtIdx.
// I.e., we turn ldp I, Paired into ldp Paired, I.
// Update the index accordingly.
if (SExtIdx != -1)
SExtIdx = (SExtIdx + 1) % 2;
} else {
RtMI = I;
Rt2MI = Paired;
}
int OffsetImm = getLdStOffsetOp(RtMI).getImm();
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
// Scale the immediate offset, if necessary.
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
if (TII->isUnscaledLdSt(RtMI->getOpcode())) {
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
assert(!(OffsetImm % getMemScale(RtMI)) &&
"Unscaled offset cannot be scaled.");
OffsetImm /= getMemScale(RtMI);
[AArch64] Add an assert to ensure we don't scale an offset that can't be scaled. llvm-svn: 260272
2016-02-10 04:18:07 +08:00
}
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
// Construct the new instruction.
MachineInstrBuilder MIB;
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
DebugLoc DL = I->getDebugLoc();
MachineBasicBlock *MBB = I->getParent();
MIB = BuildMI(*MBB, InsertionPoint, DL, TII->get(getMatchingPairOpcode(Opc)))
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
.addOperand(getLdStRegOp(RtMI))
.addOperand(getLdStRegOp(Rt2MI))
.addOperand(BaseRegOp)
[AArch64] Add MMOs to unscaled pairs. Test to be committed in follow up commit, per discussion in D17097. http://reviews.llvm.org/D17097 llvm-svn: 262942
2016-03-09 01:16:38 +08:00
.addImm(OffsetImm)
.setMemRefs(I->mergeMemRefsWith(*Paired));
[AArch64] Separage the codegen logic for widening vs. pairing. NFC. llvm-svn: 260249
2016-02-10 03:02:12 +08:00
(void)MIB;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
DEBUG(dbgs() << "Creating pair load/store. Replacing instructions:\n ");
DEBUG(I->print(dbgs()));
DEBUG(dbgs() << " ");
DEBUG(Paired->print(dbgs()));
DEBUG(dbgs() << " with instruction:\n ");
[AArch64][LoadStoreOptimizer] Generate LDP + SXTW instead of LD[U]R + LD[U]RSW. Teach the load store optimizer how to sign extend a result of a load pair when it helps creating more pairs. The rational is that loads are more expensive than sign extensions, so if we gather some in one instruction this is better! <rdar://problem/20072968> llvm-svn: 231527
2015-03-07 06:42:10 +08:00
if (SExtIdx != -1) {
// Generate the sign extension for the proper result of the ldp.
// I.e., with X1, that would be:
// %W1<def> = KILL %W1, %X1<imp-def>
// %X1<def> = SBFMXri %X1<kill>, 0, 31
MachineOperand &DstMO = MIB->getOperand(SExtIdx);
// Right now, DstMO has the extended register, since it comes from an
// extended opcode.
unsigned DstRegX = DstMO.getReg();
// Get the W variant of that register.
unsigned DstRegW = TRI->getSubReg(DstRegX, AArch64::sub_32);
// Update the result of LDP to use the W instead of the X variant.
DstMO.setReg(DstRegW);
DEBUG(((MachineInstr *)MIB)->print(dbgs()));
DEBUG(dbgs() << "\n");
// Make the machine verifier happy by providing a definition for
// the X register.
// Insert this definition right after the generated LDP, i.e., before
// InsertionPoint.
MachineInstrBuilder MIBKill =
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BuildMI(*MBB, InsertionPoint, DL, TII->get(TargetOpcode::KILL), DstRegW)
[AArch64][LoadStoreOptimizer] Generate LDP + SXTW instead of LD[U]R + LD[U]RSW. Teach the load store optimizer how to sign extend a result of a load pair when it helps creating more pairs. The rational is that loads are more expensive than sign extensions, so if we gather some in one instruction this is better! <rdar://problem/20072968> llvm-svn: 231527
2015-03-07 06:42:10 +08:00
.addReg(DstRegW)
.addReg(DstRegX, RegState::Define);
MIBKill->getOperand(2).setImplicit();
// Create the sign extension.
MachineInstrBuilder MIBSXTW =
[AArch64] Remove redundant calls and clang format. NFC. llvm-svn: 260260
2016-02-10 03:33:42 +08:00
BuildMI(*MBB, InsertionPoint, DL, TII->get(AArch64::SBFMXri), DstRegX)
[AArch64][LoadStoreOptimizer] Generate LDP + SXTW instead of LD[U]R + LD[U]RSW. Teach the load store optimizer how to sign extend a result of a load pair when it helps creating more pairs. The rational is that loads are more expensive than sign extensions, so if we gather some in one instruction this is better! <rdar://problem/20072968> llvm-svn: 231527
2015-03-07 06:42:10 +08:00
.addReg(DstRegX)
.addImm(0)
.addImm(31);
(void)MIBSXTW;
DEBUG(dbgs() << " Extend operand:\n ");
DEBUG(((MachineInstr *)MIBSXTW)->print(dbgs()));
} else {
DEBUG(((MachineInstr *)MIB)->print(dbgs()));
}
[AArch64] Simplify. NFC. llvm-svn: 260273
2016-02-10 04:27:45 +08:00
DEBUG(dbgs() << "\n");
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Erase the old instructions.
I->eraseFromParent();
Paired->eraseFromParent();
return NextI;
}
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
MachineBasicBlock::iterator
AArch64LoadStoreOpt::promoteLoadFromStore(MachineBasicBlock::iterator LoadI,
MachineBasicBlock::iterator StoreI) {
MachineBasicBlock::iterator NextI = LoadI;
++NextI;
int LoadSize = getMemScale(LoadI);
int StoreSize = getMemScale(StoreI);
unsigned LdRt = getLdStRegOp(LoadI).getReg();
unsigned StRt = getLdStRegOp(StoreI).getReg();
bool IsStoreXReg = TRI->getRegClass(AArch64::GPR64RegClassID)->contains(StRt);
assert((IsStoreXReg ||
TRI->getRegClass(AArch64::GPR32RegClassID)->contains(StRt)) &&
"Unexpected RegClass");
MachineInstr *BitExtMI;
if (LoadSize == StoreSize && (LoadSize == 4 || LoadSize == 8)) {
// Remove the load, if the destination register of the loads is the same
// register for stored value.
if (StRt == LdRt && LoadSize == 8) {
DEBUG(dbgs() << "Remove load instruction:\n ");
DEBUG(LoadI->print(dbgs()));
DEBUG(dbgs() << "\n");
LoadI->eraseFromParent();
return NextI;
}
// Replace the load with a mov if the load and store are in the same size.
BitExtMI =
BuildMI(*LoadI->getParent(), LoadI, LoadI->getDebugLoc(),
TII->get(IsStoreXReg ? AArch64::ORRXrs : AArch64::ORRWrs), LdRt)
.addReg(IsStoreXReg ? AArch64::XZR : AArch64::WZR)
.addReg(StRt)
.addImm(AArch64_AM::getShifterImm(AArch64_AM::LSL, 0));
} else {
// FIXME: Currently we disable this transformation in big-endian targets as
// performance and correctness are verified only in little-endian.
if (!Subtarget->isLittleEndian())
return NextI;
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
bool IsUnscaled = TII->isUnscaledLdSt(LoadI);
assert(IsUnscaled == TII->isUnscaledLdSt(StoreI) &&
"Unsupported ld/st match");
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
assert(LoadSize <= StoreSize && "Invalid load size");
int UnscaledLdOffset = IsUnscaled
? getLdStOffsetOp(LoadI).getImm()
: getLdStOffsetOp(LoadI).getImm() * LoadSize;
int UnscaledStOffset = IsUnscaled
? getLdStOffsetOp(StoreI).getImm()
: getLdStOffsetOp(StoreI).getImm() * StoreSize;
int Width = LoadSize * 8;
int Immr = 8 * (UnscaledLdOffset - UnscaledStOffset);
int Imms = Immr + Width - 1;
unsigned DestReg = IsStoreXReg
? TRI->getMatchingSuperReg(LdRt, AArch64::sub_32,
&AArch64::GPR64RegClass)
: LdRt;
assert((UnscaledLdOffset >= UnscaledStOffset &&
(UnscaledLdOffset + LoadSize) <= UnscaledStOffset + StoreSize) &&
"Invalid offset");
Immr = 8 * (UnscaledLdOffset - UnscaledStOffset);
Imms = Immr + Width - 1;
if (UnscaledLdOffset == UnscaledStOffset) {
uint32_t AndMaskEncoded = ((IsStoreXReg ? 1 : 0) << 12) // N
| ((Immr) << 6) // immr
| ((Imms) << 0) // imms
;
BitExtMI =
BuildMI(*LoadI->getParent(), LoadI, LoadI->getDebugLoc(),
TII->get(IsStoreXReg ? AArch64::ANDXri : AArch64::ANDWri),
DestReg)
.addReg(StRt)
.addImm(AndMaskEncoded);
} else {
BitExtMI =
BuildMI(*LoadI->getParent(), LoadI, LoadI->getDebugLoc(),
TII->get(IsStoreXReg ? AArch64::UBFMXri : AArch64::UBFMWri),
DestReg)
.addReg(StRt)
.addImm(Immr)
.addImm(Imms);
}
}
[AArch64] Fix warnings pointed out by Hal. llvm-svn: 264882
2016-03-31 02:08:51 +08:00
(void)BitExtMI;
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
DEBUG(dbgs() << "Promoting load by replacing :\n ");
DEBUG(StoreI->print(dbgs()));
DEBUG(dbgs() << " ");
DEBUG(LoadI->print(dbgs()));
DEBUG(dbgs() << " with instructions:\n ");
DEBUG(StoreI->print(dbgs()));
DEBUG(dbgs() << " ");
DEBUG((BitExtMI)->print(dbgs()));
DEBUG(dbgs() << "\n");
// Erase the old instructions.
LoadI->eraseFromParent();
return NextI;
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
/// trackRegDefsUses - Remember what registers the specified instruction uses
/// and modifies.
Convert some AArch64 code to foreach loops. NFC. Also converted a cast<> to dyn_cast while i was working on the same line of code. llvm-svn: 243894
2015-08-04 03:04:32 +08:00
static void trackRegDefsUses(const MachineInstr *MI, BitVector &ModifiedRegs,
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
BitVector &UsedRegs,
const TargetRegisterInfo *TRI) {
Convert some AArch64 code to foreach loops. NFC. Also converted a cast<> to dyn_cast while i was working on the same line of code. llvm-svn: 243894
2015-08-04 03:04:32 +08:00
for (const MachineOperand &MO : MI->operands()) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
if (MO.isRegMask())
ModifiedRegs.setBitsNotInMask(MO.getRegMask());
if (!MO.isReg())
continue;
unsigned Reg = MO.getReg();
[AArch64] AArch64LoadStoreOptimizer: fix bug in pre-inc check iterator Summary: Fix case where a pre-inc/dec load/store would not be formed if the add/sub that forms the inc/dec part of the operation was the first instruction in the block being examined. Reviewers: mcrosier, jmolloy, t.p.northover, junbuml Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16785 llvm-svn: 260275
2016-02-10 04:47:21 +08:00
if (!Reg)
continue;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
if (MO.isDef()) {
for (MCRegAliasIterator AI(Reg, TRI, true); AI.isValid(); ++AI)
ModifiedRegs.set(*AI);
} else {
assert(MO.isUse() && "Reg operand not a def and not a use?!?");
for (MCRegAliasIterator AI(Reg, TRI, true); AI.isValid(); ++AI)
UsedRegs.set(*AI);
}
}
}
static bool inBoundsForPair(bool IsUnscaled, int Offset, int OffsetStride) {
[AArch64] Simplify the logic for computing in bounds offset. NFC. llvm-svn: 245307
2015-08-19 00:20:03 +08:00
// Convert the byte-offset used by unscaled into an "element" offset used
// by the scaled pair load/store instructions.
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
if (IsUnscaled) {
// If the byte-offset isn't a multiple of the stride, there's no point
// trying to match it.
if (Offset % OffsetStride)
return false;
[AArch64] Simplify the logic for computing in bounds offset. NFC. llvm-svn: 245307
2015-08-19 00:20:03 +08:00
Offset /= OffsetStride;
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
}
[AArch64] Simplify the logic for computing in bounds offset. NFC. llvm-svn: 245307
2015-08-19 00:20:03 +08:00
return Offset <= 63 && Offset >= -64;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
// Do alignment, specialized to power of 2 and for signed ints,
// avoiding having to do a C-style cast from uint_64t to int when
Update to use new name alignTo(). llvm-svn: 257804
2016-01-15 05:06:47 +08:00
// using alignTo from include/llvm/Support/MathExtras.h.
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// FIXME: Move this function to include/MathExtras.h?
static int alignTo(int Num, int PowOf2) {
return (Num + PowOf2 - 1) & ~(PowOf2 - 1);
}
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
static bool mayAlias(MachineInstr *MIa, MachineInstr *MIb,
const AArch64InstrInfo *TII) {
// One of the instructions must modify memory.
if (!MIa->mayStore() && !MIb->mayStore())
return false;
// Both instructions must be memory operations.
if (!MIa->mayLoadOrStore() && !MIb->mayLoadOrStore())
return false;
return !TII->areMemAccessesTriviallyDisjoint(MIa, MIb);
}
static bool mayAlias(MachineInstr *MIa,
SmallVectorImpl<MachineInstr *> &MemInsns,
const AArch64InstrInfo *TII) {
for (auto &MIb : MemInsns)
if (mayAlias(MIa, MIb, TII))
return true;
return false;
}
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
bool AArch64LoadStoreOpt::findMatchingStore(
MachineBasicBlock::iterator I, unsigned Limit,
MachineBasicBlock::iterator &StoreI) {
[AArch64] Refactoring findMatchingStore() in aarch64-ldst-opt; NFC Summary: This change makes findMatchingStore() follow the same coding style introduced in r260275. Reviewers: gberry, junbuml Subscribers: aemerson, rengolin, haicheng, bmakam, mssimpso Differential Revision: http://reviews.llvm.org/D17083 llvm-svn: 260534
2016-02-12 00:18:24 +08:00
MachineBasicBlock::iterator B = I->getParent()->begin();
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
MachineBasicBlock::iterator MBBI = I;
[AArch64] Rename variable to improve readability. NFC. llvm-svn: 260228
2016-02-09 23:59:57 +08:00
MachineInstr *LoadMI = I;
unsigned BaseReg = getLdStBaseOp(LoadMI).getReg();
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
[AArch64] Refactoring findMatchingStore() in aarch64-ldst-opt; NFC Summary: This change makes findMatchingStore() follow the same coding style introduced in r260275. Reviewers: gberry, junbuml Subscribers: aemerson, rengolin, haicheng, bmakam, mssimpso Differential Revision: http://reviews.llvm.org/D17083 llvm-svn: 260534
2016-02-12 00:18:24 +08:00
// If the load is the first instruction in the block, there's obviously
// not any matching store.
if (MBBI == B)
return false;
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
// Track which registers have been modified and used between the first insn
// and the second insn.
[AArch64] Allocate the modified and used regs only once per function. llvm-svn: 259510
2016-02-02 23:02:30 +08:00
ModifiedRegs.reset();
UsedRegs.reset();
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
[AArch64] Refactoring findMatchingStore() in aarch64-ldst-opt; NFC Summary: This change makes findMatchingStore() follow the same coding style introduced in r260275. Reviewers: gberry, junbuml Subscribers: aemerson, rengolin, haicheng, bmakam, mssimpso Differential Revision: http://reviews.llvm.org/D17083 llvm-svn: 260534
2016-02-12 00:18:24 +08:00
unsigned Count = 0;
do {
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
--MBBI;
MachineInstr *MI = MBBI;
[AArch64] Refactoring findMatchingStore() in aarch64-ldst-opt; NFC Summary: This change makes findMatchingStore() follow the same coding style introduced in r260275. Reviewers: gberry, junbuml Subscribers: aemerson, rengolin, haicheng, bmakam, mssimpso Differential Revision: http://reviews.llvm.org/D17083 llvm-svn: 260534
2016-02-12 00:18:24 +08:00
// Don't count DBG_VALUE instructions towards the search limit.
if (!MI->isDebugValue())
++Count;
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
// If the load instruction reads directly from the address to which the
// store instruction writes and the stored value is not modified, we can
// promote the load. Since we do not handle stores with pre-/post-index,
// it's unnecessary to check if BaseReg is modified by the store itself.
[AArch64] Rename variable to improve readability. NFC. llvm-svn: 260228
2016-02-09 23:59:57 +08:00
if (MI->mayStore() && isMatchingStore(LoadMI, MI) &&
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
BaseReg == getLdStBaseOp(MI).getReg() &&
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
isLdOffsetInRangeOfSt(LoadMI, MI, TII) &&
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
!ModifiedRegs[getLdStRegOp(MI).getReg()]) {
StoreI = MBBI;
return true;
}
if (MI->isCall())
return false;
// Update modified / uses register lists.
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
// Otherwise, if the base register is modified, we have no match, so
// return early.
if (ModifiedRegs[BaseReg])
return false;
// If we encounter a store aliased with the load, return early.
[AArch64] Rename variable to improve readability. NFC. llvm-svn: 260228
2016-02-09 23:59:57 +08:00
if (MI->mayStore() && mayAlias(LoadMI, MI, TII))
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
return false;
[AArch64] Refactoring findMatchingStore() in aarch64-ldst-opt; NFC Summary: This change makes findMatchingStore() follow the same coding style introduced in r260275. Reviewers: gberry, junbuml Subscribers: aemerson, rengolin, haicheng, bmakam, mssimpso Differential Revision: http://reviews.llvm.org/D17083 llvm-svn: 260534
2016-02-12 00:18:24 +08:00
} while (MBBI != B && Count < Limit);
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
return false;
}
[AArch64] Refactor is logic into a helper function. NFC. llvm-svn: 260419
2016-02-11 03:45:48 +08:00
// Returns true if these two opcodes can be merged or paired. Otherwise,
// returns false.
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
static bool canMergeOpc(unsigned OpcA, unsigned OpcB, LdStPairFlags &Flags,
const AArch64InstrInfo *TII) {
[AArch64] Refactor is logic into a helper function. NFC. llvm-svn: 260419
2016-02-11 03:45:48 +08:00
// Opcodes match: nothing more to check.
if (OpcA == OpcB)
return true;
// Try to match a sign-extended load/store with a zero-extended load/store.
bool IsValidLdStrOpc, PairIsValidLdStrOpc;
unsigned NonSExtOpc = getMatchingNonSExtOpcode(OpcA, &IsValidLdStrOpc);
assert(IsValidLdStrOpc &&
"Given Opc should be a Load or Store with an immediate");
// OpcA will be the first instruction in the pair.
if (NonSExtOpc == getMatchingNonSExtOpcode(OpcB, &PairIsValidLdStrOpc)) {
Flags.setSExtIdx(NonSExtOpc == (unsigned)OpcA ? 1 : 0);
return true;
}
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
// If the second instruction isn't even a load/store, bail out.
if (!PairIsValidLdStrOpc)
return false;
// FIXME: We don't support merging narrow loads/stores with mixed
// scaled/unscaled offsets.
if (isNarrowLoadOrStore(OpcA) || isNarrowLoadOrStore(OpcB))
return false;
// Try to match an unscaled load/store with a scaled load/store.
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
return TII->isUnscaledLdSt(OpcA) != TII->isUnscaledLdSt(OpcB) &&
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
getMatchingPairOpcode(OpcA) == getMatchingPairOpcode(OpcB);
// FIXME: Can we also match a mixed sext/zext unscaled/scaled pair?
[AArch64] Refactor is logic into a helper function. NFC. llvm-svn: 260419
2016-02-11 03:45:48 +08:00
}
[AArch64] Update comment to match reality. NFC. llvm-svn: 260406
2016-02-11 02:49:28 +08:00
/// Scan the instructions looking for a load/store that can be combined with the
/// current instruction into a wider equivalent or a load/store pair.
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
AArch64LoadStoreOpt::findMatchingInsn(MachineBasicBlock::iterator I,
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
LdStPairFlags &Flags, unsigned Limit,
bool FindNarrowMerge) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator E = I->getParent()->end();
MachineBasicBlock::iterator MBBI = I;
MachineInstr *FirstMI = I;
++MBBI;
MachineInstr: Change return value of getOpcode() to unsigned. This was previously returning int. However there are no negative opcode numbers and more importantly this was needlessly different from MCInstrDesc::getOpcode() (which even is the value returned here) and SDValue::getOpcode()/SDNode::getOpcode(). llvm-svn: 237611
2015-05-19 04:27:55 +08:00
unsigned Opc = FirstMI->getOpcode();
[AArch64] Fix some LLVM Coding Standards violations in the load/store optimizer. Variable names should start with an upper case letter. No change in functionality. llvm-svn: 210181
2014-06-04 20:36:28 +08:00
bool MayLoad = FirstMI->mayLoad();
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
bool IsUnscaled = TII->isUnscaledLdSt(FirstMI);
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
unsigned Reg = getLdStRegOp(FirstMI).getReg();
unsigned BaseReg = getLdStBaseOp(FirstMI).getReg();
int Offset = getLdStOffsetOp(FirstMI).getImm();
[AArch64] Deprecate a command-line option used for testing. Support for pairing unscaled loads and stores has been enabled since the original ARM64 port. This feature is no longer experimental, AFAICT. llvm-svn: 249049
2015-10-02 02:17:12 +08:00
int OffsetStride = IsUnscaled ? getMemScale(FirstMI) : 1;
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
bool IsPromotableZeroStore = isPromotableZeroStoreInst(FirstMI);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Track which registers have been modified and used between the first insn
// (inclusive) and the second insn.
[AArch64] Allocate the modified and used regs only once per function. llvm-svn: 259510
2016-02-02 23:02:30 +08:00
ModifiedRegs.reset();
UsedRegs.reset();
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
// Remember any instructions that read/write memory between FirstMI and MI.
SmallVector<MachineInstr *, 4> MemInsns;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
for (unsigned Count = 0; MBBI != E && Count < Limit; ++MBBI) {
MachineInstr *MI = MBBI;
// Skip DBG_VALUE instructions. Otherwise debug info can affect the
// optimization by changing how far we scan.
if (MI->isDebugValue())
continue;
// Now that we know this is a real instruction, count it.
++Count;
Revert "[AArch64] Improve load/store optimizer to handle LDUR + LDR." This reverts commit r259790. tramp3d-v4 is still having problems. llvm-svn: 259795
2016-02-05 00:01:40 +08:00
Flags.setSExtIdx(-1);
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
if (canMergeOpc(Opc, MI->getOpcode(), Flags, TII) &&
[AArch64] Refactor is logic into a helper function. NFC. llvm-svn: 260419
2016-02-11 03:45:48 +08:00
getLdStOffsetOp(MI).isImm()) {
[AArch64] Convert a conditional check that will always be true to an assert. NFC. llvm-svn: 244479
2015-08-11 02:42:45 +08:00
assert(MI->mayLoadOrStore() && "Expected memory operation.");
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// If we've found another instruction with the same opcode, check to see
// if the base and offset are compatible with our starting instruction.
// These instructions all have scaled immediate operands, so we just
// check for +1/-1. Make sure to check the new instruction offset is
// actually an immediate and not a symbolic reference destined for
// a relocation.
//
// Pairwise instructions have a 7-bit signed offset field. Single insns
// have a 12-bit unsigned offset field. To be a valid combine, the
// final offset must be in range.
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
unsigned MIBaseReg = getLdStBaseOp(MI).getReg();
int MIOffset = getLdStOffsetOp(MI).getImm();
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
bool MIIsUnscaled = TII->isUnscaledLdSt(MI);
[AArch64] Improve load/store optimizer to handle LDUR + LDR. This patch allows the mixing of scaled and unscaled load/stores to form load/store pairs. This is a reapplication of r259812, which had an incorrect assert. The test_stur_str_no_assert() test is a reduced version of the issue hit in the AArch64 self-host. PR24465 llvm-svn: 260523
2016-02-11 22:25:08 +08:00
if (IsUnscaled != MIIsUnscaled) {
// We're trying to pair instructions that differ in how they are scaled.
// If FirstMI is scaled then scale the offset of MI accordingly.
// Otherwise, do the opposite (i.e., make MI's offset unscaled).
int MemSize = getMemScale(MI);
if (MIIsUnscaled) {
// If the unscaled offset isn't a multiple of the MemSize, we can't
// pair the operations together: bail and keep looking.
if (MIOffset % MemSize)
continue;
MIOffset /= MemSize;
} else {
MIOffset *= MemSize;
}
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
if (BaseReg == MIBaseReg && ((Offset == MIOffset + OffsetStride) ||
(Offset + OffsetStride == MIOffset))) {
int MinOffset = Offset < MIOffset ? Offset : MIOffset;
// If this is a volatile load/store that otherwise matched, stop looking
// as something is going on that we don't have enough information to
// safely transform. Similarly, stop if we see a hint to avoid pairs.
if (MI->hasOrderedMemoryRef() || TII->isLdStPairSuppressed(MI))
return E;
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
if (FindNarrowMerge) {
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
// If the alignment requirements of the scaled wide load/store
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
// instruction can't express the offset of the scaled narrow input,
// bail and keep looking. For promotable zero stores, allow only when
// the stored value is the same (i.e., WZR).
if ((!IsUnscaled && alignTo(MinOffset, 2) != MinOffset) ||
(IsPromotableZeroStore && Reg != getLdStRegOp(MI).getReg())) {
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
MemInsns.push_back(MI);
continue;
}
} else {
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
// If the resultant immediate offset of merging these instructions
// is out of range for a pairwise instruction, bail and keep looking.
if (!inBoundsForPair(IsUnscaled, MinOffset, OffsetStride)) {
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
MemInsns.push_back(MI);
continue;
}
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
// If the alignment requirements of the paired (scaled) instruction
// can't express the offset of the unscaled input, bail and keep
// looking.
if (IsUnscaled && (alignTo(MinOffset, OffsetStride) != MinOffset)) {
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
MemInsns.push_back(MI);
continue;
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
// If the destination register of the loads is the same register, bail
// and keep looking. A load-pair instruction with both destination
// registers the same is UNPREDICTABLE and will result in an exception.
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
if (MayLoad && Reg == getLdStRegOp(MI).getReg()) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
[AArch64] Convert a conditional check that will always be true to an assert. NFC. llvm-svn: 244479
2015-08-11 02:42:45 +08:00
MemInsns.push_back(MI);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
continue;
}
// If the Rt of the second instruction was not modified or used between
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
// the two instructions and none of the instructions between the second
// and first alias with the second, we can combine the second into the
// first.
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
if (!ModifiedRegs[getLdStRegOp(MI).getReg()] &&
!(MI->mayLoad() && UsedRegs[getLdStRegOp(MI).getReg()]) &&
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
!mayAlias(MI, MemInsns, TII)) {
[AArch64] Simplify the passing of arguments. NFC. This is setup for future work planned for the AArch64 Load/Store Opt pass. llvm-svn: 242810
2015-07-22 01:42:04 +08:00
Flags.setMergeForward(false);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return MBBI;
}
// Likewise, if the Rt of the first instruction is not modified or used
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
// between the two instructions and none of the instructions between the
// first and the second alias with the first, we can combine the first
// into the second.
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
if (!ModifiedRegs[getLdStRegOp(FirstMI).getReg()] &&
[AArch64] Reuse MayLoad. NFC. llvm-svn: 246767
2015-09-03 22:19:43 +08:00
!(MayLoad && UsedRegs[getLdStRegOp(FirstMI).getReg()]) &&
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
!mayAlias(FirstMI, MemInsns, TII)) {
[AArch64] Simplify the passing of arguments. NFC. This is setup for future work planned for the AArch64 Load/Store Opt pass. llvm-svn: 242810
2015-07-22 01:42:04 +08:00
Flags.setMergeForward(true);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return MBBI;
}
// Unable to combine these instructions due to interference in between.
// Keep looking.
}
}
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
// If the instruction wasn't a matching load or store. Stop searching if we
// encounter a call instruction that might modify memory.
if (MI->isCall())
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return E;
// Update modified / uses register lists.
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
// Otherwise, if the base register is modified, we have no match, so
// return early.
if (ModifiedRegs[BaseReg])
return E;
[AArch64] Enhance the load/store optimizer with target-specific alias analysis. Phabricator: http://reviews.llvm.org/D9863 llvm-svn: 237963
2015-05-22 05:36:46 +08:00
// Update list of instructions that read/write memory.
if (MI->mayLoadOrStore())
MemInsns.push_back(MI);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
return E;
}
MachineBasicBlock::iterator
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
AArch64LoadStoreOpt::mergeUpdateInsn(MachineBasicBlock::iterator I,
MachineBasicBlock::iterator Update,
bool IsPreIdx) {
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
assert((Update->getOpcode() == AArch64::ADDXri ||
Update->getOpcode() == AArch64::SUBXri) &&
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
"Unexpected base register update instruction to merge!");
MachineBasicBlock::iterator NextI = I;
// Return the instruction following the merged instruction, which is
// the instruction following our unmerged load. Unless that's the add/sub
// instruction we're merging, in which case it's the one after that.
if (++NextI == Update)
++NextI;
int Value = Update->getOperand(2).getImm();
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
assert(AArch64_AM::getShiftValue(Update->getOperand(3).getImm()) == 0 &&
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
"Can't merge 1 << 12 offset into pre-/post-indexed load / store");
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
if (Update->getOpcode() == AArch64::SUBXri)
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
Value = -Value;
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
unsigned NewOpc = IsPreIdx ? getPreIndexedOpcode(I->getOpcode())
: getPostIndexedOpcode(I->getOpcode());
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
MachineInstrBuilder MIB;
if (!isPairedLdSt(I)) {
// Non-paired instruction.
MIB = BuildMI(*I->getParent(), I, I->getDebugLoc(), TII->get(NewOpc))
.addOperand(getLdStRegOp(Update))
.addOperand(getLdStRegOp(I))
.addOperand(getLdStBaseOp(I))
[AArch64] Set MMOs on pre- and post-index instructions. Without the MMOs the MI scheduler is unable to reason about the dependencies of these instructions. llvm-svn: 259052
2016-01-28 23:38:24 +08:00
.addImm(Value)
.setMemRefs(I->memoperands_begin(), I->memoperands_end());
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
} else {
// Paired instruction.
[AArch64] Scale offsets by the size of the memory operation. NFC. The immediate in the load/store should be scaled by the size of the memory operation, not the size of the register being loaded/stored. This change gets us one step closer to forming LDPSW instructions. This change also enables pre- and post-indexing for halfword and byte loads and stores. llvm-svn: 248804
2015-09-30 00:07:32 +08:00
int Scale = getMemScale(I);
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
MIB = BuildMI(*I->getParent(), I, I->getDebugLoc(), TII->get(NewOpc))
.addOperand(getLdStRegOp(Update))
.addOperand(getLdStRegOp(I, 0))
.addOperand(getLdStRegOp(I, 1))
.addOperand(getLdStBaseOp(I))
[AArch64] Set MMOs on pre- and post-index instructions. Without the MMOs the MI scheduler is unable to reason about the dependencies of these instructions. llvm-svn: 259052
2016-01-28 23:38:24 +08:00
.addImm(Value / Scale)
.setMemRefs(I->memoperands_begin(), I->memoperands_end());
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
(void)MIB;
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
if (IsPreIdx)
DEBUG(dbgs() << "Creating pre-indexed load/store.");
else
DEBUG(dbgs() << "Creating post-indexed load/store.");
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
DEBUG(dbgs() << " Replacing instructions:\n ");
DEBUG(I->print(dbgs()));
DEBUG(dbgs() << " ");
DEBUG(Update->print(dbgs()));
DEBUG(dbgs() << " with instruction:\n ");
DEBUG(((MachineInstr *)MIB)->print(dbgs()));
DEBUG(dbgs() << "\n");
// Erase the old instructions for the block.
I->eraseFromParent();
Update->eraseFromParent();
return NextI;
}
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
bool AArch64LoadStoreOpt::isMatchingUpdateInsn(MachineInstr *MemMI,
MachineInstr *MI,
unsigned BaseReg, int Offset) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
switch (MI->getOpcode()) {
default:
break;
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::SUBXri:
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Negate the offset for a SUB instruction.
Offset *= -1;
// FALLTHROUGH
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::ADDXri:
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Make sure it's a vanilla immediate operand, not a relocation or
// anything else we can't handle.
if (!MI->getOperand(2).isImm())
break;
// Watch out for 1 << 12 shifted value.
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
if (AArch64_AM::getShiftValue(MI->getOperand(3).getImm()))
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
break;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// The update instruction source and destination register must be the
// same as the load/store base register.
if (MI->getOperand(0).getReg() != BaseReg ||
MI->getOperand(1).getReg() != BaseReg)
break;
bool IsPairedInsn = isPairedLdSt(MemMI);
int UpdateOffset = MI->getOperand(2).getImm();
// For non-paired load/store instructions, the immediate must fit in a
// signed 9-bit integer.
if (!IsPairedInsn && (UpdateOffset > 255 || UpdateOffset < -256))
break;
// For paired load/store instructions, the immediate must be a multiple of
// the scaling factor. The scaled offset must also fit into a signed 7-bit
// integer.
if (IsPairedInsn) {
[AArch64] Scale offsets by the size of the memory operation. NFC. The immediate in the load/store should be scaled by the size of the memory operation, not the size of the register being loaded/stored. This change gets us one step closer to forming LDPSW instructions. This change also enables pre- and post-indexing for halfword and byte loads and stores. llvm-svn: 248804
2015-09-30 00:07:32 +08:00
int Scale = getMemScale(MemMI);
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
if (UpdateOffset % Scale != 0)
break;
int ScaledOffset = UpdateOffset / Scale;
if (ScaledOffset > 64 || ScaledOffset < -64)
break;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// If we have a non-zero Offset, we check that it matches the amount
// we're adding to the register.
if (!Offset || Offset == MI->getOperand(2).getImm())
return true;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
break;
}
return false;
}
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
MachineBasicBlock::iterator AArch64LoadStoreOpt::findMatchingUpdateInsnForward(
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
MachineBasicBlock::iterator I, int UnscaledOffset, unsigned Limit) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator E = I->getParent()->end();
MachineInstr *MemMI = I;
MachineBasicBlock::iterator MBBI = I;
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
unsigned BaseReg = getLdStBaseOp(MemMI).getReg();
[AArch64] Rename variable to improve readability. NFC. llvm-svn: 249008
2015-10-01 21:33:31 +08:00
int MIUnscaledOffset = getLdStOffsetOp(MemMI).getImm() * getMemScale(MemMI);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Hoist commonly failing check. NFC. llvm-svn: 249011
2015-10-01 21:43:05 +08:00
// Scan forward looking for post-index opportunities. Updating instructions
// can't be formed if the memory instruction doesn't have the offset we're
// looking for.
if (MIUnscaledOffset != UnscaledOffset)
return E;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// If the base register overlaps a destination register, we can't
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// merge the update.
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
bool IsPairedInsn = isPairedLdSt(MemMI);
for (unsigned i = 0, e = IsPairedInsn ? 2 : 1; i != e; ++i) {
unsigned DestReg = getLdStRegOp(MemMI, i).getReg();
if (DestReg == BaseReg || TRI->isSubRegister(BaseReg, DestReg))
return E;
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Track which registers have been modified and used between the first insn
// (inclusive) and the second insn.
[AArch64] Allocate the modified and used regs only once per function. llvm-svn: 259510
2016-02-02 23:02:30 +08:00
ModifiedRegs.reset();
UsedRegs.reset();
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
++MBBI;
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
for (unsigned Count = 0; MBBI != E && Count < Limit; ++MBBI) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineInstr *MI = MBBI;
[AArch64] Remove more dead code after r258093. llvm-svn: 258191
2016-01-20 05:27:05 +08:00
// Skip DBG_VALUE instructions.
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
if (MI->isDebugValue())
continue;
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
// Now that we know this is a real instruction, count it.
++Count;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// If we found a match, return it.
[AArch64] Rename variable to improve readability. NFC. llvm-svn: 249008
2015-10-01 21:33:31 +08:00
if (isMatchingUpdateInsn(I, MI, BaseReg, UnscaledOffset))
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return MBBI;
// Update the status of what the instruction clobbered and used.
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
// Otherwise, if the base register is used or modified, we have no match, so
// return early.
if (ModifiedRegs[BaseReg] || UsedRegs[BaseReg])
return E;
}
return E;
}
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
MachineBasicBlock::iterator AArch64LoadStoreOpt::findMatchingUpdateInsnBackward(
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
MachineBasicBlock::iterator I, unsigned Limit) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator B = I->getParent()->begin();
MachineBasicBlock::iterator E = I->getParent()->end();
MachineInstr *MemMI = I;
MachineBasicBlock::iterator MBBI = I;
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
unsigned BaseReg = getLdStBaseOp(MemMI).getReg();
int Offset = getLdStOffsetOp(MemMI).getImm();
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// If the load/store is the first instruction in the block, there's obviously
// not any matching update. Ditto if the memory offset isn't zero.
if (MBBI == B || Offset != 0)
return E;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// If the base register overlaps a destination register, we can't
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// merge the update.
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
bool IsPairedInsn = isPairedLdSt(MemMI);
for (unsigned i = 0, e = IsPairedInsn ? 2 : 1; i != e; ++i) {
unsigned DestReg = getLdStRegOp(MemMI, i).getReg();
if (DestReg == BaseReg || TRI->isSubRegister(BaseReg, DestReg))
return E;
}
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Track which registers have been modified and used between the first insn
// (inclusive) and the second insn.
[AArch64] Allocate the modified and used regs only once per function. llvm-svn: 259510
2016-02-02 23:02:30 +08:00
ModifiedRegs.reset();
UsedRegs.reset();
[AArch64] AArch64LoadStoreOptimizer: fix bug in pre-inc check iterator Summary: Fix case where a pre-inc/dec load/store would not be formed if the add/sub that forms the inc/dec part of the operation was the first instruction in the block being examined. Reviewers: mcrosier, jmolloy, t.p.northover, junbuml Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16785 llvm-svn: 260275
2016-02-10 04:47:21 +08:00
unsigned Count = 0;
do {
--MBBI;
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineInstr *MI = MBBI;
[AArch64] AArch64LoadStoreOptimizer: fix bug in pre-inc check iterator Summary: Fix case where a pre-inc/dec load/store would not be formed if the add/sub that forms the inc/dec part of the operation was the first instruction in the block being examined. Reviewers: mcrosier, jmolloy, t.p.northover, junbuml Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16785 llvm-svn: 260275
2016-02-10 04:47:21 +08:00
// Don't count DBG_VALUE instructions towards the search limit.
if (!MI->isDebugValue())
++Count;
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// If we found a match, return it.
[AArch64] Remove an unnecessary restriction on pre-index instructions. Previously, the index was constrained to the size of the memory operation for no apparent reason. This change removes that constraint so that we can form pre-index instructions with any valid offset. llvm-svn: 248931
2015-10-01 03:44:40 +08:00
if (isMatchingUpdateInsn(I, MI, BaseReg, Offset))
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return MBBI;
// Update the status of what the instruction clobbered and used.
trackRegDefsUses(MI, ModifiedRegs, UsedRegs, TRI);
// Otherwise, if the base register is used or modified, we have no match, so
// return early.
if (ModifiedRegs[BaseReg] || UsedRegs[BaseReg])
return E;
[AArch64] AArch64LoadStoreOptimizer: fix bug in pre-inc check iterator Summary: Fix case where a pre-inc/dec load/store would not be formed if the add/sub that forms the inc/dec part of the operation was the first instruction in the block being examined. Reviewers: mcrosier, jmolloy, t.p.northover, junbuml Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16785 llvm-svn: 260275
2016-02-10 04:47:21 +08:00
} while (MBBI != B && Count < Limit);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return E;
}
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
bool AArch64LoadStoreOpt::tryToPromoteLoadFromStore(
MachineBasicBlock::iterator &MBBI) {
MachineInstr *MI = MBBI;
// If this is a volatile load, don't mess with it.
if (MI->hasOrderedMemoryRef())
return false;
// Make sure this is a reg+imm.
// FIXME: It is possible to extend it to handle reg+reg cases.
if (!getLdStOffsetOp(MI).isImm())
return false;
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
// Look backward up to LdStLimit instructions.
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
MachineBasicBlock::iterator StoreI;
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
if (findMatchingStore(MBBI, LdStLimit, StoreI)) {
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
++NumLoadsFromStoresPromoted;
// Promote the load. Keeping the iterator straight is a
// pain, so we let the merge routine tell us what the next instruction
// is after it's done mucking about.
MBBI = promoteLoadFromStore(MBBI, StoreI);
return true;
}
return false;
}
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
// Find narrow loads that can be converted into a single wider load with
// bitfield extract instructions. Also merge adjacent zero stores into a wider
// store.
bool AArch64LoadStoreOpt::tryToMergeLdStInst(
MachineBasicBlock::iterator &MBBI) {
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
assert((isNarrowLoad(MBBI) || isPromotableZeroStoreOpcode(MBBI)) &&
"Expected narrow op.");
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
MachineInstr *MI = MBBI;
MachineBasicBlock::iterator E = MI->getParent()->end();
[AArch64] Enable more load clustering in the MI Scheduler. This patch adds unscaled loads and sign-extend loads to the TII getMemOpBaseRegImmOfs API, which is used to control clustering in the MI scheduler. This is done to create more opportunities for load pairing. I've also added the scaled LDRSWui instruction, which was missing from the scaled instructions. Finally, I've added support in shouldClusterLoads for clustering adjacent sext and zext loads that too can be paired by the load/store optimizer. Differential Revision: http://reviews.llvm.org/D18048 llvm-svn: 263819
2016-03-19 03:21:02 +08:00
if (!TII->isCandidateToMergeOrPair(MI))
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
return false;
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
// For promotable zero stores, the stored value should be WZR.
if (isPromotableZeroStoreOpcode(MI) &&
getLdStRegOp(MI).getReg() != AArch64::WZR)
[AArch64] This check is specific to merging instructions. NFC. llvm-svn: 260283
2016-02-10 05:20:12 +08:00
return false;
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
// Look ahead up to LdStLimit instructions for a mergable instruction.
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
LdStPairFlags Flags;
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
MachineBasicBlock::iterator MergeMI =
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
findMatchingInsn(MBBI, Flags, LdStLimit, /* FindNarrowMerge = */ true);
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
if (MergeMI != E) {
[AArch64] Refactoring aarch64-ldst-opt. NCF. Summary : * Rename isSmallTypeLdMerge() to isNarrowLoad(). * Rename NumSmallTypeMerged to NumNarrowTypePromoted. * Use Subtarget defined as a member variable. llvm-svn: 253587
2015-11-20 02:41:27 +08:00
if (isNarrowLoad(MI)) {
++NumNarrowLoadsPromoted;
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
} else if (isPromotableZeroStoreInst(MI)) {
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
++NumZeroStoresPromoted;
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
}
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
// Keeping the iterator straight is a pain, so we let the merge routine tell
// us what the next instruction is after it's done mucking about.
[AArch64] Rename variable to make it clear we're merging here, not pairing. llvm-svn: 260256
2016-02-10 03:09:22 +08:00
MBBI = mergeNarrowInsns(MBBI, MergeMI, Flags);
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
return true;
}
return false;
}
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
// Find loads and stores that can be merged into a single load or store pair
// instruction.
bool AArch64LoadStoreOpt::tryToPairLdStInst(MachineBasicBlock::iterator &MBBI) {
MachineInstr *MI = MBBI;
MachineBasicBlock::iterator E = MI->getParent()->end();
[AArch64] Enable more load clustering in the MI Scheduler. This patch adds unscaled loads and sign-extend loads to the TII getMemOpBaseRegImmOfs API, which is used to control clustering in the MI scheduler. This is done to create more opportunities for load pairing. I've also added the scaled LDRSWui instruction, which was missing from the scaled instructions. Finally, I've added support in shouldClusterLoads for clustering adjacent sext and zext loads that too can be paired by the load/store optimizer. Differential Revision: http://reviews.llvm.org/D18048 llvm-svn: 263819
2016-03-19 03:21:02 +08:00
if (!TII->isCandidateToMergeOrPair(MI))
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
return false;
[AArch64] This bit of logic is specific to pairing. NFC. llvm-svn: 260383
2016-02-10 23:52:46 +08:00
// Early exit if the offset is not possible to match. (6 bits of positive
// range, plus allow an extra one in case we find a later insn that matches
// with Offset-1)
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
bool IsUnscaled = TII->isUnscaledLdSt(MI);
[AArch64] This bit of logic is specific to pairing. NFC. llvm-svn: 260383
2016-02-10 23:52:46 +08:00
int Offset = getLdStOffsetOp(MI).getImm();
int OffsetStride = IsUnscaled ? getMemScale(MI) : 1;
if (!inBoundsForPair(IsUnscaled, Offset, OffsetStride))
return false;
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
// Look ahead up to LdStLimit instructions for a pairable instruction.
LdStPairFlags Flags;
[AArch64] Handle missing store pair opportunity Summary: This change will handle missing store pair opportunity where the first store instruction stores zero followed by the non-zero store. For example, this change will convert : str wzr, [x8] str w1, [x8, #4] into: stp wzr, w1, [x8] Reviewers: jmolloy, t.p.northover, mcrosier Subscribers: flyingforyou, aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D18570 llvm-svn: 265021
2016-03-31 22:47:24 +08:00
MachineBasicBlock::iterator Paired =
findMatchingInsn(MBBI, Flags, LdStLimit, /* FindNarrowMerge = */ false);
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
if (Paired != E) {
++NumPairCreated;
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
if (TII->isUnscaledLdSt(MI))
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
++NumUnscaledPairCreated;
// Keeping the iterator straight is a pain, so we let the merge routine tell
// us what the next instruction is after it's done mucking about.
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
MBBI = mergePairedInsns(MBBI, Paired, Flags);
return true;
}
return false;
}
[AArch64]Enable the narrow ld promotion only on profitable microarchitectures The benefit from converting narrow loads into a wider load (r251438) could be micro-architecturally dependent, as it assumes that a single load with two bitfield extracts is cheaper than two narrow loads. Currently, this conversion is enabled only in cortex-a57 on which performance benefits were verified. llvm-svn: 252316
2015-11-07 00:27:47 +08:00
bool AArch64LoadStoreOpt::optimizeBlock(MachineBasicBlock &MBB,
bool enableNarrowLdOpt) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
bool Modified = false;
Move comments a bit closer to associated code. NFC. llvm-svn: 259411
2016-02-02 05:38:31 +08:00
// Four tranformations to do here:
[AArch64] Promote loads from stored This is a recommit of r256004 which was reverted in r256160. The issue was the incorrect promotion for half and byte loads transformed into mov instructions. This fix will replace half and byte type loads only with bit field extracts. Original commit message: This change promotes load instructions which directly read from stored by replacing them with mov instructions. If the store is wider than the load, the load will be replaced with a bitfield extract. For example : STRWui %W1, %X0, 1 %W0 = LDRHHui %X0, 3 becomes STRWui %W1, %X0, 1 %W0 = UBFMWri %W1, 16, 31 llvm-svn: 256249
2015-12-23 00:36:16 +08:00
// 1) Find loads that directly read from stores and promote them by
// replacing with mov instructions. If the store is wider than the load,
// the load will be replaced with a bitfield extract.
// e.g.,
// str w1, [x0, #4]
// ldrh w2, [x0, #6]
// ; becomes
// str w1, [x0, #4]
// lsr w2, w1, #16
for (MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
MBBI != E;) {
MachineInstr *MI = MBBI;
switch (MI->getOpcode()) {
default:
// Just move on to the next instruction.
++MBBI;
break;
// Scaled instructions.
case AArch64::LDRBBui:
case AArch64::LDRHHui:
case AArch64::LDRWui:
case AArch64::LDRXui:
// Unscaled instructions.
case AArch64::LDURBBi:
case AArch64::LDURHHi:
case AArch64::LDURWi:
case AArch64::LDURXi: {
if (tryToPromoteLoadFromStore(MBBI)) {
Modified = true;
break;
}
++MBBI;
break;
}
}
}
Move comments a bit closer to associated code. NFC. llvm-svn: 259411
2016-02-02 05:38:31 +08:00
// 2) Find narrow loads that can be converted into a single wider load
// with bitfield extract instructions.
// e.g.,
// ldrh w0, [x2]
// ldrh w1, [x2, #2]
// ; becomes
// ldr w0, [x2]
// ubfx w1, w0, #16, #16
// and w0, w0, #ffff
[AArch64] Refactoring aarch64-ldst-opt. NCF. Remove narrow load / store instructions from getMatchingPairOpcode(), and add getMatchingWideOpcode(). llvm-svn: 259914
2016-02-06 04:02:03 +08:00
//
// Also merge adjacent zero stores into a wider store.
// e.g.,
// strh wzr, [x0]
// strh wzr, [x0, #2]
// ; becomes
// str wzr, [x0]
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
for (MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
[AArch64]Enable the narrow ld promotion only on profitable microarchitectures The benefit from converting narrow loads into a wider load (r251438) could be micro-architecturally dependent, as it assumes that a single load with two bitfield extracts is cheaper than two narrow loads. Currently, this conversion is enabled only in cortex-a57 on which performance benefits were verified. llvm-svn: 252316
2015-11-07 00:27:47 +08:00
enableNarrowLdOpt && MBBI != E;) {
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
MachineInstr *MI = MBBI;
switch (MI->getOpcode()) {
default:
// Just move on to the next instruction.
++MBBI;
break;
// Scaled instructions.
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
case AArch64::LDRBBui:
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
case AArch64::LDRHHui:
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
case AArch64::LDRSBWui:
case AArch64::LDRSHWui:
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
case AArch64::STRBBui:
case AArch64::STRHHui:
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
case AArch64::STRWui:
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
// Unscaled instructions.
[AArch64]Extend merging narrow loads into a wider load This change extends r251438 to handle more narrow load promotions including byte type, unscaled, and signed. For example, this change will convert : ldursh w1, [x0, #-2] ldurh w2, [x0, #-4] into ldur w2, [x0, #-4] asr w1, w2, #16 and w2, w2, #0xffff llvm-svn: 253577
2015-11-20 01:21:41 +08:00
case AArch64::LDURBBi:
case AArch64::LDURHHi:
case AArch64::LDURSBWi:
[AArch64]Merge narrow zero stores to a wider store This change merges adjacent zero stores into a wider single store. For example : strh wzr, [x0] strh wzr, [x0, #2] becomes str wzr, [x0] This will fix PR25410. llvm-svn: 253711
2015-11-21 05:14:07 +08:00
case AArch64::LDURSHWi:
case AArch64::STURBBi:
[AArch64] Merge two adjacent str WZR into str XZR Summary: This change merges adjacent 32 bit zero stores into a 64 bit zero store. e.g., str wzr, [x0] str wzr, [x0, #4] becomes str xzr, [x0] Therefore, four adjacent 32 bit zero stores will be a single stp. e.g., str wzr, [x0] str wzr, [x0, #4] str wzr, [x0, #8] str wzr, [x0, #12] becomes stp xzr, xzr, [x0] Reviewers: mcrosier, jmolloy, gberry, t.p.northover Subscribers: aemerson, rengolin, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D16933 llvm-svn: 260682
2016-02-12 23:25:39 +08:00
case AArch64::STURHHi:
case AArch64::STURWi: {
[AArch64]Merge halfword loads into a 32-bit load This recommits r250719, which caused a failure in SPEC2000.gcc because of the incorrect insert point for the new wider load. Convert two halfword loads into a single 32-bit word load with bitfield extract instructions. For example : ldrh w0, [x2] ldrh w1, [x2, #2] becomes ldr w0, [x2] ubfx w1, w0, #16, #16 and w0, w0, #ffff llvm-svn: 251438
2015-10-28 03:16:03 +08:00
if (tryToMergeLdStInst(MBBI)) {
Modified = true;
break;
}
++MBBI;
break;
}
}
}
Move comments a bit closer to associated code. NFC. llvm-svn: 259411
2016-02-02 05:38:31 +08:00
// 3) Find loads and stores that can be merged into a single load or store
// pair instruction.
// e.g.,
// ldr x0, [x2]
// ldr x1, [x2, #8]
// ; becomes
// ldp x0, x1, [x2]
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
for (MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
MBBI != E;) {
MachineInstr *MI = MBBI;
switch (MI->getOpcode()) {
default:
// Just move on to the next instruction.
++MBBI;
break;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// Scaled instructions.
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::STRSui:
case AArch64::STRDui:
case AArch64::STRQui:
case AArch64::STRXui:
case AArch64::STRWui:
case AArch64::LDRSui:
case AArch64::LDRDui:
case AArch64::LDRQui:
case AArch64::LDRXui:
case AArch64::LDRWui:
[AArch64][LoadStoreOptimizer] Form LDPSW when possible. This patch adds the missing LD[U]RSW variants to the load store optimizer, so that we generate LDPSW when possible. <rdar://problem/19583480> llvm-svn: 226978
2015-01-24 09:25:54 +08:00
case AArch64::LDRSWui:
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// Unscaled instructions.
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::STURSi:
case AArch64::STURDi:
case AArch64::STURQi:
case AArch64::STURWi:
case AArch64::STURXi:
case AArch64::LDURSi:
case AArch64::LDURDi:
case AArch64::LDURQi:
case AArch64::LDURWi:
[AArch64][LoadStoreOptimizer] Form LDPSW when possible. This patch adds the missing LD[U]RSW variants to the load store optimizer, so that we generate LDPSW when possible. <rdar://problem/19583480> llvm-svn: 226978
2015-01-24 09:25:54 +08:00
case AArch64::LDURXi:
case AArch64::LDURSWi: {
[AArch64] Cleanup to simplify logic when widening vs. pairing loads/stores. NFC. The logic to pair instructions and merge narrow instructions has become cloogy and error prone. This patch beings to unravel these two similar, but distinct optimizations. llvm-svn: 260242
2016-02-10 02:10:20 +08:00
if (tryToPairLdStInst(MBBI)) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
Modified = true;
break;
}
++MBBI;
break;
}
}
}
Move comments a bit closer to associated code. NFC. llvm-svn: 259411
2016-02-02 05:38:31 +08:00
// 4) Find base register updates that can be merged into the load or store
// as a base-reg writeback.
// e.g.,
// ldr x0, [x2]
// add x2, x2, #4
// ; becomes
// ldr x0, [x2], #4
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
for (MachineBasicBlock::iterator MBBI = MBB.begin(), E = MBB.end();
MBBI != E;) {
MachineInstr *MI = MBBI;
// Do update merging. It's simpler to keep this separate from the above
Move comments a bit closer to associated code. NFC. llvm-svn: 259411
2016-02-02 05:38:31 +08:00
// switchs, though not strictly necessary.
MachineInstr: Change return value of getOpcode() to unsigned. This was previously returning int. However there are no negative opcode numbers and more importantly this was needlessly different from MCInstrDesc::getOpcode() (which even is the value returned here) and SDValue::getOpcode()/SDNode::getOpcode(). llvm-svn: 237611
2015-05-19 04:27:55 +08:00
unsigned Opc = MI->getOpcode();
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
switch (Opc) {
default:
// Just move on to the next instruction.
++MBBI;
break;
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// Scaled instructions.
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::STRSui:
case AArch64::STRDui:
case AArch64::STRQui:
case AArch64::STRXui:
case AArch64::STRWui:
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::STRHHui:
case AArch64::STRBBui:
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::LDRSui:
case AArch64::LDRDui:
case AArch64::LDRQui:
case AArch64::LDRXui:
case AArch64::LDRWui:
[AArch64] Add integer pre- and post-index halfword/byte loads and stores. llvm-svn: 248817
2015-09-30 02:26:15 +08:00
case AArch64::LDRHHui:
case AArch64::LDRBBui:
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// Unscaled instructions.
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
case AArch64::STURSi:
case AArch64::STURDi:
case AArch64::STURQi:
case AArch64::STURWi:
case AArch64::STURXi:
case AArch64::LDURSi:
case AArch64::LDURDi:
case AArch64::LDURQi:
case AArch64::LDURWi:
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
case AArch64::LDURXi:
// Paired instructions.
case AArch64::LDPSi:
[AArch64] Add support for pre- and post-index LDPSWs. llvm-svn: 248825
2015-09-30 04:39:55 +08:00
case AArch64::LDPSWi:
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
case AArch64::LDPDi:
case AArch64::LDPQi:
case AArch64::LDPWi:
case AArch64::LDPXi:
case AArch64::STPSi:
case AArch64::STPDi:
case AArch64::STPQi:
case AArch64::STPWi:
case AArch64::STPXi: {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Make sure this is a reg+imm (as opposed to an address reloc).
[AArch64] Improve the readability of the ld/st optimization pass. NFC. llvm-svn: 244222
2015-08-06 23:50:12 +08:00
if (!getLdStOffsetOp(MI).isImm()) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
++MBBI;
break;
}
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// Look forward to try to form a post-index instruction. For example,
// ldr x0, [x20]
// add x20, x20, #32
// merged into:
// ldr x0, [x20], #32
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
MachineBasicBlock::iterator Update =
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
findMatchingUpdateInsnForward(MBBI, 0, UpdateLimit);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
if (Update != E) {
// Merge the update into the ld/st.
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
MBBI = mergeUpdateInsn(MBBI, Update, /*IsPreIdx=*/false);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
Modified = true;
++NumPostFolded;
break;
}
// Don't know how to handle pre/post-index versions, so move to the next
// instruction.
[AArch64] Move helper functions into TII, so they can be reused elsewhere. NFC. llvm-svn: 263032
2016-03-10 01:29:48 +08:00
if (TII->isUnscaledLdSt(Opc)) {
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
++MBBI;
break;
}
// Look back to try to find a pre-index instruction. For example,
// add x0, x0, #8
// ldr x1, [x0]
// merged into:
// ldr x1, [x0, #8]!
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
Update = findMatchingUpdateInsnBackward(MBBI, UpdateLimit);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
if (Update != E) {
// Merge the update into the ld/st.
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
MBBI = mergeUpdateInsn(MBBI, Update, /*IsPreIdx=*/true);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
Modified = true;
++NumPreFolded;
break;
}
[AArch64] Update comment to reflect reality. llvm-svn: 249007
2015-10-01 21:09:44 +08:00
// The immediate in the load/store is scaled by the size of the memory
// operation. The immediate in the add we're looking for,
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
// however, is not, so adjust here.
[AArch64] Rename variable to improve readability. NFC. llvm-svn: 249008
2015-10-01 21:33:31 +08:00
int UnscaledOffset = getLdStOffsetOp(MI).getImm() * getMemScale(MI);
[AArch64] Add support for generating pre- and post-index load/store pairs. llvm-svn: 248593
2015-09-26 01:48:17 +08:00
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
// Look forward to try to find a post-index instruction. For example,
// ldr x1, [x0, #64]
// add x0, x0, #64
// merged into:
[ARM64] Fix wrong comment in load/store optimization pass. ldr x1, [x0, #64] add x0, x0, #64 -> ldr x1, [x0], #64 is not a valid transformation, the correct transformation (and what the code actually does) is: ldr x1, [x0, #64] add x0, x0, #64 -> ldr x1, [x0, #64]! llvm-svn: 208998
2014-05-17 00:50:13 +08:00
// ldr x1, [x0, #64]!
[AArch64] Bound the number of instructions we scan when searching for updates. This only impacts the creation of pre-/post-index instructions. The bound was set high enough such that it did not change code generation for SPEC200X. llvm-svn: 259828
2016-02-05 05:26:02 +08:00
Update = findMatchingUpdateInsnForward(MBBI, UnscaledOffset, UpdateLimit);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
if (Update != E) {
// Merge the update into the ld/st.
[AArch64] Refactor pre- and post-index merge fuctions into a single function. NFC. llvm-svn: 248377
2015-09-23 21:51:44 +08:00
MBBI = mergeUpdateInsn(MBBI, Update, /*IsPreIdx=*/true);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
Modified = true;
++NumPreFolded;
break;
}
// Nothing found. Just move to the next instruction.
++MBBI;
break;
}
}
}
return Modified;
}
[AArch64]Enable the narrow ld promotion only on profitable microarchitectures The benefit from converting narrow loads into a wider load (r251438) could be micro-architecturally dependent, as it assumes that a single load with two bitfield extracts is cheaper than two narrow loads. Currently, this conversion is enabled only in cortex-a57 on which performance benefits were verified. llvm-svn: 252316
2015-11-07 00:27:47 +08:00
bool AArch64LoadStoreOpt::enableNarrowLdMerge(MachineFunction &Fn) {
[AArch64] Add support for Qualcomm Kryo CPU. Machine model description by Dave Estes <cestes@codeaurora.org>. llvm-svn: 260686
2016-02-12 23:51:51 +08:00
bool ProfitableArch = Subtarget->isCortexA57() || Subtarget->isKryo();
[AArch64]Enable the narrow ld promotion only on profitable microarchitectures The benefit from converting narrow loads into a wider load (r251438) could be micro-architecturally dependent, as it assumes that a single load with two bitfield extracts is cheaper than two narrow loads. Currently, this conversion is enabled only in cortex-a57 on which performance benefits were verified. llvm-svn: 252316
2015-11-07 00:27:47 +08:00
// FIXME: The benefit from converting narrow loads into a wider load could be
// microarchitectural as it assumes that a single load with two bitfield
// extracts is cheaper than two narrow loads. Currently, this conversion is
// enabled only in cortex-a57 on which performance benefits were verified.
[AArch64] Refactoring aarch64-ldst-opt. NCF. Summary : * Rename isSmallTypeLdMerge() to isNarrowLoad(). * Rename NumSmallTypeMerged to NumNarrowTypePromoted. * Use Subtarget defined as a member variable. llvm-svn: 253587
2015-11-20 02:41:27 +08:00
return ProfitableArch && !Subtarget->requiresStrictAlign();
[AArch64]Enable the narrow ld promotion only on profitable microarchitectures The benefit from converting narrow loads into a wider load (r251438) could be micro-architecturally dependent, as it assumes that a single load with two bitfield extracts is cheaper than two narrow loads. Currently, this conversion is enabled only in cortex-a57 on which performance benefits were verified. llvm-svn: 252316
2015-11-07 00:27:47 +08:00
}
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
bool AArch64LoadStoreOpt::runOnMachineFunction(MachineFunction &Fn) {
[AArch64] Fix halfword load merging for big-endian targets For big-endian targets, when we merge two halfword loads into a word load, the order of the halfwords in the loaded value is reversed compared to little-endian, so the load-store optimiser needs to swap the destination registers. This does not affect merging of two word loads, as we use ldp, which treats the memory as two separate 32-bit words. llvm-svn: 252597
2015-11-10 19:04:18 +08:00
Subtarget = &static_cast<const AArch64Subtarget &>(Fn.getSubtarget());
TII = static_cast<const AArch64InstrInfo *>(Subtarget->getInstrInfo());
TRI = Subtarget->getRegisterInfo();
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
[AArch64] Allocate the modified and used regs only once per function. llvm-svn: 259510
2016-02-02 23:02:30 +08:00
// Resize the modified and used register bitfield trackers. We do this once
// per function and then clear the bitfield each time we optimize a load or
// store.
ModifiedRegs.resize(TRI->getNumRegs());
UsedRegs.resize(TRI->getNumRegs());
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
bool Modified = false;
[AArch64]Enable the narrow ld promotion only on profitable microarchitectures The benefit from converting narrow loads into a wider load (r251438) could be micro-architecturally dependent, as it assumes that a single load with two bitfield extracts is cheaper than two narrow loads. Currently, this conversion is enabled only in cortex-a57 on which performance benefits were verified. llvm-svn: 252316
2015-11-07 00:27:47 +08:00
bool enableNarrowLdOpt = enableNarrowLdMerge(Fn);
[C++11,ARM64] Range based for loops in load/store pair optimizer. No functional change intended. llvm-svn: 205444
2014-04-03 02:00:53 +08:00
for (auto &MBB : Fn)
[AArch64]Enable the narrow ld promotion only on profitable microarchitectures The benefit from converting narrow loads into a wider load (r251438) could be micro-architecturally dependent, as it assumes that a single load with two bitfield extracts is cheaper than two narrow loads. Currently, this conversion is enabled only in cortex-a57 on which performance benefits were verified. llvm-svn: 252316
2015-11-07 00:27:47 +08:00
Modified |= optimizeBlock(MBB, enableNarrowLdOpt);
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
return Modified;
}
// FIXME: Do we need/want a pre-alloc pass like ARM has to try to keep
// loads and stores near one another?
[AArch64] Add a FIXME about invalid KILL markers after the ld/st opt pass. llvm-svn: 260264
2016-02-10 03:42:19 +08:00
// FIXME: When pairing store instructions it's very possible for this pass to
// hoist a store with a KILL marker above another use (without a KILL marker).
// The resulting IR is invalid, but nothing uses the KILL markers after this
// pass, so it's never caused a problem in practice.
Update comment. NFC. llvm-svn: 244038
2015-08-05 20:40:13 +08:00
/// createAArch64LoadStoreOptimizationPass - returns an instance of the
/// load / store optimization pass.
AArch64/ARM64: move ARM64 into AArch64's place This commit starts with a "git mv ARM64 AArch64" and continues out from there, renaming the C++ classes, intrinsics, and other target-local objects for consistency. "ARM64" test directories are also moved, and tests that began their life in ARM64 use an arm64 triple, those from AArch64 use an aarch64 triple. Both should be equivalent though. This finishes the AArch64 merge, and everyone should feel free to continue committing as normal now. llvm-svn: 209577
2014-05-24 20:50:23 +08:00
FunctionPass *llvm::createAArch64LoadStoreOptimizationPass() {
return new AArch64LoadStoreOpt();
ARM64: initial backend import This adds a second implementation of the AArch64 architecture to LLVM, accessible in parallel via the "arm64" triple. The plan over the coming weeks & months is to merge the two into a single backend, during which time thorough code review should naturally occur. Everything will be easier with the target in-tree though, hence this commit. llvm-svn: 205090
2014-03-29 18:18:08 +08:00
}