2017-09-23 07:46:57 +08:00
|
|
|
//===- InterleavedAccessPass.cpp ------------------------------------------===//
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
//
|
2019-01-19 16:50:56 +08:00
|
|
|
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
|
|
|
|
// See https://llvm.org/LICENSE.txt for license information.
|
|
|
|
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
//
|
|
|
|
//===----------------------------------------------------------------------===//
|
|
|
|
//
|
|
|
|
// This file implements the Interleaved Access pass, which identifies
|
2016-05-02 22:32:17 +08:00
|
|
|
// interleaved memory accesses and transforms them into target specific
|
|
|
|
// intrinsics.
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
//
|
|
|
|
// An interleaved load reads data from memory into several vectors, with
|
|
|
|
// DE-interleaving the data on a factor. An interleaved store writes several
|
|
|
|
// vectors to memory with RE-interleaving the data on a factor.
|
|
|
|
//
|
2016-05-02 22:32:17 +08:00
|
|
|
// As interleaved accesses are difficult to identified in CodeGen (mainly
|
|
|
|
// because the VECTOR_SHUFFLE DAG node is quite different from the shufflevector
|
|
|
|
// IR), we identify and transform them to intrinsics in this pass so the
|
|
|
|
// intrinsics can be easily matched into target specific instructions later in
|
|
|
|
// CodeGen.
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
//
|
|
|
|
// E.g. An interleaved load (Factor = 2):
|
|
|
|
// %wide.vec = load <8 x i32>, <8 x i32>* %ptr
|
2021-01-10 16:22:54 +08:00
|
|
|
// %v0 = shuffle <8 x i32> %wide.vec, <8 x i32> poison, <0, 2, 4, 6>
|
|
|
|
// %v1 = shuffle <8 x i32> %wide.vec, <8 x i32> poison, <1, 3, 5, 7>
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
//
|
|
|
|
// It could be transformed into a ld2 intrinsic in AArch64 backend or a vld2
|
|
|
|
// intrinsic in ARM backend.
|
|
|
|
//
|
2016-10-15 02:20:41 +08:00
|
|
|
// In X86, this can be further optimized into a set of target
|
|
|
|
// specific loads followed by an optimized sequence of shuffles.
|
|
|
|
//
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
// E.g. An interleaved store (Factor = 3):
|
|
|
|
// %i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1,
|
|
|
|
// <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
|
|
|
|
// store <12 x i32> %i.vec, <12 x i32>* %ptr
|
|
|
|
//
|
|
|
|
// It could be transformed into a st3 intrinsic in AArch64 backend or a vst3
|
|
|
|
// intrinsic in ARM backend.
|
|
|
|
//
|
2016-10-15 02:20:41 +08:00
|
|
|
// Similarly, a set of interleaved stores can be transformed into an optimized
|
|
|
|
// sequence of shuffles followed by a set of target specific stores for X86.
|
2017-09-23 07:46:57 +08:00
|
|
|
//
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
//===----------------------------------------------------------------------===//
|
|
|
|
|
2017-09-23 07:46:57 +08:00
|
|
|
#include "llvm/ADT/ArrayRef.h"
|
|
|
|
#include "llvm/ADT/DenseMap.h"
|
|
|
|
#include "llvm/ADT/SmallVector.h"
|
2017-11-17 09:07:10 +08:00
|
|
|
#include "llvm/CodeGen/TargetLowering.h"
|
2017-05-19 01:21:13 +08:00
|
|
|
#include "llvm/CodeGen/TargetPassConfig.h"
|
2017-11-17 09:07:10 +08:00
|
|
|
#include "llvm/CodeGen/TargetSubtargetInfo.h"
|
2017-09-23 07:46:57 +08:00
|
|
|
#include "llvm/IR/Constants.h"
|
2016-05-20 05:39:00 +08:00
|
|
|
#include "llvm/IR/Dominators.h"
|
2017-09-23 07:46:57 +08:00
|
|
|
#include "llvm/IR/Function.h"
|
|
|
|
#include "llvm/IR/IRBuilder.h"
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
#include "llvm/IR/InstIterator.h"
|
2017-09-23 07:46:57 +08:00
|
|
|
#include "llvm/IR/Instruction.h"
|
|
|
|
#include "llvm/IR/Instructions.h"
|
|
|
|
#include "llvm/IR/Type.h"
|
Sink all InitializePasses.h includes
This file lists every pass in LLVM, and is included by Pass.h, which is
very popular. Every time we add, remove, or rename a pass in LLVM, it
caused lots of recompilation.
I found this fact by looking at this table, which is sorted by the
number of times a file was changed over the last 100,000 git commits
multiplied by the number of object files that depend on it in the
current checkout:
recompiles touches affected_files header
342380 95 3604 llvm/include/llvm/ADT/STLExtras.h
314730 234 1345 llvm/include/llvm/InitializePasses.h
307036 118 2602 llvm/include/llvm/ADT/APInt.h
213049 59 3611 llvm/include/llvm/Support/MathExtras.h
170422 47 3626 llvm/include/llvm/Support/Compiler.h
162225 45 3605 llvm/include/llvm/ADT/Optional.h
158319 63 2513 llvm/include/llvm/ADT/Triple.h
140322 39 3598 llvm/include/llvm/ADT/StringRef.h
137647 59 2333 llvm/include/llvm/Support/Error.h
131619 73 1803 llvm/include/llvm/Support/FileSystem.h
Before this change, touching InitializePasses.h would cause 1345 files
to recompile. After this change, touching it only causes 550 compiles in
an incremental rebuild.
Reviewers: bkramer, asbirlea, bollu, jdoerfert
Differential Revision: https://reviews.llvm.org/D70211
2019-11-14 05:15:01 +08:00
|
|
|
#include "llvm/InitializePasses.h"
|
2017-09-23 07:46:57 +08:00
|
|
|
#include "llvm/Pass.h"
|
|
|
|
#include "llvm/Support/Casting.h"
|
|
|
|
#include "llvm/Support/CommandLine.h"
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
#include "llvm/Support/Debug.h"
|
|
|
|
#include "llvm/Support/MathExtras.h"
|
2015-06-26 12:38:21 +08:00
|
|
|
#include "llvm/Support/raw_ostream.h"
|
2017-09-23 07:46:57 +08:00
|
|
|
#include "llvm/Target/TargetMachine.h"
|
2020-10-29 17:13:23 +08:00
|
|
|
#include "llvm/Transforms/Utils/Local.h"
|
2017-09-23 07:46:57 +08:00
|
|
|
#include <cassert>
|
|
|
|
#include <utility>
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
|
|
|
using namespace llvm;
|
|
|
|
|
|
|
|
#define DEBUG_TYPE "interleaved-access"
|
|
|
|
|
|
|
|
static cl::opt<bool> LowerInterleavedAccesses(
|
|
|
|
"lower-interleaved-accesses",
|
|
|
|
cl::desc("Enable lowering interleaved accesses to intrinsics"),
|
2015-09-01 19:12:35 +08:00
|
|
|
cl::init(true), cl::Hidden);
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
|
|
|
namespace {
|
|
|
|
|
|
|
|
class InterleavedAccess : public FunctionPass {
|
|
|
|
public:
|
|
|
|
static char ID;
|
2017-09-23 07:46:57 +08:00
|
|
|
|
|
|
|
InterleavedAccess() : FunctionPass(ID) {
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
initializeInterleavedAccessPass(*PassRegistry::getPassRegistry());
|
|
|
|
}
|
|
|
|
|
2016-10-01 10:56:57 +08:00
|
|
|
StringRef getPassName() const override { return "Interleaved Access Pass"; }
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
|
|
|
bool runOnFunction(Function &F) override;
|
|
|
|
|
2016-05-20 05:39:00 +08:00
|
|
|
void getAnalysisUsage(AnalysisUsage &AU) const override {
|
|
|
|
AU.addRequired<DominatorTreeWrapperPass>();
|
|
|
|
AU.addPreserved<DominatorTreeWrapperPass>();
|
|
|
|
}
|
|
|
|
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
private:
|
2017-09-23 07:46:57 +08:00
|
|
|
DominatorTree *DT = nullptr;
|
|
|
|
const TargetLowering *TLI = nullptr;
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
2016-10-19 02:59:58 +08:00
|
|
|
/// The maximum supported interleave factor.
|
|
|
|
unsigned MaxFactor;
|
|
|
|
|
2018-05-01 23:54:18 +08:00
|
|
|
/// Transform an interleaved load into target specific intrinsics.
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
bool lowerInterleavedLoad(LoadInst *LI,
|
|
|
|
SmallVector<Instruction *, 32> &DeadInsts);
|
|
|
|
|
2018-05-01 23:54:18 +08:00
|
|
|
/// Transform an interleaved store into target specific intrinsics.
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
bool lowerInterleavedStore(StoreInst *SI,
|
|
|
|
SmallVector<Instruction *, 32> &DeadInsts);
|
2016-05-20 05:39:00 +08:00
|
|
|
|
2018-05-01 23:54:18 +08:00
|
|
|
/// Returns true if the uses of an interleaved load by the
|
2016-05-20 05:39:00 +08:00
|
|
|
/// extractelement instructions in \p Extracts can be replaced by uses of the
|
|
|
|
/// shufflevector instructions in \p Shuffles instead. If so, the necessary
|
|
|
|
/// replacements are also performed.
|
|
|
|
bool tryReplaceExtracts(ArrayRef<ExtractElementInst *> Extracts,
|
|
|
|
ArrayRef<ShuffleVectorInst *> Shuffles);
|
2020-10-29 17:13:23 +08:00
|
|
|
|
|
|
|
/// Given a number of shuffles of the form shuffle(binop(x,y)), convert them
|
|
|
|
/// to binop(shuffle(x), shuffle(y)) to allow the formation of an
|
|
|
|
/// interleaving load. Any newly created shuffles that operate on \p LI will
|
2021-01-04 23:49:47 +08:00
|
|
|
/// be added to \p Shuffles. Returns true, if any changes to the IR have been
|
|
|
|
/// made.
|
|
|
|
bool replaceBinOpShuffles(ArrayRef<ShuffleVectorInst *> BinOpShuffles,
|
|
|
|
SmallVectorImpl<ShuffleVectorInst *> &Shuffles,
|
|
|
|
LoadInst *LI);
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
};
|
2017-09-23 07:46:57 +08:00
|
|
|
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
} // end anonymous namespace.
|
|
|
|
|
|
|
|
char InterleavedAccess::ID = 0;
|
2017-09-23 07:46:57 +08:00
|
|
|
|
2017-05-26 05:26:32 +08:00
|
|
|
INITIALIZE_PASS_BEGIN(InterleavedAccess, DEBUG_TYPE,
|
2016-05-20 05:39:00 +08:00
|
|
|
"Lower interleaved memory accesses to target specific intrinsics", false,
|
|
|
|
false)
|
|
|
|
INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
|
2017-05-26 05:26:32 +08:00
|
|
|
INITIALIZE_PASS_END(InterleavedAccess, DEBUG_TYPE,
|
2016-05-20 05:39:00 +08:00
|
|
|
"Lower interleaved memory accesses to target specific intrinsics", false,
|
|
|
|
false)
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
2017-05-19 01:21:13 +08:00
|
|
|
FunctionPass *llvm::createInterleavedAccessPass() {
|
|
|
|
return new InterleavedAccess();
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
}
|
|
|
|
|
2018-05-01 23:54:18 +08:00
|
|
|
/// Check if the mask is a DE-interleave mask of the given factor
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
/// \p Factor like:
|
|
|
|
/// <Index, Index+Factor, ..., Index+(NumElts-1)*Factor>
|
|
|
|
static bool isDeInterleaveMaskOfFactor(ArrayRef<int> Mask, unsigned Factor,
|
|
|
|
unsigned &Index) {
|
|
|
|
// Check all potential start indices from 0 to (Factor - 1).
|
|
|
|
for (Index = 0; Index < Factor; Index++) {
|
|
|
|
unsigned i = 0;
|
|
|
|
|
|
|
|
// Check that elements are in ascending order by Factor. Ignore undef
|
|
|
|
// elements.
|
|
|
|
for (; i < Mask.size(); i++)
|
|
|
|
if (Mask[i] >= 0 && static_cast<unsigned>(Mask[i]) != Index + i * Factor)
|
|
|
|
break;
|
|
|
|
|
|
|
|
if (i == Mask.size())
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2018-05-01 23:54:18 +08:00
|
|
|
/// Check if the mask is a DE-interleave mask for an interleaved load.
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
///
|
|
|
|
/// E.g. DE-interleave masks (Factor = 2) could be:
|
|
|
|
/// <0, 2, 4, 6> (mask of index 0 to extract even elements)
|
|
|
|
/// <1, 3, 5, 7> (mask of index 1 to extract odd elements)
|
|
|
|
static bool isDeInterleaveMask(ArrayRef<int> Mask, unsigned &Factor,
|
2019-03-29 04:44:50 +08:00
|
|
|
unsigned &Index, unsigned MaxFactor,
|
|
|
|
unsigned NumLoadElements) {
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
if (Mask.size() < 2)
|
|
|
|
return false;
|
|
|
|
|
|
|
|
// Check potential Factors.
|
2019-03-29 04:44:50 +08:00
|
|
|
for (Factor = 2; Factor <= MaxFactor; Factor++) {
|
|
|
|
// Make sure we don't produce a load wider than the input load.
|
|
|
|
if (Mask.size() * Factor > NumLoadElements)
|
|
|
|
return false;
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
if (isDeInterleaveMaskOfFactor(Mask, Factor, Index))
|
|
|
|
return true;
|
2019-03-29 04:44:50 +08:00
|
|
|
}
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2018-05-01 23:54:18 +08:00
|
|
|
/// Check if the mask can be used in an interleaved store.
|
Generalize strided store pattern in interleave access pass
Summary:
This patch aims to generalize matching of the strided store accesses to more general masks.
The more general rule is to have consecutive accesses based on the stride:
[x, y, ... z, x+1, y+1, ...z+1, x+2, y+2, ...z+2, ...]
All elements in the masks need not form a contiguous space, there may be gaps.
As before, undefs are allowed and filled in with adjacent element loads.
Reviewers: HaoLiu, mssimpso
Subscribers: mkuper, delena, llvm-commits
Differential Revision: https://reviews.llvm.org/D23646
llvm-svn: 289573
2016-12-14 03:32:36 +08:00
|
|
|
//
|
|
|
|
/// It checks for a more general pattern than the RE-interleave mask.
|
|
|
|
/// I.e. <x, y, ... z, x+1, y+1, ...z+1, x+2, y+2, ...z+2, ...>
|
|
|
|
/// E.g. For a Factor of 2 (LaneLen=4): <4, 32, 5, 33, 6, 34, 7, 35>
|
|
|
|
/// E.g. For a Factor of 3 (LaneLen=4): <4, 32, 16, 5, 33, 17, 6, 34, 18, 7, 35, 19>
|
|
|
|
/// E.g. For a Factor of 4 (LaneLen=2): <8, 2, 12, 4, 9, 3, 13, 5>
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
///
|
Generalize strided store pattern in interleave access pass
Summary:
This patch aims to generalize matching of the strided store accesses to more general masks.
The more general rule is to have consecutive accesses based on the stride:
[x, y, ... z, x+1, y+1, ...z+1, x+2, y+2, ...z+2, ...]
All elements in the masks need not form a contiguous space, there may be gaps.
As before, undefs are allowed and filled in with adjacent element loads.
Reviewers: HaoLiu, mssimpso
Subscribers: mkuper, delena, llvm-commits
Differential Revision: https://reviews.llvm.org/D23646
llvm-svn: 289573
2016-12-14 03:32:36 +08:00
|
|
|
/// The particular case of an RE-interleave mask is:
|
|
|
|
/// I.e. <0, LaneLen, ... , LaneLen*(Factor - 1), 1, LaneLen + 1, ...>
|
|
|
|
/// E.g. For a Factor of 2 (LaneLen=4): <0, 4, 1, 5, 2, 6, 3, 7>
|
2016-10-19 02:59:58 +08:00
|
|
|
static bool isReInterleaveMask(ArrayRef<int> Mask, unsigned &Factor,
|
2017-02-01 02:37:53 +08:00
|
|
|
unsigned MaxFactor, unsigned OpNumElts) {
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
unsigned NumElts = Mask.size();
|
|
|
|
if (NumElts < 4)
|
|
|
|
return false;
|
|
|
|
|
|
|
|
// Check potential Factors.
|
|
|
|
for (Factor = 2; Factor <= MaxFactor; Factor++) {
|
|
|
|
if (NumElts % Factor)
|
|
|
|
continue;
|
|
|
|
|
Generalize strided store pattern in interleave access pass
Summary:
This patch aims to generalize matching of the strided store accesses to more general masks.
The more general rule is to have consecutive accesses based on the stride:
[x, y, ... z, x+1, y+1, ...z+1, x+2, y+2, ...z+2, ...]
All elements in the masks need not form a contiguous space, there may be gaps.
As before, undefs are allowed and filled in with adjacent element loads.
Reviewers: HaoLiu, mssimpso
Subscribers: mkuper, delena, llvm-commits
Differential Revision: https://reviews.llvm.org/D23646
llvm-svn: 289573
2016-12-14 03:32:36 +08:00
|
|
|
unsigned LaneLen = NumElts / Factor;
|
|
|
|
if (!isPowerOf2_32(LaneLen))
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
continue;
|
|
|
|
|
Generalize strided store pattern in interleave access pass
Summary:
This patch aims to generalize matching of the strided store accesses to more general masks.
The more general rule is to have consecutive accesses based on the stride:
[x, y, ... z, x+1, y+1, ...z+1, x+2, y+2, ...z+2, ...]
All elements in the masks need not form a contiguous space, there may be gaps.
As before, undefs are allowed and filled in with adjacent element loads.
Reviewers: HaoLiu, mssimpso
Subscribers: mkuper, delena, llvm-commits
Differential Revision: https://reviews.llvm.org/D23646
llvm-svn: 289573
2016-12-14 03:32:36 +08:00
|
|
|
// Check whether each element matches the general interleaved rule.
|
|
|
|
// Ignore undef elements, as long as the defined elements match the rule.
|
|
|
|
// Outer loop processes all factors (x, y, z in the above example)
|
|
|
|
unsigned I = 0, J;
|
|
|
|
for (; I < Factor; I++) {
|
|
|
|
unsigned SavedLaneValue;
|
|
|
|
unsigned SavedNoUndefs = 0;
|
|
|
|
|
|
|
|
// Inner loop processes consecutive accesses (x, x+1... in the example)
|
|
|
|
for (J = 0; J < LaneLen - 1; J++) {
|
|
|
|
// Lane computes x's position in the Mask
|
|
|
|
unsigned Lane = J * Factor + I;
|
|
|
|
unsigned NextLane = Lane + Factor;
|
|
|
|
int LaneValue = Mask[Lane];
|
|
|
|
int NextLaneValue = Mask[NextLane];
|
|
|
|
|
|
|
|
// If both are defined, values must be sequential
|
|
|
|
if (LaneValue >= 0 && NextLaneValue >= 0 &&
|
|
|
|
LaneValue + 1 != NextLaneValue)
|
|
|
|
break;
|
|
|
|
|
|
|
|
// If the next value is undef, save the current one as reference
|
|
|
|
if (LaneValue >= 0 && NextLaneValue < 0) {
|
|
|
|
SavedLaneValue = LaneValue;
|
|
|
|
SavedNoUndefs = 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
// Undefs are allowed, but defined elements must still be consecutive:
|
|
|
|
// i.e.: x,..., undef,..., x + 2,..., undef,..., undef,..., x + 5, ....
|
|
|
|
// Verify this by storing the last non-undef followed by an undef
|
|
|
|
// Check that following non-undef masks are incremented with the
|
|
|
|
// corresponding distance.
|
|
|
|
if (SavedNoUndefs > 0 && LaneValue < 0) {
|
|
|
|
SavedNoUndefs++;
|
|
|
|
if (NextLaneValue >= 0 &&
|
|
|
|
SavedLaneValue + SavedNoUndefs != (unsigned)NextLaneValue)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (J < LaneLen - 1)
|
|
|
|
break;
|
|
|
|
|
|
|
|
int StartMask = 0;
|
|
|
|
if (Mask[I] >= 0) {
|
|
|
|
// Check that the start of the I range (J=0) is greater than 0
|
|
|
|
StartMask = Mask[I];
|
|
|
|
} else if (Mask[(LaneLen - 1) * Factor + I] >= 0) {
|
|
|
|
// StartMask defined by the last value in lane
|
|
|
|
StartMask = Mask[(LaneLen - 1) * Factor + I] - J;
|
|
|
|
} else if (SavedNoUndefs > 0) {
|
|
|
|
// StartMask defined by some non-zero value in the j loop
|
|
|
|
StartMask = SavedLaneValue - (LaneLen - 1 - SavedNoUndefs);
|
|
|
|
}
|
|
|
|
// else StartMask remains set to 0, i.e. all elements are undefs
|
|
|
|
|
|
|
|
if (StartMask < 0)
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
break;
|
2017-02-01 02:37:53 +08:00
|
|
|
// We must stay within the vectors; This case can happen with undefs.
|
|
|
|
if (StartMask + LaneLen > OpNumElts*2)
|
|
|
|
break;
|
Generalize strided store pattern in interleave access pass
Summary:
This patch aims to generalize matching of the strided store accesses to more general masks.
The more general rule is to have consecutive accesses based on the stride:
[x, y, ... z, x+1, y+1, ...z+1, x+2, y+2, ...z+2, ...]
All elements in the masks need not form a contiguous space, there may be gaps.
As before, undefs are allowed and filled in with adjacent element loads.
Reviewers: HaoLiu, mssimpso
Subscribers: mkuper, delena, llvm-commits
Differential Revision: https://reviews.llvm.org/D23646
llvm-svn: 289573
2016-12-14 03:32:36 +08:00
|
|
|
}
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
Generalize strided store pattern in interleave access pass
Summary:
This patch aims to generalize matching of the strided store accesses to more general masks.
The more general rule is to have consecutive accesses based on the stride:
[x, y, ... z, x+1, y+1, ...z+1, x+2, y+2, ...z+2, ...]
All elements in the masks need not form a contiguous space, there may be gaps.
As before, undefs are allowed and filled in with adjacent element loads.
Reviewers: HaoLiu, mssimpso
Subscribers: mkuper, delena, llvm-commits
Differential Revision: https://reviews.llvm.org/D23646
llvm-svn: 289573
2016-12-14 03:32:36 +08:00
|
|
|
// Found an interleaved mask of current factor.
|
|
|
|
if (I == Factor)
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
bool InterleavedAccess::lowerInterleavedLoad(
|
|
|
|
LoadInst *LI, SmallVector<Instruction *, 32> &DeadInsts) {
|
2020-07-10 02:51:03 +08:00
|
|
|
if (!LI->isSimple() || isa<ScalableVectorType>(LI->getType()))
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
return false;
|
|
|
|
|
2020-10-29 17:13:23 +08:00
|
|
|
// Check if all users of this load are shufflevectors. If we encounter any
|
|
|
|
// users that are extractelement instructions or binary operators, we save
|
|
|
|
// them to later check if they can be modified to extract from one of the
|
|
|
|
// shufflevectors instead of the load.
|
|
|
|
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
SmallVector<ShuffleVectorInst *, 4> Shuffles;
|
2016-05-20 05:39:00 +08:00
|
|
|
SmallVector<ExtractElementInst *, 4> Extracts;
|
2020-10-29 17:13:23 +08:00
|
|
|
// BinOpShuffles need to be handled a single time in case both operands of the
|
|
|
|
// binop are the same load.
|
|
|
|
SmallSetVector<ShuffleVectorInst *, 4> BinOpShuffles;
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
2020-10-29 17:13:23 +08:00
|
|
|
for (auto *User : LI->users()) {
|
|
|
|
auto *Extract = dyn_cast<ExtractElementInst>(User);
|
2016-05-20 05:39:00 +08:00
|
|
|
if (Extract && isa<ConstantInt>(Extract->getIndexOperand())) {
|
|
|
|
Extracts.push_back(Extract);
|
|
|
|
continue;
|
|
|
|
}
|
2020-10-29 17:13:23 +08:00
|
|
|
auto *BI = dyn_cast<BinaryOperator>(User);
|
|
|
|
if (BI && BI->hasOneUse()) {
|
|
|
|
if (auto *SVI = dyn_cast<ShuffleVectorInst>(*BI->user_begin())) {
|
|
|
|
BinOpShuffles.insert(SVI);
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
auto *SVI = dyn_cast<ShuffleVectorInst>(User);
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
if (!SVI || !isa<UndefValue>(SVI->getOperand(1)))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
Shuffles.push_back(SVI);
|
|
|
|
}
|
|
|
|
|
2020-10-29 17:13:23 +08:00
|
|
|
if (Shuffles.empty() && BinOpShuffles.empty())
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
return false;
|
|
|
|
|
|
|
|
unsigned Factor, Index;
|
|
|
|
|
2020-07-10 02:51:03 +08:00
|
|
|
unsigned NumLoadElements =
|
|
|
|
cast<FixedVectorType>(LI->getType())->getNumElements();
|
2020-10-29 17:13:23 +08:00
|
|
|
auto *FirstSVI = Shuffles.size() > 0 ? Shuffles[0] : BinOpShuffles[0];
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
// Check if the first shufflevector is DE-interleave shuffle.
|
2020-10-29 17:13:23 +08:00
|
|
|
if (!isDeInterleaveMask(FirstSVI->getShuffleMask(), Factor, Index, MaxFactor,
|
|
|
|
NumLoadElements))
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
return false;
|
|
|
|
|
|
|
|
// Holds the corresponding index for each DE-interleave shuffle.
|
|
|
|
SmallVector<unsigned, 4> Indices;
|
|
|
|
|
2020-10-29 17:13:23 +08:00
|
|
|
Type *VecTy = FirstSVI->getType();
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
|
|
|
// Check if other shufflevectors are also DE-interleaved of the same type
|
|
|
|
// and factor as the first shufflevector.
|
2020-10-29 17:13:23 +08:00
|
|
|
for (auto *Shuffle : Shuffles) {
|
|
|
|
if (Shuffle->getType() != VecTy)
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
return false;
|
2020-10-29 17:13:23 +08:00
|
|
|
if (!isDeInterleaveMaskOfFactor(Shuffle->getShuffleMask(), Factor,
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
Index))
|
|
|
|
return false;
|
|
|
|
|
2021-01-10 16:22:54 +08:00
|
|
|
assert(Shuffle->getShuffleMask().size() <= NumLoadElements);
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
Indices.push_back(Index);
|
|
|
|
}
|
2020-10-29 17:13:23 +08:00
|
|
|
for (auto *Shuffle : BinOpShuffles) {
|
|
|
|
if (Shuffle->getType() != VecTy)
|
|
|
|
return false;
|
|
|
|
if (!isDeInterleaveMaskOfFactor(Shuffle->getShuffleMask(), Factor,
|
|
|
|
Index))
|
|
|
|
return false;
|
|
|
|
|
2021-01-10 16:22:54 +08:00
|
|
|
assert(Shuffle->getShuffleMask().size() <= NumLoadElements);
|
|
|
|
|
2020-10-29 17:13:23 +08:00
|
|
|
if (cast<Instruction>(Shuffle->getOperand(0))->getOperand(0) == LI)
|
|
|
|
Indices.push_back(Index);
|
|
|
|
if (cast<Instruction>(Shuffle->getOperand(0))->getOperand(1) == LI)
|
|
|
|
Indices.push_back(Index);
|
|
|
|
}
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
2016-05-20 05:39:00 +08:00
|
|
|
// Try and modify users of the load that are extractelement instructions to
|
|
|
|
// use the shufflevector instructions instead of the load.
|
|
|
|
if (!tryReplaceExtracts(Extracts, Shuffles))
|
|
|
|
return false;
|
2021-01-04 23:49:47 +08:00
|
|
|
|
|
|
|
bool BinOpShuffleChanged =
|
|
|
|
replaceBinOpShuffles(BinOpShuffles.getArrayRef(), Shuffles, LI);
|
2016-05-20 05:39:00 +08:00
|
|
|
|
2018-05-14 20:53:11 +08:00
|
|
|
LLVM_DEBUG(dbgs() << "IA: Found an interleaved load: " << *LI << "\n");
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
|
|
|
// Try to create target specific intrinsics to replace the load and shuffles.
|
2021-01-04 23:49:47 +08:00
|
|
|
if (!TLI->lowerInterleavedLoad(LI, Shuffles, Indices, Factor)) {
|
|
|
|
// If Extracts is not empty, tryReplaceExtracts made changes earlier.
|
|
|
|
return !Extracts.empty() || BinOpShuffleChanged;
|
|
|
|
}
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
2021-01-28 15:25:41 +08:00
|
|
|
append_range(DeadInsts, Shuffles);
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
|
|
|
DeadInsts.push_back(LI);
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2021-01-04 23:49:47 +08:00
|
|
|
bool InterleavedAccess::replaceBinOpShuffles(
|
2020-10-29 17:13:23 +08:00
|
|
|
ArrayRef<ShuffleVectorInst *> BinOpShuffles,
|
|
|
|
SmallVectorImpl<ShuffleVectorInst *> &Shuffles, LoadInst *LI) {
|
|
|
|
for (auto *SVI : BinOpShuffles) {
|
|
|
|
BinaryOperator *BI = cast<BinaryOperator>(SVI->getOperand(0));
|
2021-01-10 16:22:54 +08:00
|
|
|
Type *BIOp0Ty = BI->getOperand(0)->getType();
|
2020-10-29 17:13:23 +08:00
|
|
|
ArrayRef<int> Mask = SVI->getShuffleMask();
|
2021-01-10 16:22:54 +08:00
|
|
|
assert(all_of(Mask, [&](int Idx) {
|
|
|
|
return Idx < (int)cast<FixedVectorType>(BIOp0Ty)->getNumElements();
|
|
|
|
}));
|
2020-10-29 17:13:23 +08:00
|
|
|
|
2021-01-10 16:22:54 +08:00
|
|
|
auto *NewSVI1 =
|
|
|
|
new ShuffleVectorInst(BI->getOperand(0), PoisonValue::get(BIOp0Ty),
|
|
|
|
Mask, SVI->getName(), SVI);
|
2020-10-29 17:13:23 +08:00
|
|
|
auto *NewSVI2 = new ShuffleVectorInst(
|
2021-01-10 16:22:54 +08:00
|
|
|
BI->getOperand(1), PoisonValue::get(BI->getOperand(1)->getType()), Mask,
|
2020-10-29 17:13:23 +08:00
|
|
|
SVI->getName(), SVI);
|
2021-06-17 16:53:33 +08:00
|
|
|
BinaryOperator *NewBI = BinaryOperator::CreateWithCopiedFlags(
|
|
|
|
BI->getOpcode(), NewSVI1, NewSVI2, BI, BI->getName(), SVI);
|
2020-10-29 17:13:23 +08:00
|
|
|
SVI->replaceAllUsesWith(NewBI);
|
|
|
|
LLVM_DEBUG(dbgs() << " Replaced: " << *BI << "\n And : " << *SVI
|
|
|
|
<< "\n With : " << *NewSVI1 << "\n And : "
|
|
|
|
<< *NewSVI2 << "\n And : " << *NewBI << "\n");
|
|
|
|
RecursivelyDeleteTriviallyDeadInstructions(SVI);
|
|
|
|
if (NewSVI1->getOperand(0) == LI)
|
|
|
|
Shuffles.push_back(NewSVI1);
|
|
|
|
if (NewSVI2->getOperand(0) == LI)
|
|
|
|
Shuffles.push_back(NewSVI2);
|
|
|
|
}
|
2021-01-04 23:49:47 +08:00
|
|
|
|
|
|
|
return !BinOpShuffles.empty();
|
2020-10-29 17:13:23 +08:00
|
|
|
}
|
|
|
|
|
2016-05-20 05:39:00 +08:00
|
|
|
bool InterleavedAccess::tryReplaceExtracts(
|
|
|
|
ArrayRef<ExtractElementInst *> Extracts,
|
|
|
|
ArrayRef<ShuffleVectorInst *> Shuffles) {
|
|
|
|
// If there aren't any extractelement instructions to modify, there's nothing
|
|
|
|
// to do.
|
|
|
|
if (Extracts.empty())
|
|
|
|
return true;
|
|
|
|
|
|
|
|
// Maps extractelement instructions to vector-index pairs. The extractlement
|
|
|
|
// instructions will be modified to use the new vector and index operands.
|
|
|
|
DenseMap<ExtractElementInst *, std::pair<Value *, int>> ReplacementMap;
|
|
|
|
|
|
|
|
for (auto *Extract : Extracts) {
|
|
|
|
// The vector index that is extracted.
|
|
|
|
auto *IndexOperand = cast<ConstantInt>(Extract->getIndexOperand());
|
|
|
|
auto Index = IndexOperand->getSExtValue();
|
|
|
|
|
|
|
|
// Look for a suitable shufflevector instruction. The goal is to modify the
|
|
|
|
// extractelement instruction (which uses an interleaved load) to use one
|
|
|
|
// of the shufflevector instructions instead of the load.
|
|
|
|
for (auto *Shuffle : Shuffles) {
|
|
|
|
// If the shufflevector instruction doesn't dominate the extract, we
|
|
|
|
// can't create a use of it.
|
|
|
|
if (!DT->dominates(Shuffle, Extract))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
// Inspect the indices of the shufflevector instruction. If the shuffle
|
|
|
|
// selects the same index that is extracted, we can modify the
|
|
|
|
// extractelement instruction.
|
|
|
|
SmallVector<int, 4> Indices;
|
|
|
|
Shuffle->getShuffleMask(Indices);
|
|
|
|
for (unsigned I = 0; I < Indices.size(); ++I)
|
|
|
|
if (Indices[I] == Index) {
|
|
|
|
assert(Extract->getOperand(0) == Shuffle->getOperand(0) &&
|
|
|
|
"Vector operations do not match");
|
|
|
|
ReplacementMap[Extract] = std::make_pair(Shuffle, I);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
// If we found a suitable shufflevector instruction, stop looking.
|
|
|
|
if (ReplacementMap.count(Extract))
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
// If we did not find a suitable shufflevector instruction, the
|
|
|
|
// extractelement instruction cannot be modified, so we must give up.
|
|
|
|
if (!ReplacementMap.count(Extract))
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
// Finally, perform the replacements.
|
|
|
|
IRBuilder<> Builder(Extracts[0]->getContext());
|
|
|
|
for (auto &Replacement : ReplacementMap) {
|
|
|
|
auto *Extract = Replacement.first;
|
|
|
|
auto *Vector = Replacement.second.first;
|
|
|
|
auto Index = Replacement.second.second;
|
|
|
|
Builder.SetInsertPoint(Extract);
|
|
|
|
Extract->replaceAllUsesWith(Builder.CreateExtractElement(Vector, Index));
|
|
|
|
Extract->eraseFromParent();
|
|
|
|
}
|
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
bool InterleavedAccess::lowerInterleavedStore(
|
|
|
|
StoreInst *SI, SmallVector<Instruction *, 32> &DeadInsts) {
|
|
|
|
if (!SI->isSimple())
|
|
|
|
return false;
|
|
|
|
|
2020-10-29 17:13:23 +08:00
|
|
|
auto *SVI = dyn_cast<ShuffleVectorInst>(SI->getValueOperand());
|
2020-07-10 02:51:03 +08:00
|
|
|
if (!SVI || !SVI->hasOneUse() || isa<ScalableVectorType>(SVI->getType()))
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
return false;
|
|
|
|
|
|
|
|
// Check if the shufflevector is RE-interleave shuffle.
|
|
|
|
unsigned Factor;
|
2020-04-11 05:23:20 +08:00
|
|
|
unsigned OpNumElts =
|
2020-07-10 02:51:03 +08:00
|
|
|
cast<FixedVectorType>(SVI->getOperand(0)->getType())->getNumElements();
|
2017-02-01 02:37:53 +08:00
|
|
|
if (!isReInterleaveMask(SVI->getShuffleMask(), Factor, MaxFactor, OpNumElts))
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
return false;
|
|
|
|
|
2018-05-14 20:53:11 +08:00
|
|
|
LLVM_DEBUG(dbgs() << "IA: Found an interleaved store: " << *SI << "\n");
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
|
|
|
// Try to create target specific intrinsics to replace the store and shuffle.
|
|
|
|
if (!TLI->lowerInterleavedStore(SI, SVI, Factor))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
// Already have a new target specific interleaved store. Erase the old store.
|
|
|
|
DeadInsts.push_back(SI);
|
|
|
|
DeadInsts.push_back(SVI);
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
bool InterleavedAccess::runOnFunction(Function &F) {
|
2017-05-19 01:21:13 +08:00
|
|
|
auto *TPC = getAnalysisIfAvailable<TargetPassConfig>();
|
|
|
|
if (!TPC || !LowerInterleavedAccesses)
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
return false;
|
|
|
|
|
2018-05-14 20:53:11 +08:00
|
|
|
LLVM_DEBUG(dbgs() << "*** " << getPassName() << ": " << F.getName() << "\n");
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
|
2016-05-20 05:39:00 +08:00
|
|
|
DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
|
2017-05-19 01:21:13 +08:00
|
|
|
auto &TM = TPC->getTM<TargetMachine>();
|
|
|
|
TLI = TM.getSubtargetImpl(F)->getTargetLowering();
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
MaxFactor = TLI->getMaxSupportedInterleaveFactor();
|
|
|
|
|
|
|
|
// Holds dead instructions that will be erased later.
|
|
|
|
SmallVector<Instruction *, 32> DeadInsts;
|
|
|
|
bool Changed = false;
|
|
|
|
|
2015-08-07 03:10:45 +08:00
|
|
|
for (auto &I : instructions(F)) {
|
2020-10-29 17:13:23 +08:00
|
|
|
if (auto *LI = dyn_cast<LoadInst>(&I))
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
Changed |= lowerInterleavedLoad(LI, DeadInsts);
|
|
|
|
|
2020-10-29 17:13:23 +08:00
|
|
|
if (auto *SI = dyn_cast<StoreInst>(&I))
|
[InterleavedAccess] Add a pass InterleavedAccess to identify interleaved memory accesses and transform into target specific intrinsics.
E.g. An interleaved load (Factor = 2):
%wide.vec = load <8 x i32>, <8 x i32>* %ptr
%v0 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <0, 2, 4, 6>
%v1 = shuffle <8 x i32> %wide.vec, <8 x i32> undef, <1, 3, 5, 7>
It can be transformed into a ld2 intrinsic in AArch64 backend or a vld2 intrinsic in ARM backend.
E.g. An interleaved store (Factor = 3):
%i.vec = shuffle <8 x i32> %v0, <8 x i32> %v1, <0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11>
store <12 x i32> %i.vec, <12 x i32>* %ptr
It can be transformed into a st3 intrinsic in AArch64 backend or a vst3 intrinsic in ARM backend.
Differential Revision: http://reviews.llvm.org/D10533
llvm-svn: 240751
2015-06-26 10:10:27 +08:00
|
|
|
Changed |= lowerInterleavedStore(SI, DeadInsts);
|
|
|
|
}
|
|
|
|
|
|
|
|
for (auto I : DeadInsts)
|
|
|
|
I->eraseFromParent();
|
|
|
|
|
|
|
|
return Changed;
|
|
|
|
}
|