maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/llvm/lib/Transforms/Vectorize/BBVectorize.cpp

3064 lines
123 KiB
C++
Raw Normal View History

Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
//===- BBVectorize.cpp - A Basic-Block Vectorizer -------------------------===//
//
// The LLVM Compiler Infrastructure
//
// This file is distributed under the University of Illinois Open Source
// License. See LICENSE.TXT for details.
//
//===----------------------------------------------------------------------===//
//
// This file implements a basic-block vectorization pass. The algorithm was
// inspired by that used by the Vienna MAP Vectorizor by Franchetti and Kral,
// et al. It works by looking for chains of pairable operations and then
// pairing them.
//
//===----------------------------------------------------------------------===//
#define BBV_NAME "bb-vectorize"
#define DEBUG_TYPE BBV_NAME
Use the new script to sort the includes of every file under lib. Sooooo many of these had incorrect or strange main module includes. I have manually inspected all of these, and fixed the main module include to be the nearest plausible thing I could find. If you own or care about any of these source files, I encourage you to take some time and check that these edits were sensible. I can't have broken anything (I strictly added headers, and reordered them, never removed), but they may not be the headers you'd really like to identify as containing the API being implemented. Many forward declarations and missing includes were added to a header files to allow them to parse cleanly when included first. The main module rule does in fact have its merits. =] llvm-svn: 169131
2012-12-04 00:50:05 +08:00
#include "llvm/Transforms/Vectorize.h"
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#include "llvm/ADT/DenseMap.h"
#include "llvm/ADT/DenseSet.h"
Use the new script to sort the includes of every file under lib. Sooooo many of these had incorrect or strange main module includes. I have manually inspected all of these, and fixed the main module include to be the nearest plausible thing I could find. If you own or care about any of these source files, I encourage you to take some time and check that these edits were sensible. I can't have broken anything (I strictly added headers, and reordered them, never removed), but they may not be the headers you'd really like to identify as containing the API being implemented. Many forward declarations and missing includes were added to a header files to allow them to parse cleanly when included first. The main module rule does in fact have its merits. =] llvm-svn: 169131
2012-12-04 00:50:05 +08:00
#include "llvm/ADT/STLExtras.h"
BBVectorize: Use a more sophisticated check for input cost The old checking code, which assumed that input shuffles and insert-elements could always be folded (and thus were free) is too simple. This can only happen in special circumstances. Using the simple check caused infinite recursion. llvm-svn: 167750
2012-11-13 05:21:02 +08:00
#include "llvm/ADT/SmallSet.h"
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/Statistic.h"
#include "llvm/ADT/StringExtras.h"
#include "llvm/Analysis/AliasAnalysis.h"
#include "llvm/Analysis/AliasSetTracker.h"
BBVectorize should ignore unreachable blocks. Unreachable blocks can have invalid instructions. For example, jump threading can produce self-referential instructions in unreachable blocks. Also, we should not be spending time optimizing unreachable code. Fixes PR14133. llvm-svn: 166423
2012-10-23 02:00:55 +08:00
#include "llvm/Analysis/Dominators.h"
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#include "llvm/Analysis/ScalarEvolution.h"
#include "llvm/Analysis/ScalarEvolutionExpressions.h"
Move TargetTransformInfo to live under the Analysis library. This no longer would violate any dependency layering and it is in fact an analysis. =] llvm-svn: 171686
2013-01-07 11:08:10 +08:00
#include "llvm/Analysis/TargetTransformInfo.h"
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#include "llvm/Analysis/ValueTracking.h"
Move all of the header files which are involved in modelling the LLVM IR into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. llvm-svn: 171366
2013-01-02 19:36:10 +08:00
#include "llvm/IR/Constants.h"
#include "llvm/IR/DataLayout.h"
#include "llvm/IR/DerivedTypes.h"
#include "llvm/IR/Function.h"
#include "llvm/IR/Instructions.h"
#include "llvm/IR/IntrinsicInst.h"
#include "llvm/IR/Intrinsics.h"
#include "llvm/IR/LLVMContext.h"
#include "llvm/IR/Metadata.h"
#include "llvm/IR/Type.h"
Use the new script to sort the includes of every file under lib. Sooooo many of these had incorrect or strange main module includes. I have manually inspected all of these, and fixed the main module include to be the nearest plausible thing I could find. If you own or care about any of these source files, I encourage you to take some time and check that these edits were sensible. I can't have broken anything (I strictly added headers, and reordered them, never removed), but they may not be the headers you'd really like to identify as containing the API being implemented. Many forward declarations and missing includes were added to a header files to allow them to parse cleanly when included first. The main module rule does in fact have its merits. =] llvm-svn: 169131
2012-12-04 00:50:05 +08:00
#include "llvm/Pass.h"
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Debug.h"
#include "llvm/Support/ValueHandle.h"
Use the new script to sort the includes of every file under lib. Sooooo many of these had incorrect or strange main module includes. I have manually inspected all of these, and fixed the main module include to be the nearest plausible thing I could find. If you own or care about any of these source files, I encourage you to take some time and check that these edits were sensible. I can't have broken anything (I strictly added headers, and reordered them, never removed), but they may not be the headers you'd really like to identify as containing the API being implemented. Many forward declarations and missing includes were added to a header files to allow them to parse cleanly when included first. The main module rule does in fact have its merits. =] llvm-svn: 169131
2012-12-04 00:50:05 +08:00
#include "llvm/Support/raw_ostream.h"
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
#include "llvm/Transforms/Utils/Local.h"
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#include <algorithm>
#include <map>
using namespace llvm;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
static cl::opt<bool>
IgnoreTargetInfo("bb-vectorize-ignore-target-info", cl::init(false),
cl::Hidden, cl::desc("Ignore target information"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<unsigned>
ReqChainDepth("bb-vectorize-req-chain-depth", cl::init(6), cl::Hidden,
cl::desc("The required chain depth for vectorization"));
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
static cl::opt<bool>
UseChainDepthWithTI("bb-vectorize-use-chain-depth", cl::init(false),
cl::Hidden, cl::desc("Use the chain depth requirement with"
" target information"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<unsigned>
SearchLimit("bb-vectorize-search-limit", cl::init(400), cl::Hidden,
cl::desc("The maximum search distance for instruction pairs"));
static cl::opt<bool>
SplatBreaksChain("bb-vectorize-splat-breaks-chain", cl::init(false), cl::Hidden,
cl::desc("Replicating one element to a pair breaks the chain"));
static cl::opt<unsigned>
VectorBits("bb-vectorize-vector-bits", cl::init(128), cl::Hidden,
cl::desc("The size of the native vector registers"));
static cl::opt<unsigned>
MaxIter("bb-vectorize-max-iter", cl::init(0), cl::Hidden,
cl::desc("The maximum number of pairing iterations"));
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
static cl::opt<bool>
Pow2LenOnly("bb-vectorize-pow2-len-only", cl::init(false), cl::Hidden,
cl::desc("Don't try to form non-2^n-length vectors"));
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
static cl::opt<unsigned>
MaxInsts("bb-vectorize-max-instr-per-group", cl::init(500), cl::Hidden,
cl::desc("The maximum number of pairable instructions per group"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<unsigned>
MaxCandPairsForCycleCheck("bb-vectorize-max-cycle-check-pairs", cl::init(200),
cl::Hidden, cl::desc("The maximum number of candidate pairs with which to use"
" a full cycle check"));
Allow controlling vectorization of boolean values separately from other integer types. These are used as the result of comparisons, and often handled differently from larger integer types. llvm-svn: 159111
2012-06-24 21:28:01 +08:00
static cl::opt<bool>
NoBools("bb-vectorize-no-bools", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize boolean (i1) values"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<bool>
NoInts("bb-vectorize-no-ints", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize integer values"));
static cl::opt<bool>
NoFloats("bb-vectorize-no-floats", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize floating-point values"));
Disable generation of pointer vectors by BBVectorize. Once vector-of-pointer support works, then this can be reverted. llvm-svn: 166741
2012-10-26 08:05:26 +08:00
// FIXME: This should default to false once pointer vector support works.
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
static cl::opt<bool>
Disable generation of pointer vectors by BBVectorize. Once vector-of-pointer support works, then this can be reverted. llvm-svn: 166741
2012-10-26 08:05:26 +08:00
NoPointers("bb-vectorize-no-pointers", cl::init(/*false*/ true), cl::Hidden,
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
cl::desc("Don't try to vectorize pointer values"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<bool>
NoCasts("bb-vectorize-no-casts", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize casting (conversion) operations"));
static cl::opt<bool>
NoMath("bb-vectorize-no-math", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize floating-point math intrinsics"));
static cl::opt<bool>
NoFMA("bb-vectorize-no-fma", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize the fused-multiply-add intrinsic"));
Add support to BBVectorize for vectorizing selects. llvm-svn: 154700
2012-04-14 04:45:45 +08:00
static cl::opt<bool>
NoSelect("bb-vectorize-no-select", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize select instructions"));
Allow BBVectorize to fuse compare instructions. llvm-svn: 159088
2012-06-24 05:52:50 +08:00
static cl::opt<bool>
NoCmp("bb-vectorize-no-cmp", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize comparison instructions"));
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
static cl::opt<bool>
NoGEP("bb-vectorize-no-gep", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize getelementptr instructions"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<bool>
NoMemOps("bb-vectorize-no-mem-ops", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize loads and stores"));
static cl::opt<bool>
AlignedOnly("bb-vectorize-aligned-only", cl::init(false), cl::Hidden,
cl::desc("Only generate aligned loads and stores"));
Boost the effective chain depth of loads and stores. By default, boost the chain depth contribution of loads and stores. This will allow a load/store pair to vectorize even when it would not otherwise be long enough to satisfy the chain depth requirement. llvm-svn: 149761
2012-02-04 12:14:04 +08:00
static cl::opt<bool>
NoMemOpBoost("bb-vectorize-no-mem-op-boost",
cl::init(false), cl::Hidden,
cl::desc("Don't boost the chain-depth contribution of loads and stores"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<bool>
FastDep("bb-vectorize-fast-dep", cl::init(false), cl::Hidden,
cl::desc("Use a fast instruction dependency analysis"));
#ifndef NDEBUG
static cl::opt<bool>
DebugInstructionExamination("bb-vectorize-debug-instruction-examination",
cl::init(false), cl::Hidden,
cl::desc("When debugging is enabled, output information on the"
" instruction-examination process"));
static cl::opt<bool>
DebugCandidateSelection("bb-vectorize-debug-candidate-selection",
cl::init(false), cl::Hidden,
cl::desc("When debugging is enabled, output information on the"
" candidate-selection process"));
static cl::opt<bool>
DebugPairSelection("bb-vectorize-debug-pair-selection",
cl::init(false), cl::Hidden,
cl::desc("When debugging is enabled, output information on the"
" pair-selection process"));
static cl::opt<bool>
DebugCycleCheck("bb-vectorize-debug-cycle-check",
cl::init(false), cl::Hidden,
cl::desc("When debugging is enabled, output information on the"
" cycle-checking process"));
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
static cl::opt<bool>
PrintAfterEveryPair("bb-vectorize-debug-print-after-every-pair",
cl::init(false), cl::Hidden,
cl::desc("When debugging is enabled, dump the basic block after"
" every pair is fused"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#endif
STATISTIC(NumFusedOps, "Number of operations fused by bb-vectorize");
namespace {
struct BBVectorize : public BasicBlockPass {
static char ID; // Pass identification, replacement for typeid
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
BBVectorize: Add the const modifier to the VectorizeConfig because we won't modify it. llvm-svn: 154098
2012-04-06 00:07:49 +08:00
const VectorizeConfig Config;
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
BBVectorize(const VectorizeConfig &C = VectorizeConfig())
: BasicBlockPass(ID), Config(C) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
initializeBBVectorizePass(*PassRegistry::getPassRegistry());
}
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
BBVectorize(Pass *P, const VectorizeConfig &C)
: BasicBlockPass(ID), Config(C) {
Add the function "vectorizeBasicBlock" which allow users vectorize a BasicBlock in other passes, e.g. we can call vectorizeBasicBlock in the loop unroll pass right after the loop is unrolled. llvm-svn: 154089
2012-04-05 16:05:16 +08:00
AA = &P->getAnalysis<AliasAnalysis>();
BBVectorize should ignore unreachable blocks. Unreachable blocks can have invalid instructions. For example, jump threading can produce self-referential instructions in unreachable blocks. Also, we should not be spending time optimizing unreachable code. Fixes PR14133. llvm-svn: 166423
2012-10-23 02:00:55 +08:00
DT = &P->getAnalysis<DominatorTree>();
Add the function "vectorizeBasicBlock" which allow users vectorize a BasicBlock in other passes, e.g. we can call vectorizeBasicBlock in the loop unroll pass right after the loop is unrolled. llvm-svn: 154089
2012-04-05 16:05:16 +08:00
SE = &P->getAnalysis<ScalarEvolution>();
Move TargetData to DataLayout. llvm-svn: 165402
2012-10-09 00:38:25 +08:00
TD = P->getAnalysisIfAvailable<DataLayout>();
Switch BBVectorize to directly depend on having a TTI analysis. This could be simplified further, but Hal has a specific feature for ignoring TTI, and so I preserved that. Also, I needed to use it because a number of tests fail when switching from a null TTI to the NoTTI nonce implementation. That seems suspicious to me and so may be something that you need to look into Hal. I worked it by preserving the old behavior for these tests with the flag that ignores all target info. llvm-svn: 171722
2013-01-07 18:22:36 +08:00
TTI = IgnoreTargetInfo ? 0 : &P->getAnalysis<TargetTransformInfo>();
Add the function "vectorizeBasicBlock" which allow users vectorize a BasicBlock in other passes, e.g. we can call vectorizeBasicBlock in the loop unroll pass right after the loop is unrolled. llvm-svn: 154089
2012-04-05 16:05:16 +08:00
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
typedef std::pair<Value *, Value *> ValuePair;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
typedef std::pair<ValuePair, int> ValuePairWithCost;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
typedef std::pair<ValuePair, size_t> ValuePairWithDepth;
typedef std::pair<ValuePair, ValuePair> VPPair; // A ValuePair pair
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
typedef std::pair<VPPair, unsigned> VPPairWithType;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
typedef std::pair<std::multimap<Value *, Value *>::iterator,
std::multimap<Value *, Value *>::iterator> VPIteratorPair;
typedef std::pair<std::multimap<ValuePair, ValuePair>::iterator,
std::multimap<ValuePair, ValuePair>::iterator>
VPPIteratorPair;
AliasAnalysis *AA;
BBVectorize should ignore unreachable blocks. Unreachable blocks can have invalid instructions. For example, jump threading can produce self-referential instructions in unreachable blocks. Also, we should not be spending time optimizing unreachable code. Fixes PR14133. llvm-svn: 166423
2012-10-23 02:00:55 +08:00
DominatorTree *DT;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
ScalarEvolution *SE;
Move TargetData to DataLayout. llvm-svn: 165402
2012-10-09 00:38:25 +08:00
DataLayout *TD;
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
const TargetTransformInfo *TTI;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// FIXME: const correct?
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
bool vectorizePairs(BasicBlock &BB, bool NonPow2Len = false);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
bool getCandidatePairs(BasicBlock &BB,
BasicBlock::iterator &Start,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
std::multimap<Value *, Value *> &CandidatePairs,
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
DenseSet<ValuePair> &FixedOrderPairs,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
DenseMap<ValuePair, int> &CandidatePairCostSavings,
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
std::vector<Value *> &PairableInsts, bool NonPow2Len);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
// FIXME: The current implementation does not account for pairs that
// are connected in multiple ways. For example:
// C1 = A1 / A2; C2 = A2 / A1 (which may be both direct and a swap)
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
enum PairConnectionType {
PairConnectionDirect,
PairConnectionSwap,
PairConnectionSplat
};
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
void computeConnectedPairs(std::multimap<Value *, Value *> &CandidatePairs,
std::vector<Value *> &PairableInsts,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
std::multimap<ValuePair, ValuePair> &ConnectedPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
void buildDepMap(BasicBlock &BB,
std::multimap<Value *, Value *> &CandidatePairs,
std::vector<Value *> &PairableInsts,
DenseSet<ValuePair> &PairableInstUsers);
void choosePairs(std::multimap<Value *, Value *> &CandidatePairs,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
DenseMap<ValuePair, int> &CandidatePairCostSavings,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
std::vector<Value *> &PairableInsts,
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
DenseSet<ValuePair> &FixedOrderPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
std::multimap<ValuePair, ValuePair> &ConnectedPairs,
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
std::multimap<ValuePair, ValuePair> &ConnectedPairDeps,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DenseSet<ValuePair> &PairableInstUsers,
DenseMap<Value *, Value *>& ChosenPairs);
void fuseChosenPairs(BasicBlock &BB,
std::vector<Value *> &PairableInsts,
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
DenseMap<Value *, Value *>& ChosenPairs,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
DenseSet<ValuePair> &FixedOrderPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes,
std::multimap<ValuePair, ValuePair> &ConnectedPairs,
std::multimap<ValuePair, ValuePair> &ConnectedPairDeps);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool isInstVectorizable(Instruction *I, bool &IsSimpleLoadStore);
bool areInstsCompatible(Instruction *I, Instruction *J,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
bool IsSimpleLoadStore, bool NonPow2Len,
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
int &CostSavings, int &FixedOrder);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool trackUsesOfI(DenseSet<Value *> &Users,
AliasSetTracker &WriteSet, Instruction *I,
Instruction *J, bool UpdateUsers = true,
std::multimap<Value *, Value *> *LoadMoveSet = 0);
fix indentation llvm-svn: 149857
2012-02-06 13:29:32 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
void computePairsConnectedTo(
std::multimap<Value *, Value *> &CandidatePairs,
std::vector<Value *> &PairableInsts,
std::multimap<ValuePair, ValuePair> &ConnectedPairs,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
DenseMap<VPPair, unsigned> &PairConnectionTypes,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
ValuePair P);
bool pairsConflict(ValuePair P, ValuePair Q,
DenseSet<ValuePair> &PairableInstUsers,
std::multimap<ValuePair, ValuePair> *PairableInstUserMap = 0);
bool pairWillFormCycle(ValuePair P,
std::multimap<ValuePair, ValuePair> &PairableInstUsers,
DenseSet<ValuePair> &CurrentPairs);
void pruneTreeFor(
std::multimap<Value *, Value *> &CandidatePairs,
std::vector<Value *> &PairableInsts,
std::multimap<ValuePair, ValuePair> &ConnectedPairs,
DenseSet<ValuePair> &PairableInstUsers,
std::multimap<ValuePair, ValuePair> &PairableInstUserMap,
DenseMap<Value *, Value *> &ChosenPairs,
DenseMap<ValuePair, size_t> &Tree,
DenseSet<ValuePair> &PrunedTree, ValuePair J,
bool UseCycleCheck);
void buildInitialTreeFor(
std::multimap<Value *, Value *> &CandidatePairs,
std::vector<Value *> &PairableInsts,
std::multimap<ValuePair, ValuePair> &ConnectedPairs,
DenseSet<ValuePair> &PairableInstUsers,
DenseMap<Value *, Value *> &ChosenPairs,
DenseMap<ValuePair, size_t> &Tree, ValuePair J);
void findBestTreeFor(
std::multimap<Value *, Value *> &CandidatePairs,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
DenseMap<ValuePair, int> &CandidatePairCostSavings,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
std::vector<Value *> &PairableInsts,
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
DenseSet<ValuePair> &FixedOrderPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
std::multimap<ValuePair, ValuePair> &ConnectedPairs,
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
std::multimap<ValuePair, ValuePair> &ConnectedPairDeps,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DenseSet<ValuePair> &PairableInstUsers,
std::multimap<ValuePair, ValuePair> &PairableInstUserMap,
DenseMap<Value *, Value *> &ChosenPairs,
DenseSet<ValuePair> &BestTree, size_t &BestMaxDepth,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
int &BestEffSize, VPIteratorPair ChoiceRange,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool UseCycleCheck);
Value *getReplacementPointerInput(LLVMContext& Context, Instruction *I,
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
Instruction *J, unsigned o);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
void fillNewShuffleMask(LLVMContext& Context, Instruction *J,
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
unsigned MaskOffset, unsigned NumInElem,
unsigned NumInElem1, unsigned IdxOffset,
std::vector<Constant*> &Mask);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Value *getReplacementShuffleMask(LLVMContext& Context, Instruction *I,
Instruction *J);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
bool expandIEChain(LLVMContext& Context, Instruction *I, Instruction *J,
unsigned o, Value *&LOp, unsigned numElemL,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
Type *ArgTypeL, Type *ArgTypeR, bool IBeforeJ,
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
unsigned IdxOff = 0);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Value *getReplacementInput(LLVMContext& Context, Instruction *I,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
Instruction *J, unsigned o, bool IBeforeJ);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
void getReplacementInputsForPair(LLVMContext& Context, Instruction *I,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
Instruction *J, SmallVector<Value *, 3> &ReplacedOperands,
bool IBeforeJ);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
void replaceOutputsOfPair(LLVMContext& Context, Instruction *I,
Instruction *J, Instruction *K,
Instruction *&InsertionPt, Instruction *&K1,
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
Instruction *&K2);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
void collectPairLoadMoveSet(BasicBlock &BB,
DenseMap<Value *, Value *> &ChosenPairs,
std::multimap<Value *, Value *> &LoadMoveSet,
Instruction *I);
void collectLoadMoveSet(BasicBlock &BB,
std::vector<Value *> &PairableInsts,
DenseMap<Value *, Value *> &ChosenPairs,
std::multimap<Value *, Value *> &LoadMoveSet);
bool canMoveUsesOfIAfterJ(BasicBlock &BB,
std::multimap<Value *, Value *> &LoadMoveSet,
Instruction *I, Instruction *J);
void moveUsesOfIAfterJ(BasicBlock &BB,
std::multimap<Value *, Value *> &LoadMoveSet,
Instruction *&InsertionPt,
Instruction *I, Instruction *J);
Teach BBVectorize to combine, when possible, or discard metadata when fusing instructions. The present implementation handles only TBAA and FP metadata, discarding everything else. For debug metadata, the current behavior is maintained (the debug metadata associated with one of the instructions will be kept, discarding that attached to the other). This should address PR 13040. llvm-svn: 158606
2012-06-17 04:34:06 +08:00
void combineMetadata(Instruction *K, const Instruction *J);
Add the function "vectorizeBasicBlock" which allow users vectorize a BasicBlock in other passes, e.g. we can call vectorizeBasicBlock in the loop unroll pass right after the loop is unrolled. llvm-svn: 154089
2012-04-05 16:05:16 +08:00
bool vectorizeBB(BasicBlock &BB) {
BBVectorize should ignore unreachable blocks. Unreachable blocks can have invalid instructions. For example, jump threading can produce self-referential instructions in unreachable blocks. Also, we should not be spending time optimizing unreachable code. Fixes PR14133. llvm-svn: 166423
2012-10-23 02:00:55 +08:00
if (!DT->isReachableFromEntry(&BB)) {
DEBUG(dbgs() << "BBV: skipping unreachable " << BB.getName() <<
" in " << BB.getParent()->getName() << "\n");
return false;
}
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
DEBUG(if (TTI) dbgs() << "BBV: using target information\n");
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool changed = false;
// Iterate a sufficient number of times to merge types of size 1 bit,
// then 2 bits, then 4, etc. up to half of the target vector width of the
// target vector register.
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
unsigned n = 1;
for (unsigned v = 2;
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
(TTI || v <= Config.VectorBits) &&
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
(!Config.MaxIter || n <= Config.MaxIter);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
v *= 2, ++n) {
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
DEBUG(dbgs() << "BBV: fusing loop #" << n <<
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
" for " << BB.getName() << " in " <<
BB.getParent()->getName() << "...\n");
if (vectorizePairs(BB))
changed = true;
else
break;
}
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
if (changed && !Pow2LenOnly) {
++n;
for (; !Config.MaxIter || n <= Config.MaxIter; ++n) {
DEBUG(dbgs() << "BBV: fusing for non-2^n-length vectors loop #: " <<
n << " for " << BB.getName() << " in " <<
BB.getParent()->getName() << "...\n");
if (!vectorizePairs(BB, true)) break;
}
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DEBUG(dbgs() << "BBV: done!\n");
return changed;
}
Add the function "vectorizeBasicBlock" which allow users vectorize a BasicBlock in other passes, e.g. we can call vectorizeBasicBlock in the loop unroll pass right after the loop is unrolled. llvm-svn: 154089
2012-04-05 16:05:16 +08:00
virtual bool runOnBasicBlock(BasicBlock &BB) {
AA = &getAnalysis<AliasAnalysis>();
BBVectorize should ignore unreachable blocks. Unreachable blocks can have invalid instructions. For example, jump threading can produce self-referential instructions in unreachable blocks. Also, we should not be spending time optimizing unreachable code. Fixes PR14133. llvm-svn: 166423
2012-10-23 02:00:55 +08:00
DT = &getAnalysis<DominatorTree>();
Add the function "vectorizeBasicBlock" which allow users vectorize a BasicBlock in other passes, e.g. we can call vectorizeBasicBlock in the loop unroll pass right after the loop is unrolled. llvm-svn: 154089
2012-04-05 16:05:16 +08:00
SE = &getAnalysis<ScalarEvolution>();
Move TargetData to DataLayout. llvm-svn: 165402
2012-10-09 00:38:25 +08:00
TD = getAnalysisIfAvailable<DataLayout>();
Switch BBVectorize to directly depend on having a TTI analysis. This could be simplified further, but Hal has a specific feature for ignoring TTI, and so I preserved that. Also, I needed to use it because a number of tests fail when switching from a null TTI to the NoTTI nonce implementation. That seems suspicious to me and so may be something that you need to look into Hal. I worked it by preserving the old behavior for these tests with the flag that ignores all target info. llvm-svn: 171722
2013-01-07 18:22:36 +08:00
TTI = IgnoreTargetInfo ? 0 : &getAnalysis<TargetTransformInfo>();
Add the function "vectorizeBasicBlock" which allow users vectorize a BasicBlock in other passes, e.g. we can call vectorizeBasicBlock in the loop unroll pass right after the loop is unrolled. llvm-svn: 154089
2012-04-05 16:05:16 +08:00
return vectorizeBB(BB);
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
virtual void getAnalysisUsage(AnalysisUsage &AU) const {
BasicBlockPass::getAnalysisUsage(AU);
AU.addRequired<AliasAnalysis>();
BBVectorize should ignore unreachable blocks. Unreachable blocks can have invalid instructions. For example, jump threading can produce self-referential instructions in unreachable blocks. Also, we should not be spending time optimizing unreachable code. Fixes PR14133. llvm-svn: 166423
2012-10-23 02:00:55 +08:00
AU.addRequired<DominatorTree>();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
AU.addRequired<ScalarEvolution>();
Switch BBVectorize to directly depend on having a TTI analysis. This could be simplified further, but Hal has a specific feature for ignoring TTI, and so I preserved that. Also, I needed to use it because a number of tests fail when switching from a null TTI to the NoTTI nonce implementation. That seems suspicious to me and so may be something that you need to look into Hal. I worked it by preserving the old behavior for these tests with the flag that ignores all target info. llvm-svn: 171722
2013-01-07 18:22:36 +08:00
AU.addRequired<TargetTransformInfo>();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
AU.addPreserved<AliasAnalysis>();
BBVectorize should ignore unreachable blocks. Unreachable blocks can have invalid instructions. For example, jump threading can produce self-referential instructions in unreachable blocks. Also, we should not be spending time optimizing unreachable code. Fixes PR14133. llvm-svn: 166423
2012-10-23 02:00:55 +08:00
AU.addPreserved<DominatorTree>();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
AU.addPreserved<ScalarEvolution>();
A few of the changes suggested in code review (by Nick Lewycky) llvm-svn: 149472
2012-02-01 13:51:45 +08:00
AU.setPreservesCFG();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
static inline VectorType *getVecTypeForPair(Type *ElemTy, Type *Elem2Ty) {
assert(ElemTy->getScalarType() == Elem2Ty->getScalarType() &&
"Cannot form vector from incompatible scalar types");
Type *STy = ElemTy->getScalarType();
unsigned numElem;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (VectorType *VTy = dyn_cast<VectorType>(ElemTy)) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
numElem = VTy->getNumElements();
} else {
numElem = 1;
}
if (VectorType *VTy = dyn_cast<VectorType>(Elem2Ty)) {
numElem += VTy->getNumElements();
} else {
numElem += 1;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
A few of the changes suggested in code review (by Nick Lewycky) llvm-svn: 149472
2012-02-01 13:51:45 +08:00
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
return VectorType::get(STy, numElem);
}
static inline void getInstructionTypes(Instruction *I,
Type *&T1, Type *&T2) {
if (isa<StoreInst>(I)) {
// For stores, it is the value type, not the pointer type that matters
// because the value is what will come from a vector register.
Value *IVal = cast<StoreInst>(I)->getValueOperand();
T1 = IVal->getType();
} else {
T1 = I->getType();
}
if (I->isCast())
T2 = cast<CastInst>(I)->getSrcTy();
else
T2 = T1;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (SelectInst *SI = dyn_cast<SelectInst>(I)) {
T2 = SI->getCondition()->getType();
BBVectorize: Check the input types of shuffles for legality This fixes a bug where shuffles were being fused such that the resulting input types were not legal on the target. This would occur only when both inputs and dependencies were also foldable operations (such as other shuffles) and there were other connected pairs in the same block. llvm-svn: 167731
2012-11-12 22:50:59 +08:00
} else if (ShuffleVectorInst *SI = dyn_cast<ShuffleVectorInst>(I)) {
T2 = SI->getOperand(0)->getType();
BBVectorize: Check the types of compare instructions The pass would previously assert when trying to compute the cost of compare instructions with illegal vector types (like struct pointers). llvm-svn: 167743
2012-11-13 03:41:38 +08:00
} else if (CmpInst *CI = dyn_cast<CmpInst>(I)) {
T2 = CI->getOperand(0)->getType();
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
// Returns the weight associated with the provided value. A chain of
// candidate pairs has a length given by the sum of the weights of its
// members (one weight per pair; the weight of each member of the pair
// is assumed to be the same). This length is then compared to the
// chain-length threshold to determine if a given chain is significant
// enough to be vectorized. The length is also used in comparing
// candidate chains where longer chains are considered to be better.
// Note: when this function returns 0, the resulting instructions are
// not actually fused.
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
inline size_t getDepthFactor(Value *V) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// InsertElement and ExtractElement have a depth factor of zero. This is
// for two reasons: First, they cannot be usefully fused. Second, because
// the pass generates a lot of these, they can confuse the simple metric
// used to compare the trees in the next iteration. Thus, giving them a
// weight of zero allows the pass to essentially ignore them in
// subsequent iterations when looking for vectorization opportunities
// while still tracking dependency chains that flow through those
// instructions.
if (isa<InsertElementInst>(V) || isa<ExtractElementInst>(V))
return 0;
Boost the effective chain depth of loads and stores. By default, boost the chain depth contribution of loads and stores. This will allow a load/store pair to vectorize even when it would not otherwise be long enough to satisfy the chain depth requirement. llvm-svn: 149761
2012-02-04 12:14:04 +08:00
// Give a load or store half of the required depth so that load/store
// pairs will vectorize.
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
if (!Config.NoMemOpBoost && (isa<LoadInst>(V) || isa<StoreInst>(V)))
return Config.ReqChainDepth/2;
Boost the effective chain depth of loads and stores. By default, boost the chain depth contribution of loads and stores. This will allow a load/store pair to vectorize even when it would not otherwise be long enough to satisfy the chain depth requirement. llvm-svn: 149761
2012-02-04 12:14:04 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return 1;
}
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
// Returns the cost of the provided instruction using TTI.
Update BBVectorize to use the new VTTI instr. cost interfaces. The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. llvm-svn: 166865
2012-10-27 12:33:48 +08:00
// This does not handle loads and stores.
unsigned getInstrCost(unsigned Opcode, Type *T1, Type *T2) {
switch (Opcode) {
default: break;
case Instruction::GetElementPtr:
// We mark this instruction as zero-cost because scalar GEPs are usually
// lowered to the intruction addressing mode. At the moment we don't
// generate vector GEPs.
return 0;
case Instruction::Br:
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
return TTI->getCFInstrCost(Opcode);
Update BBVectorize to use the new VTTI instr. cost interfaces. The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. llvm-svn: 166865
2012-10-27 12:33:48 +08:00
case Instruction::PHI:
return 0;
case Instruction::Add:
case Instruction::FAdd:
case Instruction::Sub:
case Instruction::FSub:
case Instruction::Mul:
case Instruction::FMul:
case Instruction::UDiv:
case Instruction::SDiv:
case Instruction::FDiv:
case Instruction::URem:
case Instruction::SRem:
case Instruction::FRem:
case Instruction::Shl:
case Instruction::LShr:
case Instruction::AShr:
case Instruction::And:
case Instruction::Or:
case Instruction::Xor:
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
return TTI->getArithmeticInstrCost(Opcode, T1);
Update BBVectorize to use the new VTTI instr. cost interfaces. The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. llvm-svn: 166865
2012-10-27 12:33:48 +08:00
case Instruction::Select:
case Instruction::ICmp:
case Instruction::FCmp:
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
return TTI->getCmpSelInstrCost(Opcode, T1, T2);
Update BBVectorize to use the new VTTI instr. cost interfaces. The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. llvm-svn: 166865
2012-10-27 12:33:48 +08:00
case Instruction::ZExt:
case Instruction::SExt:
case Instruction::FPToUI:
case Instruction::FPToSI:
case Instruction::FPExt:
case Instruction::PtrToInt:
case Instruction::IntToPtr:
case Instruction::SIToFP:
case Instruction::UIToFP:
case Instruction::Trunc:
case Instruction::FPTrunc:
case Instruction::BitCast:
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
case Instruction::ShuffleVector:
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
return TTI->getCastInstrCost(Opcode, T1, T2);
Update BBVectorize to use the new VTTI instr. cost interfaces. The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. llvm-svn: 166865
2012-10-27 12:33:48 +08:00
}
return 1;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// This determines the relative offset of two loads or stores, returning
// true if the offset could be determined to be some constant value.
// For example, if OffsetInElmts == 1, then J accesses the memory directly
// after I; if OffsetInElmts == -1 then I accesses the memory
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
// directly after J.
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool getPairPtrInfo(Instruction *I, Instruction *J,
Value *&IPtr, Value *&JPtr, unsigned &IAlignment, unsigned &JAlignment,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
unsigned &IAddressSpace, unsigned &JAddressSpace,
BBVectorize: Don't make calls to SE when the result is unused. SE was being called during the instruction-fusion process (when the result is unreliable, and thus ignored). No functionality change is intended. llvm-svn: 167037
2012-10-31 02:55:49 +08:00
int64_t &OffsetInElmts, bool ComputeOffset = true) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
OffsetInElmts = 0;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (LoadInst *LI = dyn_cast<LoadInst>(I)) {
LoadInst *LJ = cast<LoadInst>(J);
IPtr = LI->getPointerOperand();
JPtr = LJ->getPointerOperand();
IAlignment = LI->getAlignment();
JAlignment = LJ->getAlignment();
IAddressSpace = LI->getPointerAddressSpace();
JAddressSpace = LJ->getPointerAddressSpace();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
} else {
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
StoreInst *SI = cast<StoreInst>(I), *SJ = cast<StoreInst>(J);
IPtr = SI->getPointerOperand();
JPtr = SJ->getPointerOperand();
IAlignment = SI->getAlignment();
JAlignment = SJ->getAlignment();
IAddressSpace = SI->getPointerAddressSpace();
JAddressSpace = SJ->getPointerAddressSpace();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
BBVectorize: Don't make calls to SE when the result is unused. SE was being called during the instruction-fusion process (when the result is unreliable, and thus ignored). No functionality change is intended. llvm-svn: 167037
2012-10-31 02:55:49 +08:00
if (!ComputeOffset)
return true;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
const SCEV *IPtrSCEV = SE->getSCEV(IPtr);
const SCEV *JPtrSCEV = SE->getSCEV(JPtr);
// If this is a trivial offset, then we'll get something like
// 1*sizeof(type). With target data, which we need anyway, this will get
// constant folded into a number.
const SCEV *OffsetSCEV = SE->getMinusSCEV(JPtrSCEV, IPtrSCEV);
if (const SCEVConstant *ConstOffSCEV =
dyn_cast<SCEVConstant>(OffsetSCEV)) {
ConstantInt *IntOff = ConstOffSCEV->getValue();
int64_t Offset = IntOff->getSExtValue();
Type *VTy = cast<PointerType>(IPtr->getType())->getElementType();
int64_t VTyTSS = (int64_t) TD->getTypeStoreSize(VTy);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *VTy2 = cast<PointerType>(JPtr->getType())->getElementType();
if (VTy != VTy2 && Offset < 0) {
int64_t VTy2TSS = (int64_t) TD->getTypeStoreSize(VTy2);
OffsetInElmts = Offset/VTy2TSS;
return (abs64(Offset) % VTy2TSS) == 0;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
OffsetInElmts = Offset/VTyTSS;
return (abs64(Offset) % VTyTSS) == 0;
}
return false;
}
// Returns true if the provided CallInst represents an intrinsic that can
// be vectorized.
bool isVectorizableIntrinsic(CallInst* I) {
Function *F = I->getCalledFunction();
if (!F) return false;
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
Intrinsic::ID IID = (Intrinsic::ID) F->getIntrinsicID();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (!IID) return false;
switch(IID) {
default:
return false;
case Intrinsic::sqrt:
case Intrinsic::powi:
case Intrinsic::sin:
case Intrinsic::cos:
case Intrinsic::log:
case Intrinsic::log2:
case Intrinsic::log10:
case Intrinsic::exp:
case Intrinsic::exp2:
case Intrinsic::pow:
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
return Config.VectorizeMath;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
case Intrinsic::fma:
BBVectorize: Enable vectorization of the fmuladd intrinsic llvm-svn: 171075
2012-12-26 06:36:08 +08:00
case Intrinsic::fmuladd:
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
return Config.VectorizeFMA;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
// Returns true if J is the second element in some pair referenced by
// some multimap pair iterator pair.
template <typename V>
bool isSecondInIteratorPair(V J, std::pair<
typename std::multimap<V, V>::iterator,
typename std::multimap<V, V>::iterator> PairRange) {
for (typename std::multimap<V, V>::iterator K = PairRange.first;
K != PairRange.second; ++K)
if (K->second == J) return true;
return false;
}
BBVectorize: Only some insert element operand pairs are free. This fixes another infinite recursion case when using target costs. We can only replace insert element input chains that are pure (end with inserting into an undef). llvm-svn: 167784
2012-11-13 07:55:36 +08:00
bool isPureIEChain(InsertElementInst *IE) {
InsertElementInst *IENext = IE;
do {
if (!isa<UndefValue>(IENext->getOperand(0)) &&
!isa<InsertElementInst>(IENext->getOperand(0))) {
return false;
}
} while ((IENext =
dyn_cast<InsertElementInst>(IENext->getOperand(0))));
return true;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
};
// This function implements one vectorization iteration on the provided
// basic block. It returns true if the block is changed.
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
bool BBVectorize::vectorizePairs(BasicBlock &BB, bool NonPow2Len) {
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
bool ShouldContinue;
BasicBlock::iterator Start = BB.getFirstInsertionPt();
std::vector<Value *> AllPairableInsts;
DenseMap<Value *, Value *> AllChosenPairs;
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
DenseSet<ValuePair> AllFixedOrderPairs;
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
DenseMap<VPPair, unsigned> AllPairConnectionTypes;
std::multimap<ValuePair, ValuePair> AllConnectedPairs, AllConnectedPairDeps;
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
do {
std::vector<Value *> PairableInsts;
std::multimap<Value *, Value *> CandidatePairs;
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
DenseSet<ValuePair> FixedOrderPairs;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
DenseMap<ValuePair, int> CandidatePairCostSavings;
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
ShouldContinue = getCandidatePairs(BB, Start, CandidatePairs,
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
FixedOrderPairs,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
CandidatePairCostSavings,
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
PairableInsts, NonPow2Len);
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
if (PairableInsts.empty()) continue;
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
// Now we have a map of all of the pairable instructions and we need to
// select the best possible pairing. A good pairing is one such that the
// users of the pair are also paired. This defines a (directed) forest
Revert 'Fix a typo 'iff' => 'if''. iff is an abreviation of if and only if. See: http://en.wikipedia.org/wiki/If_and_only_if Commit 164767 llvm-svn: 164768
2012-09-27 18:14:43 +08:00
// over the pairs such that two pairs are connected iff the second pair
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
// uses the first.
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
// Note that it only matters that both members of the second pair use some
// element of the first pair (to allow for splatting).
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
std::multimap<ValuePair, ValuePair> ConnectedPairs, ConnectedPairDeps;
DenseMap<VPPair, unsigned> PairConnectionTypes;
computeConnectedPairs(CandidatePairs, PairableInsts, ConnectedPairs,
PairConnectionTypes);
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
if (ConnectedPairs.empty()) continue;
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
for (std::multimap<ValuePair, ValuePair>::iterator
I = ConnectedPairs.begin(), IE = ConnectedPairs.end();
I != IE; ++I) {
ConnectedPairDeps.insert(VPPair(I->second, I->first));
}
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
// Build the pairable-instruction dependency map
DenseSet<ValuePair> PairableInstUsers;
buildDepMap(BB, CandidatePairs, PairableInsts, PairableInstUsers);
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
// There is now a graph of the connected pairs. For each variable, pick
// the pairing with the largest tree meeting the depth requirement on at
// least one branch. Then select all pairings that are part of that tree
// and remove them from the list of available pairings and pairable
// variables.
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
DenseMap<Value *, Value *> ChosenPairs;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
choosePairs(CandidatePairs, CandidatePairCostSavings,
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
PairableInsts, FixedOrderPairs, PairConnectionTypes,
ConnectedPairs, ConnectedPairDeps,
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
PairableInstUsers, ChosenPairs);
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
if (ChosenPairs.empty()) continue;
AllPairableInsts.insert(AllPairableInsts.end(), PairableInsts.begin(),
PairableInsts.end());
AllChosenPairs.insert(ChosenPairs.begin(), ChosenPairs.end());
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
// Only for the chosen pairs, propagate information on fixed-order pairs,
// pair connections, and their types to the data structures used by the
// pair fusion procedures.
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
for (DenseMap<Value *, Value *>::iterator I = ChosenPairs.begin(),
IE = ChosenPairs.end(); I != IE; ++I) {
if (FixedOrderPairs.count(*I))
AllFixedOrderPairs.insert(*I);
else if (FixedOrderPairs.count(ValuePair(I->second, I->first)))
AllFixedOrderPairs.insert(ValuePair(I->second, I->first));
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
for (DenseMap<Value *, Value *>::iterator J = ChosenPairs.begin();
J != IE; ++J) {
DenseMap<VPPair, unsigned>::iterator K =
PairConnectionTypes.find(VPPair(*I, *J));
if (K != PairConnectionTypes.end()) {
AllPairConnectionTypes.insert(*K);
} else {
K = PairConnectionTypes.find(VPPair(*J, *I));
if (K != PairConnectionTypes.end())
AllPairConnectionTypes.insert(*K);
}
}
}
for (std::multimap<ValuePair, ValuePair>::iterator
I = ConnectedPairs.begin(), IE = ConnectedPairs.end();
I != IE; ++I) {
if (AllPairConnectionTypes.count(*I)) {
AllConnectedPairs.insert(*I);
AllConnectedPairDeps.insert(VPPair(I->second, I->first));
}
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
}
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
} while (ShouldContinue);
if (AllChosenPairs.empty()) return false;
NumFusedOps += AllChosenPairs.size();
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// A set of pairs has now been selected. It is now necessary to replace the
// paired instructions with vector instructions. For this procedure each
fix typos llvm-svn: 152035
2012-03-06 01:39:47 +08:00
// operand must be replaced with a vector operand. This vector is formed
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// by using build_vector on the old operands. The replaced values are then
// replaced with a vector_extract on the result. Subsequent optimization
// passes should coalesce the build/extract combinations.
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
fuseChosenPairs(BB, AllPairableInsts, AllChosenPairs, AllFixedOrderPairs,
AllPairConnectionTypes,
AllConnectedPairs, AllConnectedPairDeps);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
// It is important to cleanup here so that future iterations of this
// function have less work to do.
Make MemoryBuiltins aware of TargetLibraryInfo. This disables malloc-specific optimization when -fno-builtin (or -ffreestanding) is specified. This has been a problem for a long time but became more severe with the recent memory builtin improvements. Since the memory builtin functions are used everywhere, this required passing TLI in many places. This means that functions that now have an optional TLI argument, like RecursivelyDeleteTriviallyDeadFunctions, won't remove dead mallocs anymore if the TLI argument is missing. I've updated most passes to do the right thing. Fixes PR13694 and probably others. llvm-svn: 162841
2012-08-29 23:32:21 +08:00
(void) SimplifyInstructionsInBlock(&BB, TD, AA->getTargetLibraryInfo());
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return true;
}
// This function returns true if the provided instruction is capable of being
// fused into a vector instruction. This determination is based only on the
// type and other attributes of the instruction.
bool BBVectorize::isInstVectorizable(Instruction *I,
bool &IsSimpleLoadStore) {
IsSimpleLoadStore = false;
if (CallInst *C = dyn_cast<CallInst>(I)) {
if (!isVectorizableIntrinsic(C))
return false;
} else if (LoadInst *L = dyn_cast<LoadInst>(I)) {
// Vectorize simple loads if possbile:
IsSimpleLoadStore = L->isSimple();
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
if (!IsSimpleLoadStore || !Config.VectorizeMemOps)
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
} else if (StoreInst *S = dyn_cast<StoreInst>(I)) {
// Vectorize simple stores if possbile:
IsSimpleLoadStore = S->isSimple();
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
if (!IsSimpleLoadStore || !Config.VectorizeMemOps)
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
} else if (CastInst *C = dyn_cast<CastInst>(I)) {
// We can vectorize casts, but not casts of pointer types, etc.
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
if (!Config.VectorizeCasts)
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
Type *SrcTy = C->getSrcTy();
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
if (!SrcTy->isSingleValueType())
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
Type *DestTy = C->getDestTy();
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
if (!DestTy->isSingleValueType())
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
Add support to BBVectorize for vectorizing selects. llvm-svn: 154700
2012-04-14 04:45:45 +08:00
} else if (isa<SelectInst>(I)) {
if (!Config.VectorizeSelect)
return false;
Allow BBVectorize to fuse compare instructions. llvm-svn: 159088
2012-06-24 05:52:50 +08:00
} else if (isa<CmpInst>(I)) {
if (!Config.VectorizeCmp)
return false;
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
} else if (GetElementPtrInst *G = dyn_cast<GetElementPtrInst>(I)) {
if (!Config.VectorizeGEP)
return false;
// Currently, vector GEPs exist only with one index.
if (G->getNumIndices() != 1)
return false;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
} else if (!(I->isBinaryOp() || isa<ShuffleVectorInst>(I) ||
isa<ExtractElementInst>(I) || isa<InsertElementInst>(I))) {
return false;
}
// We can't vectorize memory operations without target data
if (TD == 0 && IsSimpleLoadStore)
return false;
Type *T1, *T2;
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
getInstructionTypes(I, T1, T2);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Not every type can be vectorized...
if (!(VectorType::isValidElementType(T1) || T1->isVectorTy()) ||
!(VectorType::isValidElementType(T2) || T2->isVectorTy()))
return false;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (T1->getScalarSizeInBits() == 1) {
Allow controlling vectorization of boolean values separately from other integer types. These are used as the result of comparisons, and often handled differently from larger integer types. llvm-svn: 159111
2012-06-24 21:28:01 +08:00
if (!Config.VectorizeBools)
return false;
} else {
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (!Config.VectorizeInts && T1->isIntOrIntVectorTy())
Allow controlling vectorization of boolean values separately from other integer types. These are used as the result of comparisons, and often handled differently from larger integer types. llvm-svn: 159111
2012-06-24 21:28:01 +08:00
return false;
}
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (T2->getScalarSizeInBits() == 1) {
if (!Config.VectorizeBools)
return false;
} else {
if (!Config.VectorizeInts && T2->isIntOrIntVectorTy())
return false;
}
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
if (!Config.VectorizeFloats
&& (T1->isFPOrFPVectorTy() || T2->isFPOrFPVectorTy()))
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
Don't vectorize target-specific types (ppc_fp128, x86_fp80, etc.). Target specific types should not be vectorized. As a practical matter, these types are already register matched (at least in the x86 case), and codegen does not always work correctly (at least in the ppc case, and this is not worth fixing because ppc_fp128 is currently broken and will probably go away soon). llvm-svn: 155729
2012-04-28 03:34:00 +08:00
// Don't vectorize target-specific types.
if (T1->isX86_FP80Ty() || T1->isPPC_FP128Ty() || T1->isX86_MMXTy())
return false;
if (T2->isX86_FP80Ty() || T2->isPPC_FP128Ty() || T2->isX86_MMXTy())
return false;
Fix style violation in BBVectorize (pointed out by Bill Wendling) llvm-svn: 154810
2012-04-16 20:39:17 +08:00
if ((!Config.VectorizePointers || TD == 0) &&
(T1->getScalarType()->isPointerTy() ||
T2->getScalarType()->isPointerTy()))
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
return false;
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
if (!TTI && (T1->getPrimitiveSizeInBits() >= Config.VectorBits ||
T2->getPrimitiveSizeInBits() >= Config.VectorBits))
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
return true;
}
// This function returns true if the two provided instructions are compatible
// (meaning that they can be fused into a vector instruction). This assumes
// that I has already been determined to be vectorizable and that J is not
// in the use tree of I.
bool BBVectorize::areInstsCompatible(Instruction *I, Instruction *J,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
bool IsSimpleLoadStore, bool NonPow2Len,
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
int &CostSavings, int &FixedOrder) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DEBUG(if (DebugInstructionExamination) dbgs() << "BBV: looking at " << *I <<
" <-> " << *J << "\n");
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
CostSavings = 0;
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
FixedOrder = 0;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Loads and stores can be merged if they have different alignments,
// but are otherwise the same.
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
if (!J->isSameOperationAs(I, Instruction::CompareIgnoringAlignment |
(NonPow2Len ? Instruction::CompareUsingScalarTypes : 0)))
return false;
Type *IT1, *IT2, *JT1, *JT2;
getInstructionTypes(I, IT1, IT2);
getInstructionTypes(J, JT1, JT2);
unsigned MaxTypeBits = std::max(
IT1->getPrimitiveSizeInBits() + JT1->getPrimitiveSizeInBits(),
IT2->getPrimitiveSizeInBits() + JT2->getPrimitiveSizeInBits());
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
if (!TTI && MaxTypeBits > Config.VectorBits)
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
Refactor operation equivalence checking in BBVectorize by extending Instruction::isSameOperationAs. Maintaining this kind of checking in different places is dangerous, extending Instruction::isSameOperationAs consolidates this logic into one place. Here I've added an optional flags parameter and two flags that are important for vectorization: CompareIgnoringAlignment and CompareUsingScalarTypes. llvm-svn: 159329
2012-06-28 13:42:26 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// FIXME: handle addsub-type operations!
if (IsSimpleLoadStore) {
Value *IPtr, *JPtr;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
unsigned IAlignment, JAlignment, IAddressSpace, JAddressSpace;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
int64_t OffsetInElmts = 0;
if (getPairPtrInfo(I, J, IPtr, JPtr, IAlignment, JAlignment,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
IAddressSpace, JAddressSpace,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
OffsetInElmts) && abs64(OffsetInElmts) == 1) {
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
FixedOrder = (int) OffsetInElmts;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
unsigned BottomAlignment = IAlignment;
if (OffsetInElmts < 0) BottomAlignment = JAlignment;
Type *aTypeI = isa<StoreInst>(I) ?
cast<StoreInst>(I)->getValueOperand()->getType() : I->getType();
Type *aTypeJ = isa<StoreInst>(J) ?
cast<StoreInst>(J)->getValueOperand()->getType() : J->getType();
Type *VType = getVecTypeForPair(aTypeI, aTypeJ);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (Config.AlignedOnly) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// An aligned load or store is possible only if the instruction
// with the lower offset has an alignment suitable for the
// vector type.
fix indentation llvm-svn: 149857
2012-02-06 13:29:32 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
unsigned VecAlignment = TD->getPrefTypeAlignment(VType);
if (BottomAlignment < VecAlignment)
return false;
}
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
if (TTI) {
unsigned ICost = TTI->getMemoryOpCost(I->getOpcode(), aTypeI,
IAlignment, IAddressSpace);
unsigned JCost = TTI->getMemoryOpCost(J->getOpcode(), aTypeJ,
JAlignment, JAddressSpace);
unsigned VCost = TTI->getMemoryOpCost(I->getOpcode(), VType,
BottomAlignment,
IAddressSpace);
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (VCost > ICost + JCost)
return false;
BBVectorize, when using VTTI, should not form types that will be split. This is needed so that perl's SHA can be compiled (otherwise BBVectorize takes far too long to find its fixed point). I'll try to come up with a reduced test case. llvm-svn: 166738
2012-10-26 07:47:16 +08:00
Use VTTI->getNumberOfParts in BBVectorize. This change reflects VTTI refactoring; no functionality change intended. llvm-svn: 166752
2012-10-26 12:28:06 +08:00
// We don't want to fuse to a type that will be split, even
BBVectorize, when using VTTI, should not form types that will be split. This is needed so that perl's SHA can be compiled (otherwise BBVectorize takes far too long to find its fixed point). I'll try to come up with a reduced test case. llvm-svn: 166738
2012-10-26 07:47:16 +08:00
// if the two input types will also be split and there is no other
Use VTTI->getNumberOfParts in BBVectorize. This change reflects VTTI refactoring; no functionality change intended. llvm-svn: 166752
2012-10-26 12:28:06 +08:00
// associated cost.
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
unsigned VParts = TTI->getNumberOfParts(VType);
Use VTTI->getNumberOfParts in BBVectorize. This change reflects VTTI refactoring; no functionality change intended. llvm-svn: 166752
2012-10-26 12:28:06 +08:00
if (VParts > 1)
return false;
else if (!VParts && VCost == ICost + JCost)
BBVectorize, when using VTTI, should not form types that will be split. This is needed so that perl's SHA can be compiled (otherwise BBVectorize takes far too long to find its fixed point). I'll try to come up with a reduced test case. llvm-svn: 166738
2012-10-26 07:47:16 +08:00
return false;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
CostSavings = ICost + JCost - VCost;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
} else {
return false;
}
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
} else if (TTI) {
Update BBVectorize to use the new VTTI instr. cost interfaces. The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. llvm-svn: 166865
2012-10-27 12:33:48 +08:00
unsigned ICost = getInstrCost(I->getOpcode(), IT1, IT2);
unsigned JCost = getInstrCost(J->getOpcode(), JT1, JT2);
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
Type *VT1 = getVecTypeForPair(IT1, JT1),
*VT2 = getVecTypeForPair(IT2, JT2);
Update BBVectorize to use the new VTTI instr. cost interfaces. The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. llvm-svn: 166865
2012-10-27 12:33:48 +08:00
unsigned VCost = getInstrCost(I->getOpcode(), VT1, VT2);
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (VCost > ICost + JCost)
return false;
BBVectorize, when using VTTI, should not form types that will be split. This is needed so that perl's SHA can be compiled (otherwise BBVectorize takes far too long to find its fixed point). I'll try to come up with a reduced test case. llvm-svn: 166738
2012-10-26 07:47:16 +08:00
Use VTTI->getNumberOfParts in BBVectorize. This change reflects VTTI refactoring; no functionality change intended. llvm-svn: 166752
2012-10-26 12:28:06 +08:00
// We don't want to fuse to a type that will be split, even
BBVectorize, when using VTTI, should not form types that will be split. This is needed so that perl's SHA can be compiled (otherwise BBVectorize takes far too long to find its fixed point). I'll try to come up with a reduced test case. llvm-svn: 166738
2012-10-26 07:47:16 +08:00
// if the two input types will also be split and there is no other
Use VTTI->getNumberOfParts in BBVectorize. This change reflects VTTI refactoring; no functionality change intended. llvm-svn: 166752
2012-10-26 12:28:06 +08:00
// associated cost.
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
unsigned VParts1 = TTI->getNumberOfParts(VT1),
VParts2 = TTI->getNumberOfParts(VT2);
BBVectorize: Check the input types of shuffles for legality This fixes a bug where shuffles were being fused such that the resulting input types were not legal on the target. This would occur only when both inputs and dependencies were also foldable operations (such as other shuffles) and there were other connected pairs in the same block. llvm-svn: 167731
2012-11-12 22:50:59 +08:00
if (VParts1 > 1 || VParts2 > 1)
Use VTTI->getNumberOfParts in BBVectorize. This change reflects VTTI refactoring; no functionality change intended. llvm-svn: 166752
2012-10-26 12:28:06 +08:00
return false;
BBVectorize: Check the input types of shuffles for legality This fixes a bug where shuffles were being fused such that the resulting input types were not legal on the target. This would occur only when both inputs and dependencies were also foldable operations (such as other shuffles) and there were other connected pairs in the same block. llvm-svn: 167731
2012-11-12 22:50:59 +08:00
else if ((!VParts1 || !VParts2) && VCost == ICost + JCost)
BBVectorize, when using VTTI, should not form types that will be split. This is needed so that perl's SHA can be compiled (otherwise BBVectorize takes far too long to find its fixed point). I'll try to come up with a reduced test case. llvm-svn: 166738
2012-10-26 07:47:16 +08:00
return false;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
CostSavings = ICost + JCost - VCost;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Correctly vectorize powi. The powi intrinsic requires special handling because it always takes a single integer power regardless of the result type. As a result, we can vectorize only if the powers are equal. Fixes PR12364. llvm-svn: 153797
2012-03-31 11:38:40 +08:00
// The powi intrinsic is special because only the first argument is
// vectorized, the second arguments must be equal.
CallInst *CI = dyn_cast<CallInst>(I);
Function *FI;
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
if (CI && (FI = CI->getCalledFunction())) {
Intrinsic::ID IID = (Intrinsic::ID) FI->getIntrinsicID();
if (IID == Intrinsic::powi) {
Value *A1I = CI->getArgOperand(1),
*A1J = cast<CallInst>(J)->getArgOperand(1);
const SCEV *A1ISCEV = SE->getSCEV(A1I),
*A1JSCEV = SE->getSCEV(A1J);
return (A1ISCEV == A1JSCEV);
}
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
if (IID && TTI) {
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
SmallVector<Type*, 4> Tys;
for (unsigned i = 0, ie = CI->getNumArgOperands(); i != ie; ++i)
Tys.push_back(CI->getArgOperand(i)->getType());
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
unsigned ICost = TTI->getIntrinsicInstrCost(IID, IT1, Tys);
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
Tys.clear();
CallInst *CJ = cast<CallInst>(J);
for (unsigned i = 0, ie = CJ->getNumArgOperands(); i != ie; ++i)
Tys.push_back(CJ->getArgOperand(i)->getType());
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
unsigned JCost = TTI->getIntrinsicInstrCost(IID, JT1, Tys);
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
Tys.clear();
assert(CI->getNumArgOperands() == CJ->getNumArgOperands() &&
"Intrinsic argument counts differ");
for (unsigned i = 0, ie = CI->getNumArgOperands(); i != ie; ++i) {
if (IID == Intrinsic::powi && i == 1)
Tys.push_back(CI->getArgOperand(i)->getType());
else
Tys.push_back(getVecTypeForPair(CI->getArgOperand(i)->getType(),
CJ->getArgOperand(i)->getType()));
}
Type *RetTy = getVecTypeForPair(IT1, JT1);
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
unsigned VCost = TTI->getIntrinsicInstrCost(IID, RetTy, Tys);
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
if (VCost > ICost + JCost)
return false;
// We don't want to fuse to a type that will be split, even
// if the two input types will also be split and there is no other
// associated cost.
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
unsigned RetParts = TTI->getNumberOfParts(RetTy);
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
if (RetParts > 1)
return false;
else if (!RetParts && VCost == ICost + JCost)
return false;
for (unsigned i = 0, ie = CI->getNumArgOperands(); i != ie; ++i) {
if (!Tys[i]->isVectorTy())
continue;
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
unsigned NumParts = TTI->getNumberOfParts(Tys[i]);
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
if (NumParts > 1)
return false;
else if (!NumParts && VCost == ICost + JCost)
return false;
}
CostSavings = ICost + JCost - VCost;
}
Correctly vectorize powi. The powi intrinsic requires special handling because it always takes a single integer power regardless of the result type. As a result, we can vectorize only if the powers are equal. Fixes PR12364. llvm-svn: 153797
2012-03-31 11:38:40 +08:00
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return true;
}
// Figure out whether or not J uses I and update the users and write-set
// structures associated with I. Specifically, Users represents the set of
// instructions that depend on I. WriteSet represents the set
// of memory locations that are dependent on I. If UpdateUsers is true,
// and J uses I, then Users is updated to contain J and WriteSet is updated
// to contain any memory locations to which J writes. The function returns
// true if J uses I. By default, alias analysis is used to determine
// whether J reads from memory that overlaps with a location in WriteSet.
// If LoadMoveSet is not null, then it is a previously-computed multimap
// where the key is the memory-based user instruction and the value is
// the instruction to be compared with I. So, if LoadMoveSet is provided,
// then the alias analysis is not used. This is necessary because this
// function is called during the process of moving instructions during
// vectorization and the results of the alias analysis are not stable during
// that process.
bool BBVectorize::trackUsesOfI(DenseSet<Value *> &Users,
AliasSetTracker &WriteSet, Instruction *I,
Instruction *J, bool UpdateUsers,
std::multimap<Value *, Value *> *LoadMoveSet) {
bool UsesI = false;
// This instruction may already be marked as a user due, for example, to
// being a member of a selected pair.
if (Users.count(J))
UsesI = true;
if (!UsesI)
A few of the changes suggested in code review (by Nick Lewycky) llvm-svn: 149472
2012-02-01 13:51:45 +08:00
for (User::op_iterator JU = J->op_begin(), JE = J->op_end();
JU != JE; ++JU) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Value *V = *JU;
if (I == V || Users.count(V)) {
UsesI = true;
break;
}
}
if (!UsesI && J->mayReadFromMemory()) {
if (LoadMoveSet) {
VPIteratorPair JPairRange = LoadMoveSet->equal_range(J);
UsesI = isSecondInIteratorPair<Value*>(I, JPairRange);
} else {
for (AliasSetTracker::iterator W = WriteSet.begin(),
WE = WriteSet.end(); W != WE; ++W) {
Update BBVectorize to use aliasesUnknownInst. This allows BBVectorize to check the "unknown instruction" list in the alias sets. This is important to prevent instruction fusing from reordering function calls. Resolves PR11920. llvm-svn: 150250
2012-02-10 23:52:40 +08:00
if (W->aliasesUnknownInst(J, *AA)) {
UsesI = true;
break;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
}
}
if (UsesI && UpdateUsers) {
if (J->mayWriteToMemory()) WriteSet.add(J);
Users.insert(J);
}
return UsesI;
}
// This function iterates over all instruction pairs in the provided
// basic block and collects all candidate pairs for vectorization.
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
bool BBVectorize::getCandidatePairs(BasicBlock &BB,
BasicBlock::iterator &Start,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
std::multimap<Value *, Value *> &CandidatePairs,
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
DenseSet<ValuePair> &FixedOrderPairs,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
DenseMap<ValuePair, int> &CandidatePairCostSavings,
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
std::vector<Value *> &PairableInsts, bool NonPow2Len) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BasicBlock::iterator E = BB.end();
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
if (Start == E) return false;
bool ShouldContinue = false, IAfterStart = false;
for (BasicBlock::iterator I = Start++; I != E; ++I) {
if (I == Start) IAfterStart = true;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool IsSimpleLoadStore;
if (!isInstVectorizable(I, IsSimpleLoadStore)) continue;
// Look for an instruction with which to pair instruction *I...
DenseSet<Value *> Users;
AliasSetTracker WriteSet(*AA);
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
bool JAfterStart = IAfterStart;
BasicBlock::iterator J = llvm::next(I);
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
for (unsigned ss = 0; J != E && ss <= Config.SearchLimit; ++J, ++ss) {
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
if (J == Start) JAfterStart = true;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Determine if J uses I, if so, exit the loop.
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
bool UsesI = trackUsesOfI(Users, WriteSet, I, J, !Config.FastDep);
if (Config.FastDep) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Note: For this heuristic to be effective, independent operations
// must tend to be intermixed. This is likely to be true from some
// kinds of grouped loop unrolling (but not the generic LLVM pass),
// but otherwise may require some kind of reordering pass.
// When using fast dependency analysis,
// stop searching after first use:
if (UsesI) break;
} else {
if (UsesI) continue;
}
// J does not use I, and comes before the first use of I, so it can be
// merged with I if the instructions are compatible.
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
int CostSavings, FixedOrder;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (!areInstsCompatible(I, J, IsSimpleLoadStore, NonPow2Len,
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
CostSavings, FixedOrder)) continue;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// J is a candidate for merging with I.
if (!PairableInsts.size() ||
PairableInsts[PairableInsts.size()-1] != I) {
PairableInsts.push_back(I);
}
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
CandidatePairs.insert(ValuePair(I, J));
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
if (TTI)
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
CandidatePairCostSavings.insert(ValuePairWithCost(ValuePair(I, J),
CostSavings));
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
if (FixedOrder == 1)
FixedOrderPairs.insert(ValuePair(I, J));
else if (FixedOrder == -1)
FixedOrderPairs.insert(ValuePair(J, I));
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
// The next call to this function must start after the last instruction
// selected during this invocation.
if (JAfterStart) {
Start = llvm::next(J);
IAfterStart = JAfterStart = false;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DEBUG(if (DebugCandidateSelection) dbgs() << "BBV: candidate pair "
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
<< *I << " <-> " << *J << " (cost savings: " <<
CostSavings << ")\n");
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
// If we have already found too many pairs, break here and this function
// will be called again starting after the last instruction selected
// during this invocation.
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
if (PairableInsts.size() >= Config.MaxInsts) {
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
ShouldContinue = true;
break;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
if (ShouldContinue)
break;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
DEBUG(dbgs() << "BBV: found " << PairableInsts.size()
<< " instructions with candidate pairs\n");
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
return ShouldContinue;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
// Finds candidate pairs connected to the pair P = <PI, PJ>. This means that
// it looks for pairs such that both members have an input which is an
// output of PI or PJ.
void BBVectorize::computePairsConnectedTo(
std::multimap<Value *, Value *> &CandidatePairs,
std::vector<Value *> &PairableInsts,
std::multimap<ValuePair, ValuePair> &ConnectedPairs,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
DenseMap<VPPair, unsigned> &PairConnectionTypes,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
ValuePair P) {
Fix an error in BBVectorize important for vectorizing pointer types. When vectorizing pointer types it is important to realize that potential pairs cannot be connected via the address pointer argument of a load or store. This is because even after vectorization, the address is still a scalar because the address of the higher half of the pair is implicit from the address of the lower half (it need not be, and should not be, explicitly computed). llvm-svn: 154735
2012-04-14 15:32:50 +08:00
StoreInst *SI, *SJ;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// For each possible pairing for this variable, look at the uses of
// the first value...
for (Value::use_iterator I = P.first->use_begin(),
E = P.first->use_end(); I != E; ++I) {
Fix an error in BBVectorize important for vectorizing pointer types. When vectorizing pointer types it is important to realize that potential pairs cannot be connected via the address pointer argument of a load or store. This is because even after vectorization, the address is still a scalar because the address of the higher half of the pair is implicit from the address of the lower half (it need not be, and should not be, explicitly computed). llvm-svn: 154735
2012-04-14 15:32:50 +08:00
if (isa<LoadInst>(*I)) {
// A pair cannot be connected to a load because the load only takes one
// operand (the address) and it is a scalar even after vectorization.
continue;
} else if ((SI = dyn_cast<StoreInst>(*I)) &&
P.first == SI->getPointerOperand()) {
// Similarly, a pair cannot be connected to a store through its
// pointer operand.
continue;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
VPIteratorPair IPairRange = CandidatePairs.equal_range(*I);
// For each use of the first variable, look for uses of the second
// variable...
for (Value::use_iterator J = P.second->use_begin(),
E2 = P.second->use_end(); J != E2; ++J) {
Fix an error in BBVectorize important for vectorizing pointer types. When vectorizing pointer types it is important to realize that potential pairs cannot be connected via the address pointer argument of a load or store. This is because even after vectorization, the address is still a scalar because the address of the higher half of the pair is implicit from the address of the lower half (it need not be, and should not be, explicitly computed). llvm-svn: 154735
2012-04-14 15:32:50 +08:00
if ((SJ = dyn_cast<StoreInst>(*J)) &&
P.second == SJ->getPointerOperand())
continue;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
VPIteratorPair JPairRange = CandidatePairs.equal_range(*J);
// Look for <I, J>:
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
if (isSecondInIteratorPair<Value*>(*J, IPairRange)) {
VPPair VP(P, ValuePair(*I, *J));
ConnectedPairs.insert(VP);
PairConnectionTypes.insert(VPPairWithType(VP, PairConnectionDirect));
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Look for <J, I>:
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
if (isSecondInIteratorPair<Value*>(*I, JPairRange)) {
VPPair VP(P, ValuePair(*J, *I));
ConnectedPairs.insert(VP);
PairConnectionTypes.insert(VPPairWithType(VP, PairConnectionSwap));
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
if (Config.SplatBreaksChain) continue;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Look for cases where just the first value in the pair is used by
// both members of another pair (splatting).
for (Value::use_iterator J = P.first->use_begin(); J != E; ++J) {
Fix an error in BBVectorize important for vectorizing pointer types. When vectorizing pointer types it is important to realize that potential pairs cannot be connected via the address pointer argument of a load or store. This is because even after vectorization, the address is still a scalar because the address of the higher half of the pair is implicit from the address of the lower half (it need not be, and should not be, explicitly computed). llvm-svn: 154735
2012-04-14 15:32:50 +08:00
if ((SJ = dyn_cast<StoreInst>(*J)) &&
P.first == SJ->getPointerOperand())
continue;
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
if (isSecondInIteratorPair<Value*>(*J, IPairRange)) {
VPPair VP(P, ValuePair(*I, *J));
ConnectedPairs.insert(VP);
PairConnectionTypes.insert(VPPairWithType(VP, PairConnectionSplat));
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
if (Config.SplatBreaksChain) return;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Look for cases where just the second value in the pair is used by
// both members of another pair (splatting).
for (Value::use_iterator I = P.second->use_begin(),
E = P.second->use_end(); I != E; ++I) {
Fix an error in BBVectorize important for vectorizing pointer types. When vectorizing pointer types it is important to realize that potential pairs cannot be connected via the address pointer argument of a load or store. This is because even after vectorization, the address is still a scalar because the address of the higher half of the pair is implicit from the address of the lower half (it need not be, and should not be, explicitly computed). llvm-svn: 154735
2012-04-14 15:32:50 +08:00
if (isa<LoadInst>(*I))
continue;
else if ((SI = dyn_cast<StoreInst>(*I)) &&
P.second == SI->getPointerOperand())
continue;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
VPIteratorPair IPairRange = CandidatePairs.equal_range(*I);
for (Value::use_iterator J = P.second->use_begin(); J != E; ++J) {
Fix an error in BBVectorize important for vectorizing pointer types. When vectorizing pointer types it is important to realize that potential pairs cannot be connected via the address pointer argument of a load or store. This is because even after vectorization, the address is still a scalar because the address of the higher half of the pair is implicit from the address of the lower half (it need not be, and should not be, explicitly computed). llvm-svn: 154735
2012-04-14 15:32:50 +08:00
if ((SJ = dyn_cast<StoreInst>(*J)) &&
P.second == SJ->getPointerOperand())
continue;
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
if (isSecondInIteratorPair<Value*>(*J, IPairRange)) {
VPPair VP(P, ValuePair(*I, *J));
ConnectedPairs.insert(VP);
PairConnectionTypes.insert(VPPairWithType(VP, PairConnectionSplat));
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
}
// This function figures out which pairs are connected. Two pairs are
// connected if some output of the first pair forms an input to both members
// of the second pair.
void BBVectorize::computeConnectedPairs(
std::multimap<Value *, Value *> &CandidatePairs,
std::vector<Value *> &PairableInsts,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
std::multimap<ValuePair, ValuePair> &ConnectedPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
for (std::vector<Value *>::iterator PI = PairableInsts.begin(),
PE = PairableInsts.end(); PI != PE; ++PI) {
VPIteratorPair choiceRange = CandidatePairs.equal_range(*PI);
for (std::multimap<Value *, Value *>::iterator P = choiceRange.first;
P != choiceRange.second; ++P)
computePairsConnectedTo(CandidatePairs, PairableInsts,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
ConnectedPairs, PairConnectionTypes, *P);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
DEBUG(dbgs() << "BBV: found " << ConnectedPairs.size()
<< " pair connections.\n");
}
// This function builds a set of use tuples such that <A, B> is in the set
// if B is in the use tree of A. If B is in the use tree of A, then B
// depends on the output of A.
void BBVectorize::buildDepMap(
BasicBlock &BB,
std::multimap<Value *, Value *> &CandidatePairs,
std::vector<Value *> &PairableInsts,
DenseSet<ValuePair> &PairableInstUsers) {
DenseSet<Value *> IsInPair;
for (std::multimap<Value *, Value *>::iterator C = CandidatePairs.begin(),
E = CandidatePairs.end(); C != E; ++C) {
IsInPair.insert(C->first);
IsInPair.insert(C->second);
}
BBVectorize: Fix anomalous capital letter in comment llvm-svn: 173579
2013-01-27 00:49:03 +08:00
// Iterate through the basic block, recording all users of each
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// pairable instruction.
BasicBlock::iterator E = BB.end();
for (BasicBlock::iterator I = BB.getFirstInsertionPt(); I != E; ++I) {
if (IsInPair.find(I) == IsInPair.end()) continue;
DenseSet<Value *> Users;
AliasSetTracker WriteSet(*AA);
for (BasicBlock::iterator J = llvm::next(I); J != E; ++J)
(void) trackUsesOfI(Users, WriteSet, I, J);
for (DenseSet<Value *>::iterator U = Users.begin(), E = Users.end();
U != E; ++U)
PairableInstUsers.insert(ValuePair(I, *U));
}
}
// Returns true if an input to pair P is an output of pair Q and also an
// input of pair Q is an output of pair P. If this is the case, then these
// two pairs cannot be simultaneously fused.
bool BBVectorize::pairsConflict(ValuePair P, ValuePair Q,
DenseSet<ValuePair> &PairableInstUsers,
std::multimap<ValuePair, ValuePair> *PairableInstUserMap) {
// Two pairs are in conflict if they are mutual Users of eachother.
bool QUsesP = PairableInstUsers.count(ValuePair(P.first, Q.first)) ||
PairableInstUsers.count(ValuePair(P.first, Q.second)) ||
PairableInstUsers.count(ValuePair(P.second, Q.first)) ||
PairableInstUsers.count(ValuePair(P.second, Q.second));
bool PUsesQ = PairableInstUsers.count(ValuePair(Q.first, P.first)) ||
PairableInstUsers.count(ValuePair(Q.first, P.second)) ||
PairableInstUsers.count(ValuePair(Q.second, P.first)) ||
PairableInstUsers.count(ValuePair(Q.second, P.second));
if (PairableInstUserMap) {
// FIXME: The expensive part of the cycle check is not so much the cycle
// check itself but this edge insertion procedure. This needs some
// profiling and probably a different data structure (same is true of
// most uses of std::multimap).
if (PUsesQ) {
VPPIteratorPair QPairRange = PairableInstUserMap->equal_range(Q);
if (!isSecondInIteratorPair(P, QPairRange))
PairableInstUserMap->insert(VPPair(Q, P));
}
if (QUsesP) {
VPPIteratorPair PPairRange = PairableInstUserMap->equal_range(P);
if (!isSecondInIteratorPair(Q, PPairRange))
PairableInstUserMap->insert(VPPair(P, Q));
}
}
return (QUsesP && PUsesQ);
}
// This function walks the use graph of current pairs to see if, starting
// from P, the walk returns to P.
bool BBVectorize::pairWillFormCycle(ValuePair P,
std::multimap<ValuePair, ValuePair> &PairableInstUserMap,
DenseSet<ValuePair> &CurrentPairs) {
DEBUG(if (DebugCycleCheck)
dbgs() << "BBV: starting cycle check for : " << *P.first << " <-> "
<< *P.second << "\n");
// A lookup table of visisted pairs is kept because the PairableInstUserMap
// contains non-direct associations.
DenseSet<ValuePair> Visited;
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
SmallVector<ValuePair, 32> Q;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// General depth-first post-order traversal:
Q.push_back(P);
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
do {
ValuePair QTop = Q.pop_back_val();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Visited.insert(QTop);
DEBUG(if (DebugCycleCheck)
dbgs() << "BBV: cycle check visiting: " << *QTop.first << " <-> "
<< *QTop.second << "\n");
VPPIteratorPair QPairRange = PairableInstUserMap.equal_range(QTop);
for (std::multimap<ValuePair, ValuePair>::iterator C = QPairRange.first;
C != QPairRange.second; ++C) {
if (C->second == P) {
DEBUG(dbgs()
<< "BBV: rejected to prevent non-trivial cycle formation: "
<< *C->first.first << " <-> " << *C->first.second << "\n");
return true;
}
Simplify contains tests using 'count'. llvm-svn: 149813
2012-02-05 14:35:36 +08:00
if (CurrentPairs.count(C->second) && !Visited.count(C->second))
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Q.push_back(C->second);
}
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
} while (!Q.empty());
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
}
// This function builds the initial tree of connected pairs with the
// pair J at the root.
void BBVectorize::buildInitialTreeFor(
std::multimap<Value *, Value *> &CandidatePairs,
std::vector<Value *> &PairableInsts,
std::multimap<ValuePair, ValuePair> &ConnectedPairs,
DenseSet<ValuePair> &PairableInstUsers,
DenseMap<Value *, Value *> &ChosenPairs,
DenseMap<ValuePair, size_t> &Tree, ValuePair J) {
// Each of these pairs is viewed as the root node of a Tree. The Tree
// is then walked (depth-first). As this happens, we keep track of
// the pairs that compose the Tree and the maximum depth of the Tree.
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
SmallVector<ValuePairWithDepth, 32> Q;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// General depth-first post-order traversal:
Q.push_back(ValuePairWithDepth(J, getDepthFactor(J.first)));
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
do {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
ValuePairWithDepth QTop = Q.back();
// Push each child onto the queue:
bool MoreChildren = false;
size_t MaxChildDepth = QTop.second;
VPPIteratorPair qtRange = ConnectedPairs.equal_range(QTop.first);
BBVectorize.cpp: Try to fix MSVC build. map::iterator and multimap::iterator are incompatible. llvm-svn: 149475
2012-02-01 14:11:58 +08:00
for (std::multimap<ValuePair, ValuePair>::iterator k = qtRange.first;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
k != qtRange.second; ++k) {
// Make sure that this child pair is still a candidate:
bool IsStillCand = false;
VPIteratorPair checkRange =
CandidatePairs.equal_range(k->second.first);
for (std::multimap<Value *, Value *>::iterator m = checkRange.first;
m != checkRange.second; ++m) {
if (m->second == k->second.second) {
IsStillCand = true;
break;
}
}
if (IsStillCand) {
DenseMap<ValuePair, size_t>::iterator C = Tree.find(k->second);
if (C == Tree.end()) {
size_t d = getDepthFactor(k->second.first);
Q.push_back(ValuePairWithDepth(k->second, QTop.second+d));
MoreChildren = true;
} else {
MaxChildDepth = std::max(MaxChildDepth, C->second);
}
}
}
if (!MoreChildren) {
// Record the current pair as part of the Tree:
Tree.insert(ValuePairWithDepth(QTop.first, MaxChildDepth));
Q.pop_back();
}
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
} while (!Q.empty());
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
// Given some initial tree, prune it by removing conflicting pairs (pairs
// that cannot be simultaneously chosen for vectorization).
void BBVectorize::pruneTreeFor(
std::multimap<Value *, Value *> &CandidatePairs,
std::vector<Value *> &PairableInsts,
std::multimap<ValuePair, ValuePair> &ConnectedPairs,
DenseSet<ValuePair> &PairableInstUsers,
std::multimap<ValuePair, ValuePair> &PairableInstUserMap,
DenseMap<Value *, Value *> &ChosenPairs,
DenseMap<ValuePair, size_t> &Tree,
DenseSet<ValuePair> &PrunedTree, ValuePair J,
bool UseCycleCheck) {
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
SmallVector<ValuePairWithDepth, 32> Q;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// General depth-first post-order traversal:
Q.push_back(ValuePairWithDepth(J, getDepthFactor(J.first)));
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
do {
ValuePairWithDepth QTop = Q.pop_back_val();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
PrunedTree.insert(QTop.first);
// Visit each child, pruning as necessary...
Replace std::vector -> SmallVector in BBVectorize For now, this uses 8 on-stack elements. I'll need to do some profiling to see if this is the best number. Pointed out by Jakob in post-commit review. llvm-svn: 167966
2012-11-15 03:53:27 +08:00
SmallVector<ValuePairWithDepth, 8> BestChildren;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
VPPIteratorPair QTopRange = ConnectedPairs.equal_range(QTop.first);
BBVectorize.cpp: Try to fix MSVC build. map::iterator and multimap::iterator are incompatible. llvm-svn: 149475
2012-02-01 14:11:58 +08:00
for (std::multimap<ValuePair, ValuePair>::iterator K = QTopRange.first;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
K != QTopRange.second; ++K) {
DenseMap<ValuePair, size_t>::iterator C = Tree.find(K->second);
if (C == Tree.end()) continue;
// This child is in the Tree, now we need to make sure it is the
// best of any conflicting children. There could be multiple
// conflicting children, so first, determine if we're keeping
// this child, then delete conflicting children as necessary.
// It is also necessary to guard against pairing-induced
// dependencies. Consider instructions a .. x .. y .. b
// such that (a,b) are to be fused and (x,y) are to be fused
// but a is an input to x and b is an output from y. This
// means that y cannot be moved after b but x must be moved
// after b for (a,b) to be fused. In other words, after
// fusing (a,b) we have y .. a/b .. x where y is an input
// to a/b and x is an output to a/b: x and y can no longer
// be legally fused. To prevent this condition, we must
// make sure that a child pair added to the Tree is not
// both an input and output of an already-selected pair.
// Pairing-induced dependencies can also form from more complicated
// cycles. The pair vs. pair conflicts are easy to check, and so
// that is done explicitly for "fast rejection", and because for
// child vs. child conflicts, we may prefer to keep the current
// pair in preference to the already-selected child.
DenseSet<ValuePair> CurrentPairs;
bool CanAdd = true;
Replace std::vector -> SmallVector in BBVectorize For now, this uses 8 on-stack elements. I'll need to do some profiling to see if this is the best number. Pointed out by Jakob in post-commit review. llvm-svn: 167966
2012-11-15 03:53:27 +08:00
for (SmallVector<ValuePairWithDepth, 8>::iterator C2
fix typos llvm-svn: 152035
2012-03-06 01:39:47 +08:00
= BestChildren.begin(), E2 = BestChildren.end();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
C2 != E2; ++C2) {
if (C2->first.first == C->first.first ||
C2->first.first == C->first.second ||
C2->first.second == C->first.first ||
C2->first.second == C->first.second ||
pairsConflict(C2->first, C->first, PairableInstUsers,
UseCycleCheck ? &PairableInstUserMap : 0)) {
if (C2->second >= C->second) {
CanAdd = false;
break;
}
CurrentPairs.insert(C2->first);
}
}
if (!CanAdd) continue;
// Even worse, this child could conflict with another node already
// selected for the Tree. If that is the case, ignore this child.
for (DenseSet<ValuePair>::iterator T = PrunedTree.begin(),
E2 = PrunedTree.end(); T != E2; ++T) {
if (T->first == C->first.first ||
T->first == C->first.second ||
T->second == C->first.first ||
T->second == C->first.second ||
pairsConflict(*T, C->first, PairableInstUsers,
UseCycleCheck ? &PairableInstUserMap : 0)) {
CanAdd = false;
break;
}
CurrentPairs.insert(*T);
}
if (!CanAdd) continue;
// And check the queue too...
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
for (SmallVector<ValuePairWithDepth, 32>::iterator C2 = Q.begin(),
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
E2 = Q.end(); C2 != E2; ++C2) {
if (C2->first.first == C->first.first ||
C2->first.first == C->first.second ||
C2->first.second == C->first.first ||
C2->first.second == C->first.second ||
pairsConflict(C2->first, C->first, PairableInstUsers,
UseCycleCheck ? &PairableInstUserMap : 0)) {
CanAdd = false;
break;
}
CurrentPairs.insert(C2->first);
}
if (!CanAdd) continue;
// Last but not least, check for a conflict with any of the
// already-chosen pairs.
for (DenseMap<Value *, Value *>::iterator C2 =
ChosenPairs.begin(), E2 = ChosenPairs.end();
C2 != E2; ++C2) {
if (pairsConflict(*C2, C->first, PairableInstUsers,
UseCycleCheck ? &PairableInstUserMap : 0)) {
CanAdd = false;
break;
}
CurrentPairs.insert(*C2);
}
if (!CanAdd) continue;
fix indentation llvm-svn: 149857
2012-02-06 13:29:32 +08:00
// To check for non-trivial cycles formed by the addition of the
// current pair we've formed a list of all relevant pairs, now use a
// graph walk to check for a cycle. We start from the current pair and
// walk the use tree to see if we again reach the current pair. If we
// do, then the current pair is rejected.
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// FIXME: It may be more efficient to use a topological-ordering
// algorithm to improve the cycle check. This should be investigated.
if (UseCycleCheck &&
pairWillFormCycle(C->first, PairableInstUserMap, CurrentPairs))
continue;
// This child can be added, but we may have chosen it in preference
// to an already-selected child. Check for this here, and if a
// conflict is found, then remove the previously-selected child
// before adding this one in its place.
Replace std::vector -> SmallVector in BBVectorize For now, this uses 8 on-stack elements. I'll need to do some profiling to see if this is the best number. Pointed out by Jakob in post-commit review. llvm-svn: 167966
2012-11-15 03:53:27 +08:00
for (SmallVector<ValuePairWithDepth, 8>::iterator C2
fix typos llvm-svn: 152035
2012-03-06 01:39:47 +08:00
= BestChildren.begin(); C2 != BestChildren.end();) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (C2->first.first == C->first.first ||
C2->first.first == C->first.second ||
C2->first.second == C->first.first ||
C2->first.second == C->first.second ||
pairsConflict(C2->first, C->first, PairableInstUsers))
Fix the largest offender of determinism in BBVectorize Iterating over the children of each node in the potential vectorization plan must happen in a deterministic order (because it affects which children are erased when two children conflict). There was no need for this data structure to be a map in the first place, so replacing it with a vector is a small change. I believe that this was the last remaining instance if iterating over the elements of a Dense* container where the iteration order could matter. There are some remaining iterations over std::*map containers where the order might matter, but so long as the Value* for instructions in a block increase with the order of the instructions in the block (or decrease) monotonically, then this will appear to be deterministic. llvm-svn: 167942
2012-11-15 02:38:11 +08:00
C2 = BestChildren.erase(C2);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
else
++C2;
}
Fix the largest offender of determinism in BBVectorize Iterating over the children of each node in the potential vectorization plan must happen in a deterministic order (because it affects which children are erased when two children conflict). There was no need for this data structure to be a map in the first place, so replacing it with a vector is a small change. I believe that this was the last remaining instance if iterating over the elements of a Dense* container where the iteration order could matter. There are some remaining iterations over std::*map containers where the order might matter, but so long as the Value* for instructions in a block increase with the order of the instructions in the block (or decrease) monotonically, then this will appear to be deterministic. llvm-svn: 167942
2012-11-15 02:38:11 +08:00
BestChildren.push_back(ValuePairWithDepth(C->first, C->second));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Replace std::vector -> SmallVector in BBVectorize For now, this uses 8 on-stack elements. I'll need to do some profiling to see if this is the best number. Pointed out by Jakob in post-commit review. llvm-svn: 167966
2012-11-15 03:53:27 +08:00
for (SmallVector<ValuePairWithDepth, 8>::iterator C
fix typos llvm-svn: 152035
2012-03-06 01:39:47 +08:00
= BestChildren.begin(), E2 = BestChildren.end();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
C != E2; ++C) {
size_t DepthF = getDepthFactor(C->first.first);
Q.push_back(ValuePairWithDepth(C->first, QTop.second+DepthF));
}
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
} while (!Q.empty());
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
// This function finds the best tree of mututally-compatible connected
// pairs, given the choice of root pairs as an iterator range.
void BBVectorize::findBestTreeFor(
std::multimap<Value *, Value *> &CandidatePairs,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
DenseMap<ValuePair, int> &CandidatePairCostSavings,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
std::vector<Value *> &PairableInsts,
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
DenseSet<ValuePair> &FixedOrderPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
std::multimap<ValuePair, ValuePair> &ConnectedPairs,
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
std::multimap<ValuePair, ValuePair> &ConnectedPairDeps,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DenseSet<ValuePair> &PairableInstUsers,
std::multimap<ValuePair, ValuePair> &PairableInstUserMap,
DenseMap<Value *, Value *> &ChosenPairs,
DenseSet<ValuePair> &BestTree, size_t &BestMaxDepth,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
int &BestEffSize, VPIteratorPair ChoiceRange,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool UseCycleCheck) {
for (std::multimap<Value *, Value *>::iterator J = ChoiceRange.first;
J != ChoiceRange.second; ++J) {
// Before going any further, make sure that this pair does not
// conflict with any already-selected pairs (see comment below
// near the Tree pruning for more details).
DenseSet<ValuePair> ChosenPairSet;
bool DoesConflict = false;
for (DenseMap<Value *, Value *>::iterator C = ChosenPairs.begin(),
E = ChosenPairs.end(); C != E; ++C) {
if (pairsConflict(*C, *J, PairableInstUsers,
UseCycleCheck ? &PairableInstUserMap : 0)) {
DoesConflict = true;
break;
}
ChosenPairSet.insert(*C);
}
if (DoesConflict) continue;
if (UseCycleCheck &&
pairWillFormCycle(*J, PairableInstUserMap, ChosenPairSet))
continue;
DenseMap<ValuePair, size_t> Tree;
buildInitialTreeFor(CandidatePairs, PairableInsts, ConnectedPairs,
PairableInstUsers, ChosenPairs, Tree, *J);
// Because we'll keep the child with the largest depth, the largest
// depth is still the same in the unpruned Tree.
size_t MaxDepth = Tree.lookup(*J);
DEBUG(if (DebugPairSelection) dbgs() << "BBV: found Tree for pair {"
<< *J->first << " <-> " << *J->second << "} of depth " <<
MaxDepth << " and size " << Tree.size() << "\n");
// At this point the Tree has been constructed, but, may contain
// contradictory children (meaning that different children of
// some tree node may be attempting to fuse the same instruction).
// So now we walk the tree again, in the case of a conflict,
// keep only the child with the largest depth. To break a tie,
// favor the first child.
DenseSet<ValuePair> PrunedTree;
pruneTreeFor(CandidatePairs, PairableInsts, ConnectedPairs,
PairableInstUsers, PairableInstUserMap, ChosenPairs, Tree,
PrunedTree, *J, UseCycleCheck);
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
int EffSize = 0;
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
if (TTI) {
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
DenseSet<Value *> PrunedTreeInstrs;
for (DenseSet<ValuePair>::iterator S = PrunedTree.begin(),
E = PrunedTree.end(); S != E; ++S) {
PrunedTreeInstrs.insert(S->first);
PrunedTreeInstrs.insert(S->second);
}
// The set of pairs that have already contributed to the total cost.
DenseSet<ValuePair> IncomingPairs;
BBVectorize: Don't vectorize vector-manipulation chains Don't choose a vectorization plan containing only shuffles and vector inserts/extracts. Due to inperfections in the cost model, these can lead to infinite recusion. llvm-svn: 167811
2012-11-13 11:12:40 +08:00
// If the cost model were perfect, this might not be necessary; but we
// need to make sure that we don't get stuck vectorizing our own
// shuffle chains.
bool HasNontrivialInsts = false;
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
// The node weights represent the cost savings associated with
// fusing the pair of instructions.
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
for (DenseSet<ValuePair>::iterator S = PrunedTree.begin(),
E = PrunedTree.end(); S != E; ++S) {
BBVectorize: Don't vectorize vector-manipulation chains Don't choose a vectorization plan containing only shuffles and vector inserts/extracts. Due to inperfections in the cost model, these can lead to infinite recusion. llvm-svn: 167811
2012-11-13 11:12:40 +08:00
if (!isa<ShuffleVectorInst>(S->first) &&
!isa<InsertElementInst>(S->first) &&
!isa<ExtractElementInst>(S->first))
HasNontrivialInsts = true;
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
bool FlipOrder = false;
if (getDepthFactor(S->first)) {
int ESContrib = CandidatePairCostSavings.find(*S)->second;
DEBUG(if (DebugPairSelection) dbgs() << "\tweight {"
<< *S->first << " <-> " << *S->second << "} = " <<
ESContrib << "\n");
EffSize += ESContrib;
}
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
// The edge weights contribute in a negative sense: they represent
// the cost of shuffles.
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
VPPIteratorPair IP = ConnectedPairDeps.equal_range(*S);
if (IP.first != ConnectedPairDeps.end()) {
unsigned NumDepsDirect = 0, NumDepsSwap = 0;
for (std::multimap<ValuePair, ValuePair>::iterator Q = IP.first;
Q != IP.second; ++Q) {
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
if (!PrunedTree.count(Q->second))
continue;
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
DenseMap<VPPair, unsigned>::iterator R =
PairConnectionTypes.find(VPPair(Q->second, Q->first));
assert(R != PairConnectionTypes.end() &&
"Cannot find pair connection type");
if (R->second == PairConnectionDirect)
++NumDepsDirect;
else if (R->second == PairConnectionSwap)
++NumDepsSwap;
}
// If there are more swaps than direct connections, then
// the pair order will be flipped during fusion. So the real
// number of swaps is the minimum number.
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
FlipOrder = !FixedOrderPairs.count(*S) &&
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
((NumDepsSwap > NumDepsDirect) ||
FixedOrderPairs.count(ValuePair(S->second, S->first)));
for (std::multimap<ValuePair, ValuePair>::iterator Q = IP.first;
Q != IP.second; ++Q) {
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
if (!PrunedTree.count(Q->second))
continue;
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
DenseMap<VPPair, unsigned>::iterator R =
PairConnectionTypes.find(VPPair(Q->second, Q->first));
assert(R != PairConnectionTypes.end() &&
"Cannot find pair connection type");
Type *Ty1 = Q->second.first->getType(),
*Ty2 = Q->second.second->getType();
Type *VTy = getVecTypeForPair(Ty1, Ty2);
if ((R->second == PairConnectionDirect && FlipOrder) ||
(R->second == PairConnectionSwap && !FlipOrder) ||
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
R->second == PairConnectionSplat) {
int ESContrib = (int) getInstrCost(Instruction::ShuffleVector,
VTy, VTy);
DEBUG(if (DebugPairSelection) dbgs() << "\tcost {" <<
*Q->second.first << " <-> " << *Q->second.second <<
"} -> {" <<
*S->first << " <-> " << *S->second << "} = " <<
ESContrib << "\n");
EffSize -= ESContrib;
}
}
}
// Compute the cost of outgoing edges. We assume that edges outgoing
// to shuffles, inserts or extracts can be merged, and so contribute
// no additional cost.
if (!S->first->getType()->isVoidTy()) {
Type *Ty1 = S->first->getType(),
*Ty2 = S->second->getType();
Type *VTy = getVecTypeForPair(Ty1, Ty2);
bool NeedsExtraction = false;
for (Value::use_iterator I = S->first->use_begin(),
IE = S->first->use_end(); I != IE; ++I) {
BBVectorize: Use a more sophisticated check for input cost The old checking code, which assumed that input shuffles and insert-elements could always be folded (and thus were free) is too simple. This can only happen in special circumstances. Using the simple check caused infinite recursion. llvm-svn: 167750
2012-11-13 05:21:02 +08:00
if (ShuffleVectorInst *SI = dyn_cast<ShuffleVectorInst>(*I)) {
// Shuffle can be folded if it has no other input
if (isa<UndefValue>(SI->getOperand(1)))
continue;
}
if (isa<ExtractElementInst>(*I))
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
continue;
if (PrunedTreeInstrs.count(*I))
continue;
NeedsExtraction = true;
break;
}
if (NeedsExtraction) {
int ESContrib;
if (Ty1->isVectorTy())
ESContrib = (int) getInstrCost(Instruction::ShuffleVector,
Ty1, VTy);
else
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
ESContrib = (int) TTI->getVectorInstrCost(
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
Instruction::ExtractElement, VTy, 0);
DEBUG(if (DebugPairSelection) dbgs() << "\tcost {" <<
*S->first << "} = " << ESContrib << "\n");
EffSize -= ESContrib;
}
NeedsExtraction = false;
for (Value::use_iterator I = S->second->use_begin(),
IE = S->second->use_end(); I != IE; ++I) {
BBVectorize: Use a more sophisticated check for input cost The old checking code, which assumed that input shuffles and insert-elements could always be folded (and thus were free) is too simple. This can only happen in special circumstances. Using the simple check caused infinite recursion. llvm-svn: 167750
2012-11-13 05:21:02 +08:00
if (ShuffleVectorInst *SI = dyn_cast<ShuffleVectorInst>(*I)) {
// Shuffle can be folded if it has no other input
if (isa<UndefValue>(SI->getOperand(1)))
continue;
}
if (isa<ExtractElementInst>(*I))
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
continue;
if (PrunedTreeInstrs.count(*I))
continue;
NeedsExtraction = true;
break;
}
if (NeedsExtraction) {
int ESContrib;
if (Ty2->isVectorTy())
ESContrib = (int) getInstrCost(Instruction::ShuffleVector,
Ty2, VTy);
else
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
ESContrib = (int) TTI->getVectorInstrCost(
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
Instruction::ExtractElement, VTy, 1);
DEBUG(if (DebugPairSelection) dbgs() << "\tcost {" <<
*S->second << "} = " << ESContrib << "\n");
EffSize -= ESContrib;
}
}
// Compute the cost of incoming edges.
if (!isa<LoadInst>(S->first) && !isa<StoreInst>(S->first)) {
Instruction *S1 = cast<Instruction>(S->first),
*S2 = cast<Instruction>(S->second);
for (unsigned o = 0; o < S1->getNumOperands(); ++o) {
Value *O1 = S1->getOperand(o), *O2 = S2->getOperand(o);
// Combining constants into vector constants (or small vector
// constants into larger ones are assumed free).
if (isa<Constant>(O1) && isa<Constant>(O2))
continue;
if (FlipOrder)
std::swap(O1, O2);
ValuePair VP = ValuePair(O1, O2);
ValuePair VPR = ValuePair(O2, O1);
// Internal edges are not handled here.
if (PrunedTree.count(VP) || PrunedTree.count(VPR))
continue;
Type *Ty1 = O1->getType(),
*Ty2 = O2->getType();
Type *VTy = getVecTypeForPair(Ty1, Ty2);
// Combining vector operations of the same type is also assumed
// folded with other operations.
BBVectorize: Use a more sophisticated check for input cost The old checking code, which assumed that input shuffles and insert-elements could always be folded (and thus were free) is too simple. This can only happen in special circumstances. Using the simple check caused infinite recursion. llvm-svn: 167750
2012-11-13 05:21:02 +08:00
if (Ty1 == Ty2) {
// If both are insert elements, then both can be widened.
BBVectorize: Only some insert element operand pairs are free. This fixes another infinite recursion case when using target costs. We can only replace insert element input chains that are pure (end with inserting into an undef). llvm-svn: 167784
2012-11-13 07:55:36 +08:00
InsertElementInst *IEO1 = dyn_cast<InsertElementInst>(O1),
*IEO2 = dyn_cast<InsertElementInst>(O2);
if (IEO1 && IEO2 && isPureIEChain(IEO1) && isPureIEChain(IEO2))
BBVectorize: Use a more sophisticated check for input cost The old checking code, which assumed that input shuffles and insert-elements could always be folded (and thus were free) is too simple. This can only happen in special circumstances. Using the simple check caused infinite recursion. llvm-svn: 167750
2012-11-13 05:21:02 +08:00
continue;
// If both are extract elements, and both have the same input
// type, then they can be replaced with a shuffle
ExtractElementInst *EIO1 = dyn_cast<ExtractElementInst>(O1),
*EIO2 = dyn_cast<ExtractElementInst>(O2);
if (EIO1 && EIO2 &&
EIO1->getOperand(0)->getType() ==
EIO2->getOperand(0)->getType())
continue;
// If both are a shuffle with equal operand types and only two
// unqiue operands, then they can be replaced with a single
// shuffle
ShuffleVectorInst *SIO1 = dyn_cast<ShuffleVectorInst>(O1),
*SIO2 = dyn_cast<ShuffleVectorInst>(O2);
if (SIO1 && SIO2 &&
SIO1->getOperand(0)->getType() ==
SIO2->getOperand(0)->getType()) {
SmallSet<Value *, 4> SIOps;
SIOps.insert(SIO1->getOperand(0));
SIOps.insert(SIO1->getOperand(1));
SIOps.insert(SIO2->getOperand(0));
SIOps.insert(SIO2->getOperand(1));
if (SIOps.size() <= 2)
continue;
}
}
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
int ESContrib;
// This pair has already been formed.
if (IncomingPairs.count(VP)) {
continue;
} else if (IncomingPairs.count(VPR)) {
ESContrib = (int) getInstrCost(Instruction::ShuffleVector,
VTy, VTy);
} else if (!Ty1->isVectorTy() && !Ty2->isVectorTy()) {
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
ESContrib = (int) TTI->getVectorInstrCost(
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
Instruction::InsertElement, VTy, 0);
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
ESContrib += (int) TTI->getVectorInstrCost(
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
Instruction::InsertElement, VTy, 1);
} else if (!Ty1->isVectorTy()) {
// O1 needs to be inserted into a vector of size O2, and then
// both need to be shuffled together.
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
ESContrib = (int) TTI->getVectorInstrCost(
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
Instruction::InsertElement, Ty2, 0);
ESContrib += (int) getInstrCost(Instruction::ShuffleVector,
VTy, Ty2);
} else if (!Ty2->isVectorTy()) {
// O2 needs to be inserted into a vector of size O1, and then
// both need to be shuffled together.
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
ESContrib = (int) TTI->getVectorInstrCost(
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
Instruction::InsertElement, Ty1, 0);
ESContrib += (int) getInstrCost(Instruction::ShuffleVector,
VTy, Ty1);
} else {
Type *TyBig = Ty1, *TySmall = Ty2;
if (Ty2->getVectorNumElements() > Ty1->getVectorNumElements())
std::swap(TyBig, TySmall);
ESContrib = (int) getInstrCost(Instruction::ShuffleVector,
VTy, TyBig);
if (TyBig != TySmall)
ESContrib += (int) getInstrCost(Instruction::ShuffleVector,
TyBig, TySmall);
}
DEBUG(if (DebugPairSelection) dbgs() << "\tcost {"
<< *O1 << " <-> " << *O2 << "} = " <<
ESContrib << "\n");
EffSize -= ESContrib;
IncomingPairs.insert(VP);
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
}
}
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
}
BBVectorize: Don't vectorize vector-manipulation chains Don't choose a vectorization plan containing only shuffles and vector inserts/extracts. Due to inperfections in the cost model, these can lead to infinite recusion. llvm-svn: 167811
2012-11-13 11:12:40 +08:00
if (!HasNontrivialInsts) {
DEBUG(if (DebugPairSelection) dbgs() <<
"\tNo non-trivial instructions in tree;"
" override to zero effective size\n");
EffSize = 0;
}
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
} else {
for (DenseSet<ValuePair>::iterator S = PrunedTree.begin(),
E = PrunedTree.end(); S != E; ++S)
EffSize += (int) getDepthFactor(S->first);
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DEBUG(if (DebugPairSelection)
dbgs() << "BBV: found pruned Tree for pair {"
<< *J->first << " <-> " << *J->second << "} of depth " <<
MaxDepth << " and size " << PrunedTree.size() <<
" (effective size: " << EffSize << ")\n");
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
if (((TTI && !UseChainDepthWithTI) ||
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
MaxDepth >= Config.ReqChainDepth) &&
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
EffSize > 0 && EffSize > BestEffSize) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BestMaxDepth = MaxDepth;
BestEffSize = EffSize;
BestTree = PrunedTree;
}
}
}
// Given the list of candidate pairs, this function selects those
// that will be fused into vector instructions.
void BBVectorize::choosePairs(
std::multimap<Value *, Value *> &CandidatePairs,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
DenseMap<ValuePair, int> &CandidatePairCostSavings,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
std::vector<Value *> &PairableInsts,
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
DenseSet<ValuePair> &FixedOrderPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
std::multimap<ValuePair, ValuePair> &ConnectedPairs,
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
std::multimap<ValuePair, ValuePair> &ConnectedPairDeps,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DenseSet<ValuePair> &PairableInstUsers,
DenseMap<Value *, Value *>& ChosenPairs) {
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
bool UseCycleCheck =
CandidatePairs.size() <= Config.MaxCandPairsForCycleCheck;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
std::multimap<ValuePair, ValuePair> PairableInstUserMap;
for (std::vector<Value *>::iterator I = PairableInsts.begin(),
E = PairableInsts.end(); I != E; ++I) {
// The number of possible pairings for this variable:
size_t NumChoices = CandidatePairs.count(*I);
if (!NumChoices) continue;
VPIteratorPair ChoiceRange = CandidatePairs.equal_range(*I);
// The best pair to choose and its tree:
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
size_t BestMaxDepth = 0;
int BestEffSize = 0;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DenseSet<ValuePair> BestTree;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
findBestTreeFor(CandidatePairs, CandidatePairCostSavings,
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
PairableInsts, FixedOrderPairs, PairConnectionTypes,
ConnectedPairs, ConnectedPairDeps,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
PairableInstUsers, PairableInstUserMap, ChosenPairs,
BestTree, BestMaxDepth, BestEffSize, ChoiceRange,
UseCycleCheck);
// A tree has been chosen (or not) at this point. If no tree was
// chosen, then this instruction, I, cannot be paired (and is no longer
// considered).
DEBUG(if (BestTree.size() > 0)
dbgs() << "BBV: selected pairs in the best tree for: "
<< *cast<Instruction>(*I) << "\n");
for (DenseSet<ValuePair>::iterator S = BestTree.begin(),
SE2 = BestTree.end(); S != SE2; ++S) {
// Insert the members of this tree into the list of chosen pairs.
ChosenPairs.insert(ValuePair(S->first, S->second));
DEBUG(dbgs() << "BBV: selected pair: " << *S->first << " <-> " <<
*S->second << "\n");
// Remove all candidate pairs that have values in the chosen tree.
for (std::multimap<Value *, Value *>::iterator K =
CandidatePairs.begin(); K != CandidatePairs.end();) {
if (K->first == S->first || K->second == S->first ||
K->second == S->second || K->first == S->second) {
// Don't remove the actual pair chosen so that it can be used
// in subsequent tree selections.
if (!(K->first == S->first && K->second == S->second))
CandidatePairs.erase(K++);
else
++K;
} else {
++K;
}
}
}
}
DEBUG(dbgs() << "BBV: selected " << ChosenPairs.size() << " pairs.\n");
}
std::string getReplacementName(Instruction *I, bool IsInput, unsigned o,
unsigned n = 0) {
if (!I->hasName())
return "";
return (I->getName() + (IsInput ? ".v.i" : ".v.r") + utostr(o) +
(n > 0 ? "." + utostr(n) : "")).str();
}
// Returns the value that is to be used as the pointer input to the vector
// instruction that fuses I with J.
Value *BBVectorize::getReplacementPointerInput(LLVMContext& Context,
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
Instruction *I, Instruction *J, unsigned o) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Value *IPtr, *JPtr;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
unsigned IAlignment, JAlignment, IAddressSpace, JAddressSpace;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
int64_t OffsetInElmts;
Precompute SCEV pointer analysis prior to instruction fusion in BBVectorize. When both a load/store and its address computation are being vectorized, it can happen that the address-computation vectorization destroys SCEV's ability to analyize the relative pointer offsets. As a result (like with the aliasing analysis info), we need to precompute the necessary information prior to instruction fusing. This was found during stress testing (running through the test suite with a very low required chain length); unfortunately, I don't have a small test case. llvm-svn: 159332
2012-06-28 13:42:45 +08:00
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
// Note: the analysis might fail here, that is why the pair order has
Precompute SCEV pointer analysis prior to instruction fusion in BBVectorize. When both a load/store and its address computation are being vectorized, it can happen that the address-computation vectorization destroys SCEV's ability to analyize the relative pointer offsets. As a result (like with the aliasing analysis info), we need to precompute the necessary information prior to instruction fusing. This was found during stress testing (running through the test suite with a very low required chain length); unfortunately, I don't have a small test case. llvm-svn: 159332
2012-06-28 13:42:45 +08:00
// been precomputed (OffsetInElmts must be unused here).
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
(void) getPairPtrInfo(I, J, IPtr, JPtr, IAlignment, JAlignment,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
IAddressSpace, JAddressSpace,
BBVectorize: Don't make calls to SE when the result is unused. SE was being called during the instruction-fusion process (when the result is unreliable, and thus ignored). No functionality change is intended. llvm-svn: 167037
2012-10-31 02:55:49 +08:00
OffsetInElmts, false);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// The pointer value is taken to be the one with the lowest offset.
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
Value *VPtr = IPtr;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *ArgTypeI = cast<PointerType>(IPtr->getType())->getElementType();
Type *ArgTypeJ = cast<PointerType>(JPtr->getType())->getElementType();
Type *VArgType = getVecTypeForPair(ArgTypeI, ArgTypeJ);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Type *VArgPtrType = PointerType::get(VArgType,
cast<PointerType>(IPtr->getType())->getAddressSpace());
return new BitCastInst(VPtr, VArgPtrType, getReplacementName(I, true, o),
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
/* insert before */ I);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
void BBVectorize::fillNewShuffleMask(LLVMContext& Context, Instruction *J,
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
unsigned MaskOffset, unsigned NumInElem,
unsigned NumInElem1, unsigned IdxOffset,
std::vector<Constant*> &Mask) {
unsigned NumElem1 = cast<VectorType>(J->getType())->getNumElements();
for (unsigned v = 0; v < NumElem1; ++v) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
int m = cast<ShuffleVectorInst>(J)->getMaskValue(v);
if (m < 0) {
Mask[v+MaskOffset] = UndefValue::get(Type::getInt32Ty(Context));
} else {
unsigned mm = m + (int) IdxOffset;
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
if (m >= (int) NumInElem1)
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
mm += (int) NumInElem;
Mask[v+MaskOffset] =
ConstantInt::get(Type::getInt32Ty(Context), mm);
}
}
}
// Returns the value that is to be used as the vector-shuffle mask to the
// vector instruction that fuses I with J.
Value *BBVectorize::getReplacementShuffleMask(LLVMContext& Context,
Instruction *I, Instruction *J) {
// This is the shuffle mask. We need to append the second
// mask to the first, and the numbers need to be adjusted.
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *ArgTypeI = I->getType();
Type *ArgTypeJ = J->getType();
Type *VArgType = getVecTypeForPair(ArgTypeI, ArgTypeJ);
unsigned NumElemI = cast<VectorType>(ArgTypeI)->getNumElements();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Get the total number of elements in the fused vector type.
// By definition, this must equal the number of elements in
// the final mask.
unsigned NumElem = cast<VectorType>(VArgType)->getNumElements();
std::vector<Constant*> Mask(NumElem);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *OpTypeI = I->getOperand(0)->getType();
unsigned NumInElemI = cast<VectorType>(OpTypeI)->getNumElements();
Type *OpTypeJ = J->getOperand(0)->getType();
unsigned NumInElemJ = cast<VectorType>(OpTypeJ)->getNumElements();
// The fused vector will be:
// -----------------------------------------------------
// | NumInElemI | NumInElemJ | NumInElemI | NumInElemJ |
// -----------------------------------------------------
// from which we'll extract NumElem total elements (where the first NumElemI
// of them come from the mask in I and the remainder come from the mask
// in J.
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// For the mask from the first pair...
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
fillNewShuffleMask(Context, I, 0, NumInElemJ, NumInElemI,
0, Mask);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// For the mask from the second pair...
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
fillNewShuffleMask(Context, J, NumElemI, NumInElemI, NumInElemJ,
NumInElemI, Mask);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return ConstantVector::get(Mask);
}
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
bool BBVectorize::expandIEChain(LLVMContext& Context, Instruction *I,
Instruction *J, unsigned o, Value *&LOp,
unsigned numElemL,
Type *ArgTypeL, Type *ArgTypeH,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
bool IBeforeJ, unsigned IdxOff) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
bool ExpandedIEChain = false;
if (InsertElementInst *LIE = dyn_cast<InsertElementInst>(LOp)) {
// If we have a pure insertelement chain, then this can be rewritten
// into a chain that directly builds the larger type.
BBVectorize: Only some insert element operand pairs are free. This fixes another infinite recursion case when using target costs. We can only replace insert element input chains that are pure (end with inserting into an undef). llvm-svn: 167784
2012-11-13 07:55:36 +08:00
if (isPureIEChain(LIE)) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
SmallVector<Value *, 8> VectElemts(numElemL,
UndefValue::get(ArgTypeL->getScalarType()));
InsertElementInst *LIENext = LIE;
do {
unsigned Idx =
cast<ConstantInt>(LIENext->getOperand(2))->getSExtValue();
VectElemts[Idx] = LIENext->getOperand(1);
} while ((LIENext =
dyn_cast<InsertElementInst>(LIENext->getOperand(0))));
LIENext = 0;
Value *LIEPrev = UndefValue::get(ArgTypeH);
for (unsigned i = 0; i < numElemL; ++i) {
if (isa<UndefValue>(VectElemts[i])) continue;
LIENext = InsertElementInst::Create(LIEPrev, VectElemts[i],
ConstantInt::get(Type::getInt32Ty(Context),
i + IdxOff),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, i+1));
LIENext->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
LIEPrev = LIENext;
}
LOp = LIENext ? (Value*) LIENext : UndefValue::get(ArgTypeH);
ExpandedIEChain = true;
}
}
return ExpandedIEChain;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Returns the value to be used as the specified operand of the vector
// instruction that fuses I with J.
Value *BBVectorize::getReplacementInput(LLVMContext& Context, Instruction *I,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
Instruction *J, unsigned o, bool IBeforeJ) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Value *CV0 = ConstantInt::get(Type::getInt32Ty(Context), 0);
Value *CV1 = ConstantInt::get(Type::getInt32Ty(Context), 1);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
// Compute the fused vector type for this operand
Type *ArgTypeI = I->getOperand(o)->getType();
Type *ArgTypeJ = J->getOperand(o)->getType();
VectorType *VArgType = getVecTypeForPair(ArgTypeI, ArgTypeJ);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Instruction *L = I, *H = J;
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *ArgTypeL = ArgTypeI, *ArgTypeH = ArgTypeJ;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
unsigned numElemL;
if (ArgTypeL->isVectorTy())
numElemL = cast<VectorType>(ArgTypeL)->getNumElements();
else
numElemL = 1;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
unsigned numElemH;
if (ArgTypeH->isVectorTy())
numElemH = cast<VectorType>(ArgTypeH)->getNumElements();
else
numElemH = 1;
Value *LOp = L->getOperand(o);
Value *HOp = H->getOperand(o);
unsigned numElem = VArgType->getNumElements();
// First, we check if we can reuse the "original" vector outputs (if these
// exist). We might need a shuffle.
ExtractElementInst *LEE = dyn_cast<ExtractElementInst>(LOp);
ExtractElementInst *HEE = dyn_cast<ExtractElementInst>(HOp);
ShuffleVectorInst *LSV = dyn_cast<ShuffleVectorInst>(LOp);
ShuffleVectorInst *HSV = dyn_cast<ShuffleVectorInst>(HOp);
// FIXME: If we're fusing shuffle instructions, then we can't apply this
// optimization. The input vectors to the shuffle might be a different
// length from the shuffle outputs. Unfortunately, the replacement
// shuffle mask has already been formed, and the mask entries are sensitive
// to the sizes of the inputs.
bool IsSizeChangeShuffle =
isa<ShuffleVectorInst>(L) &&
(LOp->getType() != L->getType() || HOp->getType() != H->getType());
if ((LEE || LSV) && (HEE || HSV) && !IsSizeChangeShuffle) {
// We can have at most two unique vector inputs.
bool CanUseInputs = true;
Value *I1, *I2 = 0;
if (LEE) {
I1 = LEE->getOperand(0);
} else {
I1 = LSV->getOperand(0);
I2 = LSV->getOperand(1);
if (I2 == I1 || isa<UndefValue>(I2))
I2 = 0;
}
if (HEE) {
Value *I3 = HEE->getOperand(0);
if (!I2 && I3 != I1)
I2 = I3;
else if (I3 != I1 && I3 != I2)
CanUseInputs = false;
} else {
Value *I3 = HSV->getOperand(0);
if (!I2 && I3 != I1)
I2 = I3;
else if (I3 != I1 && I3 != I2)
CanUseInputs = false;
if (CanUseInputs) {
Value *I4 = HSV->getOperand(1);
if (!isa<UndefValue>(I4)) {
if (!I2 && I4 != I1)
I2 = I4;
else if (I4 != I1 && I4 != I2)
CanUseInputs = false;
}
}
}
if (CanUseInputs) {
unsigned LOpElem =
cast<VectorType>(cast<Instruction>(LOp)->getOperand(0)->getType())
->getNumElements();
unsigned HOpElem =
cast<VectorType>(cast<Instruction>(HOp)->getOperand(0)->getType())
->getNumElements();
// We have one or two input vectors. We need to map each index of the
// operands to the index of the original vector.
SmallVector<std::pair<int, int>, 8> II(numElem);
for (unsigned i = 0; i < numElemL; ++i) {
int Idx, INum;
if (LEE) {
Idx =
cast<ConstantInt>(LEE->getOperand(1))->getSExtValue();
INum = LEE->getOperand(0) == I1 ? 0 : 1;
} else {
Idx = LSV->getMaskValue(i);
if (Idx < (int) LOpElem) {
INum = LSV->getOperand(0) == I1 ? 0 : 1;
} else {
Idx -= LOpElem;
INum = LSV->getOperand(1) == I1 ? 0 : 1;
}
}
II[i] = std::pair<int, int>(Idx, INum);
}
for (unsigned i = 0; i < numElemH; ++i) {
int Idx, INum;
if (HEE) {
Idx =
cast<ConstantInt>(HEE->getOperand(1))->getSExtValue();
INum = HEE->getOperand(0) == I1 ? 0 : 1;
} else {
Idx = HSV->getMaskValue(i);
if (Idx < (int) HOpElem) {
INum = HSV->getOperand(0) == I1 ? 0 : 1;
} else {
Idx -= HOpElem;
INum = HSV->getOperand(1) == I1 ? 0 : 1;
}
}
II[i + numElemL] = std::pair<int, int>(Idx, INum);
}
// We now have an array which tells us from which index of which
// input vector each element of the operand comes.
VectorType *I1T = cast<VectorType>(I1->getType());
unsigned I1Elem = I1T->getNumElements();
if (!I2) {
// In this case there is only one underlying vector input. Check for
// the trivial case where we can use the input directly.
if (I1Elem == numElem) {
bool ElemInOrder = true;
for (unsigned i = 0; i < numElem; ++i) {
if (II[i].first != (int) i && II[i].first != -1) {
ElemInOrder = false;
break;
}
}
if (ElemInOrder)
return I1;
}
// A shuffle is needed.
std::vector<Constant *> Mask(numElem);
for (unsigned i = 0; i < numElem; ++i) {
int Idx = II[i].first;
if (Idx == -1)
Mask[i] = UndefValue::get(Type::getInt32Ty(Context));
else
Mask[i] = ConstantInt::get(Type::getInt32Ty(Context), Idx);
}
Instruction *S =
new ShuffleVectorInst(I1, UndefValue::get(I1T),
ConstantVector::get(Mask),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o));
S->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
return S;
}
VectorType *I2T = cast<VectorType>(I2->getType());
unsigned I2Elem = I2T->getNumElements();
// This input comes from two distinct vectors. The first step is to
// make sure that both vectors are the same length. If not, the
// smaller one will need to grow before they can be shuffled together.
if (I1Elem < I2Elem) {
std::vector<Constant *> Mask(I2Elem);
unsigned v = 0;
for (; v < I1Elem; ++v)
Mask[v] = ConstantInt::get(Type::getInt32Ty(Context), v);
for (; v < I2Elem; ++v)
Mask[v] = UndefValue::get(Type::getInt32Ty(Context));
Instruction *NewI1 =
new ShuffleVectorInst(I1, UndefValue::get(I1T),
ConstantVector::get(Mask),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 1));
NewI1->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
I1 = NewI1;
I1T = I2T;
I1Elem = I2Elem;
} else if (I1Elem > I2Elem) {
std::vector<Constant *> Mask(I1Elem);
unsigned v = 0;
for (; v < I2Elem; ++v)
Mask[v] = ConstantInt::get(Type::getInt32Ty(Context), v);
for (; v < I1Elem; ++v)
Mask[v] = UndefValue::get(Type::getInt32Ty(Context));
Instruction *NewI2 =
new ShuffleVectorInst(I2, UndefValue::get(I2T),
ConstantVector::get(Mask),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 1));
NewI2->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
I2 = NewI2;
I2T = I1T;
I2Elem = I1Elem;
}
// Now that both I1 and I2 are the same length we can shuffle them
// together (and use the result).
std::vector<Constant *> Mask(numElem);
for (unsigned v = 0; v < numElem; ++v) {
if (II[v].first == -1) {
Mask[v] = UndefValue::get(Type::getInt32Ty(Context));
} else {
int Idx = II[v].first + II[v].second * I1Elem;
Mask[v] = ConstantInt::get(Type::getInt32Ty(Context), Idx);
}
}
Instruction *NewOp =
new ShuffleVectorInst(I1, I2, ConstantVector::get(Mask),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J, true, o));
NewOp->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
return NewOp;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *ArgType = ArgTypeL;
if (numElemL < numElemH) {
if (numElemL == 1 && expandIEChain(Context, I, J, o, HOp, numElemH,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
ArgTypeL, VArgType, IBeforeJ, 1)) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
// This is another short-circuit case: we're combining a scalar into
// a vector that is formed by an IE chain. We've just expanded the IE
// chain, now insert the scalar and we're done.
Instruction *S = InsertElementInst::Create(HOp, LOp, CV0,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J, true, o));
S->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
return S;
} else if (!expandIEChain(Context, I, J, o, LOp, numElemL, ArgTypeL,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
ArgTypeH, IBeforeJ)) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
// The two vector inputs to the shuffle must be the same length,
// so extend the smaller vector to be the same length as the larger one.
Instruction *NLOp;
if (numElemL > 1) {
std::vector<Constant *> Mask(numElemH);
unsigned v = 0;
for (; v < numElemL; ++v)
Mask[v] = ConstantInt::get(Type::getInt32Ty(Context), v);
for (; v < numElemH; ++v)
Mask[v] = UndefValue::get(Type::getInt32Ty(Context));
NLOp = new ShuffleVectorInst(LOp, UndefValue::get(ArgTypeL),
ConstantVector::get(Mask),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 1));
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
} else {
NLOp = InsertElementInst::Create(UndefValue::get(ArgTypeH), LOp, CV0,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 1));
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
}
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
NLOp->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
LOp = NLOp;
}
ArgType = ArgTypeH;
} else if (numElemL > numElemH) {
if (numElemH == 1 && expandIEChain(Context, I, J, o, LOp, numElemL,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
ArgTypeH, VArgType, IBeforeJ)) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Instruction *S =
InsertElementInst::Create(LOp, HOp,
ConstantInt::get(Type::getInt32Ty(Context),
numElemL),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o));
S->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
return S;
} else if (!expandIEChain(Context, I, J, o, HOp, numElemH, ArgTypeH,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
ArgTypeL, IBeforeJ)) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Instruction *NHOp;
if (numElemH > 1) {
std::vector<Constant *> Mask(numElemL);
unsigned v = 0;
for (; v < numElemH; ++v)
Mask[v] = ConstantInt::get(Type::getInt32Ty(Context), v);
for (; v < numElemL; ++v)
Mask[v] = UndefValue::get(Type::getInt32Ty(Context));
NHOp = new ShuffleVectorInst(HOp, UndefValue::get(ArgTypeH),
ConstantVector::get(Mask),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 1));
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
} else {
NHOp = InsertElementInst::Create(UndefValue::get(ArgTypeL), HOp, CV0,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 1));
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
}
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
NHOp->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
HOp = NHOp;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
if (ArgType->isVectorTy()) {
unsigned numElem = cast<VectorType>(VArgType)->getNumElements();
std::vector<Constant*> Mask(numElem);
for (unsigned v = 0; v < numElem; ++v) {
unsigned Idx = v;
// If the low vector was expanded, we need to skip the extra
// undefined entries.
if (v >= numElemL && numElemH > numElemL)
Idx += (numElemH - numElemL);
Mask[v] = ConstantInt::get(Type::getInt32Ty(Context), Idx);
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Instruction *BV = new ShuffleVectorInst(LOp, HOp,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
ConstantVector::get(Mask),
getReplacementName(IBeforeJ ? I : J, true, o));
BV->insertBefore(IBeforeJ ? J : I);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return BV;
}
Instruction *BV1 = InsertElementInst::Create(
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
UndefValue::get(VArgType), LOp, CV0,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 1));
BV1->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Instruction *BV2 = InsertElementInst::Create(BV1, HOp, CV1,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 2));
BV2->insertBefore(IBeforeJ ? J : I);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return BV2;
}
// This function creates an array of values that will be used as the inputs
// to the vector instruction that fuses I with J.
void BBVectorize::getReplacementInputsForPair(LLVMContext& Context,
Instruction *I, Instruction *J,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
SmallVector<Value *, 3> &ReplacedOperands,
bool IBeforeJ) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
unsigned NumOperands = I->getNumOperands();
for (unsigned p = 0, o = NumOperands-1; p < NumOperands; ++p, --o) {
// Iterate backward so that we look at the store pointer
// first and know whether or not we need to flip the inputs.
if (isa<LoadInst>(I) || (o == 1 && isa<StoreInst>(I))) {
// This is the pointer for a load/store instruction.
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
ReplacedOperands[o] = getReplacementPointerInput(Context, I, J, o);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
continue;
Correctly vectorize powi. The powi intrinsic requires special handling because it always takes a single integer power regardless of the result type. As a result, we can vectorize only if the powers are equal. Fixes PR12364. llvm-svn: 153797
2012-03-31 11:38:40 +08:00
} else if (isa<CallInst>(I)) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Function *F = cast<CallInst>(I)->getCalledFunction();
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
Intrinsic::ID IID = (Intrinsic::ID) F->getIntrinsicID();
Correctly vectorize powi. The powi intrinsic requires special handling because it always takes a single integer power regardless of the result type. As a result, we can vectorize only if the powers are equal. Fixes PR12364. llvm-svn: 153797
2012-03-31 11:38:40 +08:00
if (o == NumOperands-1) {
BasicBlock &BB = *I->getParent();
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
Correctly vectorize powi. The powi intrinsic requires special handling because it always takes a single integer power regardless of the result type. As a result, we can vectorize only if the powers are equal. Fixes PR12364. llvm-svn: 153797
2012-03-31 11:38:40 +08:00
Module *M = BB.getParent()->getParent();
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *ArgTypeI = I->getType();
Type *ArgTypeJ = J->getType();
Type *VArgType = getVecTypeForPair(ArgTypeI, ArgTypeJ);
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
ReplacedOperands[o] = Intrinsic::getDeclaration(M, IID, VArgType);
Correctly vectorize powi. The powi intrinsic requires special handling because it always takes a single integer power regardless of the result type. As a result, we can vectorize only if the powers are equal. Fixes PR12364. llvm-svn: 153797
2012-03-31 11:38:40 +08:00
continue;
} else if (IID == Intrinsic::powi && o == 1) {
// The second argument of powi is a single integer and we've already
// checked that both arguments are equal. As a result, we just keep
// I's second argument.
ReplacedOperands[o] = I->getOperand(o);
continue;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
} else if (isa<ShuffleVectorInst>(I) && o == NumOperands-1) {
ReplacedOperands[o] = getReplacementShuffleMask(Context, I, J);
continue;
}
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
ReplacedOperands[o] = getReplacementInput(Context, I, J, o, IBeforeJ);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
// This function creates two values that represent the outputs of the
// original I and J instructions. These are generally vector shuffles
// or extracts. In many cases, these will end up being unused and, thus,
// eliminated by later passes.
void BBVectorize::replaceOutputsOfPair(LLVMContext& Context, Instruction *I,
Instruction *J, Instruction *K,
Instruction *&InsertionPt,
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
Instruction *&K1, Instruction *&K2) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (isa<StoreInst>(I)) {
AA->replaceWithNewValue(I, K);
AA->replaceWithNewValue(J, K);
} else {
Type *IType = I->getType();
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *JType = J->getType();
VectorType *VType = getVecTypeForPair(IType, JType);
unsigned numElem = VType->getNumElements();
unsigned numElemI, numElemJ;
if (IType->isVectorTy())
numElemI = cast<VectorType>(IType)->getNumElements();
else
numElemI = 1;
if (JType->isVectorTy())
numElemJ = cast<VectorType>(JType)->getNumElements();
else
numElemJ = 1;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (IType->isVectorTy()) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
std::vector<Constant*> Mask1(numElemI), Mask2(numElemI);
for (unsigned v = 0; v < numElemI; ++v) {
Mask1[v] = ConstantInt::get(Type::getInt32Ty(Context), v);
Mask2[v] = ConstantInt::get(Type::getInt32Ty(Context), numElemJ+v);
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
K1 = new ShuffleVectorInst(K, UndefValue::get(VType),
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
ConstantVector::get( Mask1),
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
getReplacementName(K, false, 1));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
} else {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Value *CV0 = ConstantInt::get(Type::getInt32Ty(Context), 0);
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
K1 = ExtractElementInst::Create(K, CV0,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
getReplacementName(K, false, 1));
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
}
if (JType->isVectorTy()) {
std::vector<Constant*> Mask1(numElemJ), Mask2(numElemJ);
for (unsigned v = 0; v < numElemJ; ++v) {
Mask1[v] = ConstantInt::get(Type::getInt32Ty(Context), v);
Mask2[v] = ConstantInt::get(Type::getInt32Ty(Context), numElemI+v);
}
K2 = new ShuffleVectorInst(K, UndefValue::get(VType),
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
ConstantVector::get( Mask2),
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
getReplacementName(K, false, 2));
} else {
Value *CV1 = ConstantInt::get(Type::getInt32Ty(Context), numElem-1);
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
K2 = ExtractElementInst::Create(K, CV1,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
getReplacementName(K, false, 2));
}
K1->insertAfter(K);
K2->insertAfter(K1);
InsertionPt = K2;
}
}
// Move all uses of the function I (including pairing-induced uses) after J.
bool BBVectorize::canMoveUsesOfIAfterJ(BasicBlock &BB,
std::multimap<Value *, Value *> &LoadMoveSet,
Instruction *I, Instruction *J) {
// Skip to the first instruction past I.
BBVectorize: Simplify code, no functionality change. Also silences warnings about bodyless for loops. llvm-svn: 149612
2012-02-03 02:52:15 +08:00
BasicBlock::iterator L = llvm::next(BasicBlock::iterator(I));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DenseSet<Value *> Users;
AliasSetTracker WriteSet(*AA);
for (; cast<Instruction>(L) != J; ++L)
(void) trackUsesOfI(Users, WriteSet, I, L, true, &LoadMoveSet);
assert(cast<Instruction>(L) == J &&
"Tracking has not proceeded far enough to check for dependencies");
// If J is now in the use set of I, then trackUsesOfI will return true
// and we have a dependency cycle (and the fusing operation must abort).
return !trackUsesOfI(Users, WriteSet, I, J, true, &LoadMoveSet);
}
// Move all uses of the function I (including pairing-induced uses) after J.
void BBVectorize::moveUsesOfIAfterJ(BasicBlock &BB,
std::multimap<Value *, Value *> &LoadMoveSet,
Instruction *&InsertionPt,
Instruction *I, Instruction *J) {
// Skip to the first instruction past I.
BBVectorize: Simplify code, no functionality change. Also silences warnings about bodyless for loops. llvm-svn: 149612
2012-02-03 02:52:15 +08:00
BasicBlock::iterator L = llvm::next(BasicBlock::iterator(I));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DenseSet<Value *> Users;
AliasSetTracker WriteSet(*AA);
for (; cast<Instruction>(L) != J;) {
if (trackUsesOfI(Users, WriteSet, I, L, true, &LoadMoveSet)) {
// Move this instruction
Instruction *InstToMove = L; ++L;
DEBUG(dbgs() << "BBV: moving: " << *InstToMove <<
" to after " << *InsertionPt << "\n");
InstToMove->removeFromParent();
InstToMove->insertAfter(InsertionPt);
InsertionPt = InstToMove;
} else {
++L;
}
}
}
// Collect all load instruction that are in the move set of a given first
// pair member. These loads depend on the first instruction, I, and so need
// to be moved after J (the second instruction) when the pair is fused.
void BBVectorize::collectPairLoadMoveSet(BasicBlock &BB,
DenseMap<Value *, Value *> &ChosenPairs,
std::multimap<Value *, Value *> &LoadMoveSet,
Instruction *I) {
// Skip to the first instruction past I.
BBVectorize: Simplify code, no functionality change. Also silences warnings about bodyless for loops. llvm-svn: 149612
2012-02-03 02:52:15 +08:00
BasicBlock::iterator L = llvm::next(BasicBlock::iterator(I));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DenseSet<Value *> Users;
AliasSetTracker WriteSet(*AA);
// Note: We cannot end the loop when we reach J because J could be moved
// farther down the use chain by another instruction pairing. Also, J
// could be before I if this is an inverted input.
for (BasicBlock::iterator E = BB.end(); cast<Instruction>(L) != E; ++L) {
if (trackUsesOfI(Users, WriteSet, I, L)) {
if (L->mayReadFromMemory())
LoadMoveSet.insert(ValuePair(L, I));
}
}
}
// In cases where both load/stores and the computation of their pointers
// are chosen for vectorization, we can end up in a situation where the
// aliasing analysis starts returning different query results as the
// process of fusing instruction pairs continues. Because the algorithm
// relies on finding the same use trees here as were found earlier, we'll
// need to precompute the necessary aliasing information here and then
// manually update it during the fusion process.
void BBVectorize::collectLoadMoveSet(BasicBlock &BB,
std::vector<Value *> &PairableInsts,
DenseMap<Value *, Value *> &ChosenPairs,
std::multimap<Value *, Value *> &LoadMoveSet) {
for (std::vector<Value *>::iterator PI = PairableInsts.begin(),
PIE = PairableInsts.end(); PI != PIE; ++PI) {
DenseMap<Value *, Value *>::iterator P = ChosenPairs.find(*PI);
if (P == ChosenPairs.end()) continue;
Instruction *I = cast<Instruction>(P->first);
collectPairLoadMoveSet(BB, ChosenPairs, LoadMoveSet, I);
}
}
Teach BBVectorize to combine, when possible, or discard metadata when fusing instructions. The present implementation handles only TBAA and FP metadata, discarding everything else. For debug metadata, the current behavior is maintained (the debug metadata associated with one of the instructions will be kept, discarding that attached to the other). This should address PR 13040. llvm-svn: 158606
2012-06-17 04:34:06 +08:00
// When the first instruction in each pair is cloned, it will inherit its
// parent's metadata. This metadata must be combined with that of the other
// instruction in a safe way.
void BBVectorize::combineMetadata(Instruction *K, const Instruction *J) {
SmallVector<std::pair<unsigned, MDNode*>, 4> Metadata;
K->getAllMetadataOtherThanDebugLoc(Metadata);
for (unsigned i = 0, n = Metadata.size(); i < n; ++i) {
unsigned Kind = Metadata[i].first;
MDNode *JMD = J->getMetadata(Kind);
MDNode *KMD = Metadata[i].second;
switch (Kind) {
default:
K->setMetadata(Kind, 0); // Remove unknown metadata
break;
case LLVMContext::MD_tbaa:
K->setMetadata(Kind, MDNode::getMostGenericTBAA(JMD, KMD));
break;
case LLVMContext::MD_fpmath:
K->setMetadata(Kind, MDNode::getMostGenericFPMath(JMD, KMD));
break;
}
}
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// This function fuses the chosen instruction pairs into vector instructions,
// taking care preserve any needed scalar outputs and, then, it reorders the
// remaining instructions as needed (users of the first member of the pair
// need to be moved to after the location of the second member of the pair
// because the vector instruction is inserted in the location of the pair's
// second member).
void BBVectorize::fuseChosenPairs(BasicBlock &BB,
std::vector<Value *> &PairableInsts,
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
DenseMap<Value *, Value *> &ChosenPairs,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
DenseSet<ValuePair> &FixedOrderPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes,
std::multimap<ValuePair, ValuePair> &ConnectedPairs,
std::multimap<ValuePair, ValuePair> &ConnectedPairDeps) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
LLVMContext& Context = BB.getContext();
// During the vectorization process, the order of the pairs to be fused
// could be flipped. So we'll add each pair, flipped, into the ChosenPairs
// list. After a pair is fused, the flipped pair is removed from the list.
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
DenseSet<ValuePair> FlippedPairs;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
for (DenseMap<Value *, Value *>::iterator P = ChosenPairs.begin(),
E = ChosenPairs.end(); P != E; ++P)
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
FlippedPairs.insert(ValuePair(P->second, P->first));
for (DenseSet<ValuePair>::iterator P = FlippedPairs.begin(),
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
E = FlippedPairs.end(); P != E; ++P)
ChosenPairs.insert(*P);
std::multimap<Value *, Value *> LoadMoveSet;
collectLoadMoveSet(BB, PairableInsts, ChosenPairs, LoadMoveSet);
DEBUG(dbgs() << "BBV: initial: \n" << BB << "\n");
for (BasicBlock::iterator PI = BB.getFirstInsertionPt(); PI != BB.end();) {
DenseMap<Value *, Value *>::iterator P = ChosenPairs.find(PI);
if (P == ChosenPairs.end()) {
++PI;
continue;
}
if (getDepthFactor(P->first) == 0) {
// These instructions are not really fused, but are tracked as though
// they are. Any case in which it would be interesting to fuse them
// will be taken care of by InstCombine.
--NumFusedOps;
++PI;
continue;
}
Instruction *I = cast<Instruction>(P->first),
*J = cast<Instruction>(P->second);
DEBUG(dbgs() << "BBV: fusing: " << *I <<
" <-> " << *J << "\n");
// Remove the pair and flipped pair from the list.
DenseMap<Value *, Value *>::iterator FP = ChosenPairs.find(P->second);
assert(FP != ChosenPairs.end() && "Flipped pair not found in list");
ChosenPairs.erase(FP);
ChosenPairs.erase(P);
if (!canMoveUsesOfIAfterJ(BB, LoadMoveSet, I, J)) {
DEBUG(dbgs() << "BBV: fusion of: " << *I <<
" <-> " << *J <<
" aborted because of non-trivial dependency cycle\n");
--NumFusedOps;
++PI;
continue;
}
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
// If the pair must have the other order, then flip it.
bool FlipPairOrder = FixedOrderPairs.count(ValuePair(J, I));
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
if (!FlipPairOrder && !FixedOrderPairs.count(ValuePair(I, J))) {
// This pair does not have a fixed order, and so we might want to
// flip it if that will yield fewer shuffles. We count the number
// of dependencies connected via swaps, and those directly connected,
// and flip the order if the number of swaps is greater.
bool OrigOrder = true;
VPPIteratorPair IP = ConnectedPairDeps.equal_range(ValuePair(I, J));
if (IP.first == ConnectedPairDeps.end()) {
IP = ConnectedPairDeps.equal_range(ValuePair(J, I));
OrigOrder = false;
}
if (IP.first != ConnectedPairDeps.end()) {
unsigned NumDepsDirect = 0, NumDepsSwap = 0;
for (std::multimap<ValuePair, ValuePair>::iterator Q = IP.first;
Q != IP.second; ++Q) {
DenseMap<VPPair, unsigned>::iterator R =
PairConnectionTypes.find(VPPair(Q->second, Q->first));
assert(R != PairConnectionTypes.end() &&
"Cannot find pair connection type");
if (R->second == PairConnectionDirect)
++NumDepsDirect;
else if (R->second == PairConnectionSwap)
++NumDepsSwap;
}
if (!OrigOrder)
std::swap(NumDepsDirect, NumDepsSwap);
if (NumDepsSwap > NumDepsDirect) {
FlipPairOrder = true;
DEBUG(dbgs() << "BBV: reordering pair: " << *I <<
" <-> " << *J << "\n");
}
}
}
Precompute SCEV pointer analysis prior to instruction fusion in BBVectorize. When both a load/store and its address computation are being vectorized, it can happen that the address-computation vectorization destroys SCEV's ability to analyize the relative pointer offsets. As a result (like with the aliasing analysis info), we need to precompute the necessary information prior to instruction fusing. This was found during stress testing (running through the test suite with a very low required chain length); unfortunately, I don't have a small test case. llvm-svn: 159332
2012-06-28 13:42:45 +08:00
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
Instruction *L = I, *H = J;
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
if (FlipPairOrder)
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
std::swap(H, L);
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
// If the pair being fused uses the opposite order from that in the pair
// connection map, then we need to flip the types.
VPPIteratorPair IP = ConnectedPairs.equal_range(ValuePair(H, L));
for (std::multimap<ValuePair, ValuePair>::iterator Q = IP.first;
Q != IP.second; ++Q) {
DenseMap<VPPair, unsigned>::iterator R = PairConnectionTypes.find(*Q);
assert(R != PairConnectionTypes.end() &&
"Cannot find pair connection type");
if (R->second == PairConnectionDirect)
R->second = PairConnectionSwap;
else if (R->second == PairConnectionSwap)
R->second = PairConnectionDirect;
}
bool LBeforeH = !FlipPairOrder;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
unsigned NumOperands = I->getNumOperands();
SmallVector<Value *, 3> ReplacedOperands(NumOperands);
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementInputsForPair(Context, L, H, ReplacedOperands,
LBeforeH);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Make a copy of the original operation, change its type to the vector
// type and replace its operands with the vector operands.
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
Instruction *K = L->clone();
if (L->hasName())
K->takeName(L);
else if (H->hasName())
K->takeName(H);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (!isa<StoreInst>(K))
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
K->mutateType(getVecTypeForPair(L->getType(), H->getType()));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
combineMetadata(K, H);
BBVectorize: Correctly merge SubclassOptionalData When two instructions are combined into a vector instruction, the resulting instruction must have the most-conservative flags. llvm-svn: 168765
2012-11-28 11:04:10 +08:00
K->intersectOptionalDataWith(H);
Teach BBVectorize to combine, when possible, or discard metadata when fusing instructions. The present implementation handles only TBAA and FP metadata, discarding everything else. For debug metadata, the current behavior is maintained (the debug metadata associated with one of the instructions will be kept, discarding that attached to the other). This should address PR 13040. llvm-svn: 158606
2012-06-17 04:34:06 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
for (unsigned o = 0; o < NumOperands; ++o)
K->setOperand(o, ReplacedOperands[o]);
K->insertAfter(J);
// Instruction insertion point:
Instruction *InsertionPt = K;
Instruction *K1 = 0, *K2 = 0;
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
replaceOutputsOfPair(Context, L, H, K, InsertionPt, K1, K2);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// The use tree of the first original instruction must be moved to after
// the location of the second instruction. The entire use tree of the
// first instruction is disjoint from the input tree of the second
// (by definition), and so commutes with it.
moveUsesOfIAfterJ(BB, LoadMoveSet, InsertionPt, I, J);
if (!isa<StoreInst>(I)) {
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
L->replaceAllUsesWith(K1);
H->replaceAllUsesWith(K2);
AA->replaceWithNewValue(L, K1);
AA->replaceWithNewValue(H, K2);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
// Instructions that may read from memory may be in the load move set.
// Once an instruction is fused, we no longer need its move set, and so
// the values of the map never need to be updated. However, when a load
// is fused, we need to merge the entries from both instructions in the
// pair in case those instructions were in the move set of some other
// yet-to-be-fused pair. The loads in question are the keys of the map.
if (I->mayReadFromMemory()) {
std::vector<ValuePair> NewSetMembers;
VPIteratorPair IPairRange = LoadMoveSet.equal_range(I);
VPIteratorPair JPairRange = LoadMoveSet.equal_range(J);
for (std::multimap<Value *, Value *>::iterator N = IPairRange.first;
N != IPairRange.second; ++N)
NewSetMembers.push_back(ValuePair(K, N->second));
for (std::multimap<Value *, Value *>::iterator N = JPairRange.first;
N != JPairRange.second; ++N)
NewSetMembers.push_back(ValuePair(K, N->second));
for (std::vector<ValuePair>::iterator A = NewSetMembers.begin(),
AE = NewSetMembers.end(); A != AE; ++A)
LoadMoveSet.insert(*A);
}
// Before removing I, set the iterator to the next instruction.
PI = llvm::next(BasicBlock::iterator(I));
if (cast<Instruction>(PI) == J)
++PI;
SE->forgetValue(I);
SE->forgetValue(J);
I->eraseFromParent();
J->eraseFromParent();
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
DEBUG(if (PrintAfterEveryPair) dbgs() << "BBV: block is now: \n" <<
BB << "\n");
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
DEBUG(dbgs() << "BBV: final: \n" << BB << "\n");
}
}
char BBVectorize::ID = 0;
static const char bb_vectorize_name[] = "Basic-Block Vectorization";
INITIALIZE_PASS_BEGIN(BBVectorize, BBV_NAME, bb_vectorize_name, false, false)
INITIALIZE_AG_DEPENDENCY(AliasAnalysis)
Switch BBVectorize to directly depend on having a TTI analysis. This could be simplified further, but Hal has a specific feature for ignoring TTI, and so I preserved that. Also, I needed to use it because a number of tests fail when switching from a null TTI to the NoTTI nonce implementation. That seems suspicious to me and so may be something that you need to look into Hal. I worked it by preserving the old behavior for these tests with the flag that ignores all target info. llvm-svn: 171722
2013-01-07 18:22:36 +08:00
INITIALIZE_AG_DEPENDENCY(TargetTransformInfo)
BBVectorize should ignore unreachable blocks. Unreachable blocks can have invalid instructions. For example, jump threading can produce self-referential instructions in unreachable blocks. Also, we should not be spending time optimizing unreachable code. Fixes PR14133. llvm-svn: 166423
2012-10-23 02:00:55 +08:00
INITIALIZE_PASS_DEPENDENCY(DominatorTree)
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
INITIALIZE_PASS_DEPENDENCY(ScalarEvolution)
INITIALIZE_PASS_END(BBVectorize, BBV_NAME, bb_vectorize_name, false, false)
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
BasicBlockPass *llvm::createBBVectorizePass(const VectorizeConfig &C) {
return new BBVectorize(C);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
bool
llvm::vectorizeBasicBlock(Pass *P, BasicBlock &BB, const VectorizeConfig &C) {
BBVectorize BBVectorizer(P, C);
Add the function "vectorizeBasicBlock" which allow users vectorize a BasicBlock in other passes, e.g. we can call vectorizeBasicBlock in the loop unroll pass right after the loop is unrolled. llvm-svn: 154089
2012-04-05 16:05:16 +08:00
return BBVectorizer.vectorizeBB(BB);
}
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
//===----------------------------------------------------------------------===//
VectorizeConfig::VectorizeConfig() {
VectorBits = ::VectorBits;
Allow controlling vectorization of boolean values separately from other integer types. These are used as the result of comparisons, and often handled differently from larger integer types. llvm-svn: 159111
2012-06-24 21:28:01 +08:00
VectorizeBools = !::NoBools;
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
VectorizeInts = !::NoInts;
VectorizeFloats = !::NoFloats;
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
VectorizePointers = !::NoPointers;
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
VectorizeCasts = !::NoCasts;
VectorizeMath = !::NoMath;
VectorizeFMA = !::NoFMA;
Add support to BBVectorize for vectorizing selects. llvm-svn: 154700
2012-04-14 04:45:45 +08:00
VectorizeSelect = !::NoSelect;
Allow BBVectorize to fuse compare instructions. llvm-svn: 159088
2012-06-24 05:52:50 +08:00
VectorizeCmp = !::NoCmp;
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
VectorizeGEP = !::NoGEP;
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
VectorizeMemOps = !::NoMemOps;
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
AlignedOnly = ::AlignedOnly;
ReqChainDepth= ::ReqChainDepth;
SearchLimit = ::SearchLimit;
MaxCandPairsForCycleCheck = ::MaxCandPairsForCycleCheck;
SplatBreaksChain = ::SplatBreaksChain;
MaxInsts = ::MaxInsts;
MaxIter = ::MaxIter;
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Pow2LenOnly = ::Pow2LenOnly;
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
NoMemOpBoost = ::NoMemOpBoost;
FastDep = ::FastDep;
}