maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/llvm/lib/Transforms/Vectorize/BBVectorize.cpp

3283 lines
133 KiB
C++
Raw Normal View History

Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
//===- BBVectorize.cpp - A Basic-Block Vectorizer -------------------------===//
//
// The LLVM Compiler Infrastructure
//
// This file is distributed under the University of Illinois Open Source
// License. See LICENSE.TXT for details.
//
//===----------------------------------------------------------------------===//
//
// This file implements a basic-block vectorization pass. The algorithm was
// inspired by that used by the Vienna MAP Vectorizor by Franchetti and Kral,
// et al. It works by looking for chains of pairable operations and then
// pairing them.
//
//===----------------------------------------------------------------------===//
#define BBV_NAME "bb-vectorize"
Use the new script to sort the includes of every file under lib. Sooooo many of these had incorrect or strange main module includes. I have manually inspected all of these, and fixed the main module include to be the nearest plausible thing I could find. If you own or care about any of these source files, I encourage you to take some time and check that these edits were sensible. I can't have broken anything (I strictly added headers, and reordered them, never removed), but they may not be the headers you'd really like to identify as containing the API being implemented. Many forward declarations and missing includes were added to a header files to allow them to parse cleanly when included first. The main module rule does in fact have its merits. =] llvm-svn: 169131
2012-12-04 00:50:05 +08:00
#include "llvm/Transforms/Vectorize.h"
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#include "llvm/ADT/DenseMap.h"
#include "llvm/ADT/DenseSet.h"
Use the new script to sort the includes of every file under lib. Sooooo many of these had incorrect or strange main module includes. I have manually inspected all of these, and fixed the main module include to be the nearest plausible thing I could find. If you own or care about any of these source files, I encourage you to take some time and check that these edits were sensible. I can't have broken anything (I strictly added headers, and reordered them, never removed), but they may not be the headers you'd really like to identify as containing the API being implemented. Many forward declarations and missing includes were added to a header files to allow them to parse cleanly when included first. The main module rule does in fact have its merits. =] llvm-svn: 169131
2012-12-04 00:50:05 +08:00
#include "llvm/ADT/STLExtras.h"
BBVectorize: Use a more sophisticated check for input cost The old checking code, which assumed that input shuffles and insert-elements could always be folded (and thus were free) is too simple. This can only happen in special circumstances. Using the simple check caused infinite recursion. llvm-svn: 167750
2012-11-13 05:21:02 +08:00
#include "llvm/ADT/SmallSet.h"
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/Statistic.h"
#include "llvm/ADT/StringExtras.h"
#include "llvm/Analysis/AliasAnalysis.h"
#include "llvm/Analysis/AliasSetTracker.h"
[PM/AA] Rebuild LLVM's alias analysis infrastructure in a way compatible with the new pass manager, and no longer relying on analysis groups. This builds essentially a ground-up new AA infrastructure stack for LLVM. The core ideas are the same that are used throughout the new pass manager: type erased polymorphism and direct composition. The design is as follows: - FunctionAAResults is a type-erasing alias analysis results aggregation interface to walk a single query across a range of results from different alias analyses. Currently this is function-specific as we always assume that aliasing queries are *within* a function. - AAResultBase is a CRTP utility providing stub implementations of various parts of the alias analysis result concept, notably in several cases in terms of other more general parts of the interface. This can be used to implement only a narrow part of the interface rather than the entire interface. This isn't really ideal, this logic should be hoisted into FunctionAAResults as currently it will cause a significant amount of redundant work, but it faithfully models the behavior of the prior infrastructure. - All the alias analysis passes are ported to be wrapper passes for the legacy PM and new-style analysis passes for the new PM with a shared result object. In some cases (most notably CFL), this is an extremely naive approach that we should revisit when we can specialize for the new pass manager. - BasicAA has been restructured to reflect that it is much more fundamentally a function analysis because it uses dominator trees and loop info that need to be constructed for each function. All of the references to getting alias analysis results have been updated to use the new aggregation interface. All the preservation and other pass management code has been updated accordingly. The way the FunctionAAResultsWrapperPass works is to detect the available alias analyses when run, and add them to the results object. This means that we should be able to continue to respect when various passes are added to the pipeline, for example adding CFL or adding TBAA passes should just cause their results to be available and to get folded into this. The exception to this rule is BasicAA which really needs to be a function pass due to using dominator trees and loop info. As a consequence, the FunctionAAResultsWrapperPass directly depends on BasicAA and always includes it in the aggregation. This has significant implications for preserving analyses. Generally, most passes shouldn't bother preserving FunctionAAResultsWrapperPass because rebuilding the results just updates the set of known AA passes. The exception to this rule are LoopPass instances which need to preserve all the function analyses that the loop pass manager will end up needing. This means preserving both BasicAAWrapperPass and the aggregating FunctionAAResultsWrapperPass. Now, when preserving an alias analysis, you do so by directly preserving that analysis. This is only necessary for non-immutable-pass-provided alias analyses though, and there are only three of interest: BasicAA, GlobalsAA (formerly GlobalsModRef), and SCEVAA. Usually BasicAA is preserved when needed because it (like DominatorTree and LoopInfo) is marked as a CFG-only pass. I've expanded GlobalsAA into the preserved set everywhere we previously were preserving all of AliasAnalysis, and I've added SCEVAA in the intersection of that with where we preserve SCEV itself. One significant challenge to all of this is that the CGSCC passes were actually using the alias analysis implementations by taking advantage of a pretty amazing set of loop holes in the old pass manager's analysis management code which allowed analysis groups to slide through in many cases. Moving away from analysis groups makes this problem much more obvious. To fix it, I've leveraged the flexibility the design of the new PM components provides to just directly construct the relevant alias analyses for the relevant functions in the IPO passes that need them. This is a bit hacky, but should go away with the new pass manager, and is already in many ways cleaner than the prior state. Another significant challenge is that various facilities of the old alias analysis infrastructure just don't fit any more. The most significant of these is the alias analysis 'counter' pass. That pass relied on the ability to snoop on AA queries at different points in the analysis group chain. Instead, I'm planning to build printing functionality directly into the aggregation layer. I've not included that in this patch merely to keep it smaller. Note that all of this needs a nearly complete rewrite of the AA documentation. I'm planning to do that, but I'd like to make sure the new design settles, and to flesh out a bit more of what it looks like in the new pass manager first. Differential Revision: http://reviews.llvm.org/D12080 llvm-svn: 247167
2015-09-10 01:55:00 +08:00
#include "llvm/Analysis/GlobalsModRef.h"
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#include "llvm/Analysis/ScalarEvolution.h"
[PM/AA] Rebuild LLVM's alias analysis infrastructure in a way compatible with the new pass manager, and no longer relying on analysis groups. This builds essentially a ground-up new AA infrastructure stack for LLVM. The core ideas are the same that are used throughout the new pass manager: type erased polymorphism and direct composition. The design is as follows: - FunctionAAResults is a type-erasing alias analysis results aggregation interface to walk a single query across a range of results from different alias analyses. Currently this is function-specific as we always assume that aliasing queries are *within* a function. - AAResultBase is a CRTP utility providing stub implementations of various parts of the alias analysis result concept, notably in several cases in terms of other more general parts of the interface. This can be used to implement only a narrow part of the interface rather than the entire interface. This isn't really ideal, this logic should be hoisted into FunctionAAResults as currently it will cause a significant amount of redundant work, but it faithfully models the behavior of the prior infrastructure. - All the alias analysis passes are ported to be wrapper passes for the legacy PM and new-style analysis passes for the new PM with a shared result object. In some cases (most notably CFL), this is an extremely naive approach that we should revisit when we can specialize for the new pass manager. - BasicAA has been restructured to reflect that it is much more fundamentally a function analysis because it uses dominator trees and loop info that need to be constructed for each function. All of the references to getting alias analysis results have been updated to use the new aggregation interface. All the preservation and other pass management code has been updated accordingly. The way the FunctionAAResultsWrapperPass works is to detect the available alias analyses when run, and add them to the results object. This means that we should be able to continue to respect when various passes are added to the pipeline, for example adding CFL or adding TBAA passes should just cause their results to be available and to get folded into this. The exception to this rule is BasicAA which really needs to be a function pass due to using dominator trees and loop info. As a consequence, the FunctionAAResultsWrapperPass directly depends on BasicAA and always includes it in the aggregation. This has significant implications for preserving analyses. Generally, most passes shouldn't bother preserving FunctionAAResultsWrapperPass because rebuilding the results just updates the set of known AA passes. The exception to this rule are LoopPass instances which need to preserve all the function analyses that the loop pass manager will end up needing. This means preserving both BasicAAWrapperPass and the aggregating FunctionAAResultsWrapperPass. Now, when preserving an alias analysis, you do so by directly preserving that analysis. This is only necessary for non-immutable-pass-provided alias analyses though, and there are only three of interest: BasicAA, GlobalsAA (formerly GlobalsModRef), and SCEVAA. Usually BasicAA is preserved when needed because it (like DominatorTree and LoopInfo) is marked as a CFG-only pass. I've expanded GlobalsAA into the preserved set everywhere we previously were preserving all of AliasAnalysis, and I've added SCEVAA in the intersection of that with where we preserve SCEV itself. One significant challenge to all of this is that the CGSCC passes were actually using the alias analysis implementations by taking advantage of a pretty amazing set of loop holes in the old pass manager's analysis management code which allowed analysis groups to slide through in many cases. Moving away from analysis groups makes this problem much more obvious. To fix it, I've leveraged the flexibility the design of the new PM components provides to just directly construct the relevant alias analyses for the relevant functions in the IPO passes that need them. This is a bit hacky, but should go away with the new pass manager, and is already in many ways cleaner than the prior state. Another significant challenge is that various facilities of the old alias analysis infrastructure just don't fit any more. The most significant of these is the alias analysis 'counter' pass. That pass relied on the ability to snoop on AA queries at different points in the analysis group chain. Instead, I'm planning to build printing functionality directly into the aggregation layer. I've not included that in this patch merely to keep it smaller. Note that all of this needs a nearly complete rewrite of the AA documentation. I'm planning to do that, but I'd like to make sure the new design settles, and to flesh out a bit more of what it looks like in the new pass manager first. Differential Revision: http://reviews.llvm.org/D12080 llvm-svn: 247167
2015-09-10 01:55:00 +08:00
#include "llvm/Analysis/ScalarEvolutionAliasAnalysis.h"
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#include "llvm/Analysis/ScalarEvolutionExpressions.h"
[PM/AA] Explicitly depend on TLI rather than getting it out of the AliasAnalysis. Same as the other commits, the TLI access from an alias analysis is going away and isn't very clean -- it is better to explicitly mark the dependencies. llvm-svn: 244785
2015-08-13 02:06:08 +08:00
#include "llvm/Analysis/TargetLibraryInfo.h"
Move TargetTransformInfo to live under the Analysis library. This no longer would violate any dependency layering and it is in fact an analysis. =] llvm-svn: 171686
2013-01-07 11:08:10 +08:00
#include "llvm/Analysis/TargetTransformInfo.h"
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#include "llvm/Analysis/ValueTracking.h"
Move all of the header files which are involved in modelling the LLVM IR into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. llvm-svn: 171366
2013-01-02 19:36:10 +08:00
#include "llvm/IR/Constants.h"
#include "llvm/IR/DataLayout.h"
#include "llvm/IR/DerivedTypes.h"
[cleanup] Move the Dominators.h and Verifier.h headers into the IR directory. These passes are already defined in the IR library, and it doesn't make any sense to have the headers in Analysis. Long term, I think there is going to be a much better way to divide these matters. The dominators code should be fully separated into the abstract graph algorithm and have that put in Support where it becomes obvious that evn Clang's CFGBlock's can use it. Then the verifier can manually construct dominance information from the Support-driven interface while the Analysis library can provide a pass which both caches, reconstructs, and supports a nice update API. But those are very long term, and so I don't want to leave the really confusing structure until that day arrives. llvm-svn: 199082
2014-01-13 17:26:24 +08:00
#include "llvm/IR/Dominators.h"
Move all of the header files which are involved in modelling the LLVM IR into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. llvm-svn: 171366
2013-01-02 19:36:10 +08:00
#include "llvm/IR/Function.h"
#include "llvm/IR/Instructions.h"
#include "llvm/IR/IntrinsicInst.h"
#include "llvm/IR/Intrinsics.h"
#include "llvm/IR/LLVMContext.h"
#include "llvm/IR/Metadata.h"
Make DataLayout Non-Optional in the Module Summary: DataLayout keeps the string used for its creation. As a side effect it is no longer needed in the Module. This is "almost" NFC, the string is no longer canonicalized, you can't rely on two "equals" DataLayout having the same string returned by getStringRepresentation(). Get rid of DataLayoutPass: the DataLayout is in the Module The DataLayout is "per-module", let's enforce this by not duplicating it more than necessary. One more step toward non-optionality of the DataLayout in the module. Make DataLayout Non-Optional in the Module Module->getDataLayout() will never returns nullptr anymore. Reviewers: echristo Subscribers: resistor, llvm-commits, jholewinski Differential Revision: http://reviews.llvm.org/D7992 From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 231270
2015-03-05 02:43:29 +08:00
#include "llvm/IR/Module.h"
Move all of the header files which are involved in modelling the LLVM IR into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. llvm-svn: 171366
2013-01-02 19:36:10 +08:00
#include "llvm/IR/Type.h"
[Modules] Move ValueHandle into the IR library where Value itself lives. Move the test for this class into the IR unittests as well. This uncovers that ValueMap too is in the IR library. Ironically, the unittest for ValueMap is useless in the Support library (honestly, so was the ValueHandle test) and so it already lives in the IR unittests. Mmmm, tasty layering. llvm-svn: 202821
2014-03-04 19:17:44 +08:00
#include "llvm/IR/ValueHandle.h"
Use the new script to sort the includes of every file under lib. Sooooo many of these had incorrect or strange main module includes. I have manually inspected all of these, and fixed the main module include to be the nearest plausible thing I could find. If you own or care about any of these source files, I encourage you to take some time and check that these edits were sensible. I can't have broken anything (I strictly added headers, and reordered them, never removed), but they may not be the headers you'd really like to identify as containing the API being implemented. Many forward declarations and missing includes were added to a header files to allow them to parse cleanly when included first. The main module rule does in fact have its merits. =] llvm-svn: 169131
2012-12-04 00:50:05 +08:00
#include "llvm/Pass.h"
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#include "llvm/Support/CommandLine.h"
#include "llvm/Support/Debug.h"
Use the new script to sort the includes of every file under lib. Sooooo many of these had incorrect or strange main module includes. I have manually inspected all of these, and fixed the main module include to be the nearest plausible thing I could find. If you own or care about any of these source files, I encourage you to take some time and check that these edits were sensible. I can't have broken anything (I strictly added headers, and reordered them, never removed), but they may not be the headers you'd really like to identify as containing the API being implemented. Many forward declarations and missing includes were added to a header files to allow them to parse cleanly when included first. The main module rule does in fact have its merits. =] llvm-svn: 169131
2012-12-04 00:50:05 +08:00
#include "llvm/Support/raw_ostream.h"
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
#include "llvm/Transforms/Utils/Local.h"
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#include <algorithm>
using namespace llvm;
[Modules] Fix potential ODR violations by sinking the DEBUG_TYPE definition below all of the header #include lines, lib/Transforms/... edition. This one is tricky for two reasons. We again have a couple of passes that define something else before the includes as well. I've sunk their name macros with the DEBUG_TYPE. Also, InstCombine contains headers that need DEBUG_TYPE, so now those headers #define and #undef DEBUG_TYPE around their code, leaving them well formed modular headers. Fixing these headers was a large motivation for all of these changes, as "leaky" macros of this form are hard on the modules implementation. llvm-svn: 206844
2014-04-22 10:55:47 +08:00
#define DEBUG_TYPE BBV_NAME
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
static cl::opt<bool>
IgnoreTargetInfo("bb-vectorize-ignore-target-info", cl::init(false),
cl::Hidden, cl::desc("Ignore target information"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<unsigned>
ReqChainDepth("bb-vectorize-req-chain-depth", cl::init(6), cl::Hidden,
cl::desc("The required chain depth for vectorization"));
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
static cl::opt<bool>
UseChainDepthWithTI("bb-vectorize-use-chain-depth", cl::init(false),
cl::Hidden, cl::desc("Use the chain depth requirement with"
" target information"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<unsigned>
SearchLimit("bb-vectorize-search-limit", cl::init(400), cl::Hidden,
cl::desc("The maximum search distance for instruction pairs"));
static cl::opt<bool>
SplatBreaksChain("bb-vectorize-splat-breaks-chain", cl::init(false), cl::Hidden,
cl::desc("Replicating one element to a pair breaks the chain"));
static cl::opt<unsigned>
VectorBits("bb-vectorize-vector-bits", cl::init(128), cl::Hidden,
cl::desc("The size of the native vector registers"));
static cl::opt<unsigned>
MaxIter("bb-vectorize-max-iter", cl::init(0), cl::Hidden,
cl::desc("The maximum number of pairing iterations"));
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
static cl::opt<bool>
Pow2LenOnly("bb-vectorize-pow2-len-only", cl::init(false), cl::Hidden,
cl::desc("Don't try to form non-2^n-length vectors"));
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
static cl::opt<unsigned>
MaxInsts("bb-vectorize-max-instr-per-group", cl::init(500), cl::Hidden,
cl::desc("The maximum number of pairable instructions per group"));
BBVectorize: Cap the number of candidate pairs in each instruction group For some basic blocks, it is possible to generate many candidate pairs for relatively few pairable instructions. When many (tens of thousands) of these pairs are generated for a single instruction group, the time taken to generate and rank the different vectorization plans can become quite large. As a result, we now cap the number of candidate pairs within each instruction group. This is done by closing out the group once the threshold is reached (set now at 3000 pairs). Although this will limit the overall compile-time impact, this may not be the best way to achieve this result. It might be better, for example, to prune excessive candidate pairs after the fact the prevent the generation of short, but highly-connected groups. We can experiment with this in the future. This change reduces the overall compile-time slowdown of the csa.ll test case in PR15222 to ~5x. If 5x is still considered too large, a lower limit can be used as the default. This represents a functionality change, but only for very large inputs (thus, there is no regression test). llvm-svn: 175251
2013-02-15 12:28:42 +08:00
static cl::opt<unsigned>
MaxPairs("bb-vectorize-max-pairs-per-group", cl::init(3000), cl::Hidden,
cl::desc("The maximum number of candidate instruction pairs per group"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<unsigned>
MaxCandPairsForCycleCheck("bb-vectorize-max-cycle-check-pairs", cl::init(200),
cl::Hidden, cl::desc("The maximum number of candidate pairs with which to use"
" a full cycle check"));
Allow controlling vectorization of boolean values separately from other integer types. These are used as the result of comparisons, and often handled differently from larger integer types. llvm-svn: 159111
2012-06-24 21:28:01 +08:00
static cl::opt<bool>
NoBools("bb-vectorize-no-bools", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize boolean (i1) values"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<bool>
NoInts("bb-vectorize-no-ints", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize integer values"));
static cl::opt<bool>
NoFloats("bb-vectorize-no-floats", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize floating-point values"));
Disable generation of pointer vectors by BBVectorize. Once vector-of-pointer support works, then this can be reverted. llvm-svn: 166741
2012-10-26 08:05:26 +08:00
// FIXME: This should default to false once pointer vector support works.
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
static cl::opt<bool>
Disable generation of pointer vectors by BBVectorize. Once vector-of-pointer support works, then this can be reverted. llvm-svn: 166741
2012-10-26 08:05:26 +08:00
NoPointers("bb-vectorize-no-pointers", cl::init(/*false*/ true), cl::Hidden,
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
cl::desc("Don't try to vectorize pointer values"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<bool>
NoCasts("bb-vectorize-no-casts", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize casting (conversion) operations"));
static cl::opt<bool>
NoMath("bb-vectorize-no-math", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize floating-point math intrinsics"));
Allow vectorization of bit intrinsics in BB Vectorizer. This patch adds support for vectorization of bit intrinsics such as bswap,ctpop,ctlz,cttz. llvm-svn: 207174
2014-04-25 11:33:48 +08:00
static cl::opt<bool>
NoBitManipulation("bb-vectorize-no-bitmanip", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize BitManipulation intrinsics"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<bool>
NoFMA("bb-vectorize-no-fma", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize the fused-multiply-add intrinsic"));
Add support to BBVectorize for vectorizing selects. llvm-svn: 154700
2012-04-14 04:45:45 +08:00
static cl::opt<bool>
NoSelect("bb-vectorize-no-select", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize select instructions"));
Allow BBVectorize to fuse compare instructions. llvm-svn: 159088
2012-06-24 05:52:50 +08:00
static cl::opt<bool>
NoCmp("bb-vectorize-no-cmp", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize comparison instructions"));
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
static cl::opt<bool>
NoGEP("bb-vectorize-no-gep", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize getelementptr instructions"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<bool>
NoMemOps("bb-vectorize-no-mem-ops", cl::init(false), cl::Hidden,
cl::desc("Don't try to vectorize loads and stores"));
static cl::opt<bool>
AlignedOnly("bb-vectorize-aligned-only", cl::init(false), cl::Hidden,
cl::desc("Only generate aligned loads and stores"));
Boost the effective chain depth of loads and stores. By default, boost the chain depth contribution of loads and stores. This will allow a load/store pair to vectorize even when it would not otherwise be long enough to satisfy the chain depth requirement. llvm-svn: 149761
2012-02-04 12:14:04 +08:00
static cl::opt<bool>
NoMemOpBoost("bb-vectorize-no-mem-op-boost",
cl::init(false), cl::Hidden,
cl::desc("Don't boost the chain-depth contribution of loads and stores"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
static cl::opt<bool>
FastDep("bb-vectorize-fast-dep", cl::init(false), cl::Hidden,
cl::desc("Use a fast instruction dependency analysis"));
#ifndef NDEBUG
static cl::opt<bool>
DebugInstructionExamination("bb-vectorize-debug-instruction-examination",
cl::init(false), cl::Hidden,
cl::desc("When debugging is enabled, output information on the"
" instruction-examination process"));
static cl::opt<bool>
DebugCandidateSelection("bb-vectorize-debug-candidate-selection",
cl::init(false), cl::Hidden,
cl::desc("When debugging is enabled, output information on the"
" candidate-selection process"));
static cl::opt<bool>
DebugPairSelection("bb-vectorize-debug-pair-selection",
cl::init(false), cl::Hidden,
cl::desc("When debugging is enabled, output information on the"
" pair-selection process"));
static cl::opt<bool>
DebugCycleCheck("bb-vectorize-debug-cycle-check",
cl::init(false), cl::Hidden,
cl::desc("When debugging is enabled, output information on the"
" cycle-checking process"));
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
static cl::opt<bool>
PrintAfterEveryPair("bb-vectorize-debug-print-after-every-pair",
cl::init(false), cl::Hidden,
cl::desc("When debugging is enabled, dump the basic block after"
" every pair is fused"));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
#endif
STATISTIC(NumFusedOps, "Number of operations fused by bb-vectorize");
namespace {
struct BBVectorize : public BasicBlockPass {
static char ID; // Pass identification, replacement for typeid
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
BBVectorize: Add the const modifier to the VectorizeConfig because we won't modify it. llvm-svn: 154098
2012-04-06 00:07:49 +08:00
const VectorizeConfig Config;
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
BBVectorize(const VectorizeConfig &C = VectorizeConfig())
: BasicBlockPass(ID), Config(C) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
initializeBBVectorizePass(*PassRegistry::getPassRegistry());
}
[multiversion] Thread a function argument through all the callers of the getTTI method used to get an actual TTI object. No functionality changed. This just threads the argument and ensures code like the inliner can correctly look up the callee's TTI rather than using a fixed one. The next change will use this to implement per-function subtarget usage by TTI. The changes after that should eliminate the need for FTTI as that will have become the default. llvm-svn: 227730
2015-02-01 20:01:35 +08:00
BBVectorize(Pass *P, Function &F, const VectorizeConfig &C)
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
: BasicBlockPass(ID), Config(C) {
[PM/AA] Rebuild LLVM's alias analysis infrastructure in a way compatible with the new pass manager, and no longer relying on analysis groups. This builds essentially a ground-up new AA infrastructure stack for LLVM. The core ideas are the same that are used throughout the new pass manager: type erased polymorphism and direct composition. The design is as follows: - FunctionAAResults is a type-erasing alias analysis results aggregation interface to walk a single query across a range of results from different alias analyses. Currently this is function-specific as we always assume that aliasing queries are *within* a function. - AAResultBase is a CRTP utility providing stub implementations of various parts of the alias analysis result concept, notably in several cases in terms of other more general parts of the interface. This can be used to implement only a narrow part of the interface rather than the entire interface. This isn't really ideal, this logic should be hoisted into FunctionAAResults as currently it will cause a significant amount of redundant work, but it faithfully models the behavior of the prior infrastructure. - All the alias analysis passes are ported to be wrapper passes for the legacy PM and new-style analysis passes for the new PM with a shared result object. In some cases (most notably CFL), this is an extremely naive approach that we should revisit when we can specialize for the new pass manager. - BasicAA has been restructured to reflect that it is much more fundamentally a function analysis because it uses dominator trees and loop info that need to be constructed for each function. All of the references to getting alias analysis results have been updated to use the new aggregation interface. All the preservation and other pass management code has been updated accordingly. The way the FunctionAAResultsWrapperPass works is to detect the available alias analyses when run, and add them to the results object. This means that we should be able to continue to respect when various passes are added to the pipeline, for example adding CFL or adding TBAA passes should just cause their results to be available and to get folded into this. The exception to this rule is BasicAA which really needs to be a function pass due to using dominator trees and loop info. As a consequence, the FunctionAAResultsWrapperPass directly depends on BasicAA and always includes it in the aggregation. This has significant implications for preserving analyses. Generally, most passes shouldn't bother preserving FunctionAAResultsWrapperPass because rebuilding the results just updates the set of known AA passes. The exception to this rule are LoopPass instances which need to preserve all the function analyses that the loop pass manager will end up needing. This means preserving both BasicAAWrapperPass and the aggregating FunctionAAResultsWrapperPass. Now, when preserving an alias analysis, you do so by directly preserving that analysis. This is only necessary for non-immutable-pass-provided alias analyses though, and there are only three of interest: BasicAA, GlobalsAA (formerly GlobalsModRef), and SCEVAA. Usually BasicAA is preserved when needed because it (like DominatorTree and LoopInfo) is marked as a CFG-only pass. I've expanded GlobalsAA into the preserved set everywhere we previously were preserving all of AliasAnalysis, and I've added SCEVAA in the intersection of that with where we preserve SCEV itself. One significant challenge to all of this is that the CGSCC passes were actually using the alias analysis implementations by taking advantage of a pretty amazing set of loop holes in the old pass manager's analysis management code which allowed analysis groups to slide through in many cases. Moving away from analysis groups makes this problem much more obvious. To fix it, I've leveraged the flexibility the design of the new PM components provides to just directly construct the relevant alias analyses for the relevant functions in the IPO passes that need them. This is a bit hacky, but should go away with the new pass manager, and is already in many ways cleaner than the prior state. Another significant challenge is that various facilities of the old alias analysis infrastructure just don't fit any more. The most significant of these is the alias analysis 'counter' pass. That pass relied on the ability to snoop on AA queries at different points in the analysis group chain. Instead, I'm planning to build printing functionality directly into the aggregation layer. I've not included that in this patch merely to keep it smaller. Note that all of this needs a nearly complete rewrite of the AA documentation. I'm planning to do that, but I'd like to make sure the new design settles, and to flesh out a bit more of what it looks like in the new pass manager first. Differential Revision: http://reviews.llvm.org/D12080 llvm-svn: 247167
2015-09-10 01:55:00 +08:00
AA = &P->getAnalysis<AAResultsWrapperPass>().getAAResults();
[PM] Split DominatorTree into a concrete analysis result object which can be used by both the new pass manager and the old. This removes it from any of the virtual mess of the pass interfaces and lets it derive cleanly from the DominatorTreeBase<> template. In turn, tons of boilerplate interface can be nuked and it turns into a very straightforward extension of the base DominatorTree interface. The old analysis pass is now a simple wrapper. The names and style of this split should match the split between CallGraph and CallGraphWrapperPass. All of the users of DominatorTree have been updated to match using many of the same tricks as with CallGraph. The goal is that the common type remains the resulting DominatorTree rather than the pass. This will make subsequent work toward the new pass manager significantly easier. Also in numerous places things became cleaner because I switched from re-running the pass (!!! mid way through some other passes run!!!) to directly recomputing the domtree. llvm-svn: 199104
2014-01-13 21:07:17 +08:00
DT = &P->getAnalysis<DominatorTreeWrapperPass>().getDomTree();
[PM] Port ScalarEvolution to the new pass manager. This change makes ScalarEvolution a stand-alone object and just produces one from a pass as needed. Making this work well requires making the object movable, using references instead of overwritten pointers in a number of places, and other refactorings. I've also wired it up to the new pass manager and added a RUN line to a test to exercise it under the new pass manager. This includes basic printing support much like with other analyses. But there is a big and somewhat scary change here. Prior to this patch ScalarEvolution was never *actually* invalidated!!! Re-running the pass just re-wired up the various other analyses and didn't remove any of the existing entries in the SCEV caches or clear out anything at all. This might seem OK as everything in SCEV that can uses ValueHandles to track updates to the values that serve as SCEV keys. However, this still means that as we ran SCEV over each function in the module, we kept accumulating more and more SCEVs into the cache. At the end, we would have a SCEV cache with every value that we ever needed a SCEV for in the entire module!!! Yowzers. The releaseMemory routine would dump all of this, but that isn't realy called during normal runs of the pipeline as far as I can see. To make matters worse, there *is* actually a key that we don't update with value handles -- there is a map keyed off of Loop*s. Because LoopInfo *does* release its memory from run to run, it is entirely possible to run SCEV over one function, then over another function, and then lookup a Loop* from the second function but find an entry inserted for the first function! Ouch. To make matters still worse, there are plenty of updates that *don't* trip a value handle. It seems incredibly unlikely that today GVN or another pass that invalidates SCEV can update values in *just* such a way that a subsequent run of SCEV will incorrectly find lookups in a cache, but it is theoretically possible and would be a nightmare to debug. With this refactoring, I've fixed all this by actually destroying and recreating the ScalarEvolution object from run to run. Technically, this could increase the amount of malloc traffic we see, but then again it is also technically correct. ;] I don't actually think we're suffering from tons of malloc traffic from SCEV because if we were, the fact that we never clear the memory would seem more likely to have come up as an actual problem before now. So, I've made the simple fix here. If in fact there are serious issues with too much allocation and deallocation, I can work on a clever fix that preserves the allocations (while clearing the data) between each run, but I'd prefer to do that kind of optimization with a test case / benchmark that shows why we need such cleverness (and that can test that we actually make it faster). It's possible that this will make some things faster by making the SCEV caches have higher locality (due to being significantly smaller) so until there is a clear benchmark, I think the simple change is best. Differential Revision: http://reviews.llvm.org/D12063 llvm-svn: 245193
2015-08-17 10:08:17 +08:00
SE = &P->getAnalysis<ScalarEvolutionWrapperPass>().getSE();
[PM/AA] Explicitly depend on TLI rather than getting it out of the AliasAnalysis. Same as the other commits, the TLI access from an alias analysis is going away and isn't very clean -- it is better to explicitly mark the dependencies. llvm-svn: 244785
2015-08-13 02:06:08 +08:00
TLI = &P->getAnalysis<TargetLibraryInfoWrapperPass>().getTLI();
[PM] Change the core design of the TTI analysis to use a polymorphic type erased interface and a single analysis pass rather than an extremely complex analysis group. The end result is that the TTI analysis can contain a type erased implementation that supports the polymorphic TTI interface. We can build one from a target-specific implementation or from a dummy one in the IR. I've also factored all of the code into "mix-in"-able base classes, including CRTP base classes to facilitate calling back up to the most specialized form when delegating horizontally across the surface. These aren't as clean as I would like and I'm planning to work on cleaning some of this up, but I wanted to start by putting into the right form. There are a number of reasons for this change, and this particular design. The first and foremost reason is that an analysis group is complete overkill, and the chaining delegation strategy was so opaque, confusing, and high overhead that TTI was suffering greatly for it. Several of the TTI functions had failed to be implemented in all places because of the chaining-based delegation making there be no checking of this. A few other functions were implemented with incorrect delegation. The message to me was very clear working on this -- the delegation and analysis group structure was too confusing to be useful here. The other reason of course is that this is *much* more natural fit for the new pass manager. This will lay the ground work for a type-erased per-function info object that can look up the correct subtarget and even cache it. Yet another benefit is that this will significantly simplify the interaction of the pass managers and the TargetMachine. See the future work below. The downside of this change is that it is very, very verbose. I'm going to work to improve that, but it is somewhat an implementation necessity in C++ to do type erasure. =/ I discussed this design really extensively with Eric and Hal prior to going down this path, and afterward showed them the result. No one was really thrilled with it, but there doesn't seem to be a substantially better alternative. Using a base class and virtual method dispatch would make the code much shorter, but as discussed in the update to the programmer's manual and elsewhere, a polymorphic interface feels like the more principled approach even if this is perhaps the least compelling example of it. ;] Ultimately, there is still a lot more to be done here, but this was the huge chunk that I couldn't really split things out of because this was the interface change to TTI. I've tried to minimize all the other parts of this. The follow up work should include at least: 1) Improving the TargetMachine interface by having it directly return a TTI object. Because we have a non-pass object with value semantics and an internal type erasure mechanism, we can narrow the interface of the TargetMachine to *just* do what we need: build and return a TTI object that we can then insert into the pass pipeline. 2) Make the TTI object be fully specialized for a particular function. This will include splitting off a minimal form of it which is sufficient for the inliner and the old pass manager. 3) Add a new pass manager analysis which produces TTI objects from the target machine for each function. This may actually be done as part of #2 in order to use the new analysis to implement #2. 4) Work on narrowing the API between TTI and the targets so that it is easier to understand and less verbose to type erase. 5) Work on narrowing the API between TTI and its clients so that it is easier to understand and less verbose to forward. 6) Try to improve the CRTP-based delegation. I feel like this code is just a bit messy and exacerbating the complexity of implementing the TTI in each target. Many thanks to Eric and Hal for their help here. I ended up blocked on this somewhat more abruptly than I expected, and so I appreciate getting it sorted out very quickly. Differential Revision: http://reviews.llvm.org/D7293 llvm-svn: 227669
2015-01-31 11:43:40 +08:00
TTI = IgnoreTargetInfo
? nullptr
[multiversion] Thread a function argument through all the callers of the getTTI method used to get an actual TTI object. No functionality changed. This just threads the argument and ensures code like the inliner can correctly look up the callee's TTI rather than using a fixed one. The next change will use this to implement per-function subtarget usage by TTI. The changes after that should eliminate the need for FTTI as that will have become the default. llvm-svn: 227730
2015-02-01 20:01:35 +08:00
: &P->getAnalysis<TargetTransformInfoWrapperPass>().getTTI(F);
Add the function "vectorizeBasicBlock" which allow users vectorize a BasicBlock in other passes, e.g. we can call vectorizeBasicBlock in the loop unroll pass right after the loop is unrolled. llvm-svn: 154089
2012-04-05 16:05:16 +08:00
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
typedef std::pair<Value *, Value *> ValuePair;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
typedef std::pair<ValuePair, int> ValuePairWithCost;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
typedef std::pair<ValuePair, size_t> ValuePairWithDepth;
typedef std::pair<ValuePair, ValuePair> VPPair; // A ValuePair pair
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
typedef std::pair<VPPair, unsigned> VPPairWithType;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
AliasAnalysis *AA;
BBVectorize should ignore unreachable blocks. Unreachable blocks can have invalid instructions. For example, jump threading can produce self-referential instructions in unreachable blocks. Also, we should not be spending time optimizing unreachable code. Fixes PR14133. llvm-svn: 166423
2012-10-23 02:00:55 +08:00
DominatorTree *DT;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
ScalarEvolution *SE;
[PM/AA] Explicitly depend on TLI rather than getting it out of the AliasAnalysis. Same as the other commits, the TLI access from an alias analysis is going away and isn't very clean -- it is better to explicitly mark the dependencies. llvm-svn: 244785
2015-08-13 02:06:08 +08:00
const TargetLibraryInfo *TLI;
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
const TargetTransformInfo *TTI;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// FIXME: const correct?
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
bool vectorizePairs(BasicBlock &BB, bool NonPow2Len = false);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
bool getCandidatePairs(BasicBlock &BB,
BasicBlock::iterator &Start,
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
DenseSet<ValuePair> &FixedOrderPairs,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
DenseMap<ValuePair, int> &CandidatePairCostSavings,
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
std::vector<Value *> &PairableInsts, bool NonPow2Len);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
// FIXME: The current implementation does not account for pairs that
// are connected in multiple ways. For example:
// C1 = A1 / A2; C2 = A2 / A1 (which may be both direct and a swap)
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
enum PairConnectionType {
PairConnectionDirect,
PairConnectionSwap,
PairConnectionSplat
};
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
void computeConnectedPairs(
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
DenseSet<ValuePair> &CandidatePairsSet,
std::vector<Value *> &PairableInsts,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
void buildDepMap(BasicBlock &BB,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
std::vector<Value *> &PairableInsts,
DenseSet<ValuePair> &PairableInstUsers);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
void choosePairs(DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseSet<ValuePair> &CandidatePairsSet,
DenseMap<ValuePair, int> &CandidatePairCostSavings,
std::vector<Value *> &PairableInsts,
DenseSet<ValuePair> &FixedOrderPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairs,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairDeps,
DenseSet<ValuePair> &PairableInstUsers,
DenseMap<Value *, Value *>& ChosenPairs);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
void fuseChosenPairs(BasicBlock &BB,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
std::vector<Value *> &PairableInsts,
DenseMap<Value *, Value *>& ChosenPairs,
DenseSet<ValuePair> &FixedOrderPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairs,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairDeps);
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool isInstVectorizable(Instruction *I, bool &IsSimpleLoadStore);
bool areInstsCompatible(Instruction *I, Instruction *J,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
bool IsSimpleLoadStore, bool NonPow2Len,
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
int &CostSavings, int &FixedOrder);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool trackUsesOfI(DenseSet<Value *> &Users,
AliasSetTracker &WriteSet, Instruction *I,
Instruction *J, bool UpdateUsers = true,
[C++] Use 'nullptr'. Transforms edition. llvm-svn: 207196
2014-04-25 13:29:35 +08:00
DenseSet<ValuePair> *LoadMoveSetPairs = nullptr);
fix indentation llvm-svn: 149857
2012-02-06 13:29:32 +08:00
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
void computePairsConnectedTo(
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
DenseSet<ValuePair> &CandidatePairsSet,
std::vector<Value *> &PairableInsts,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes,
ValuePair P);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool pairsConflict(ValuePair P, ValuePair Q,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseSet<ValuePair> &PairableInstUsers,
DenseMap<ValuePair, std::vector<ValuePair> >
[C++] Use 'nullptr'. Transforms edition. llvm-svn: 207196
2014-04-25 13:29:35 +08:00
*PairableInstUserMap = nullptr,
DenseSet<VPPair> *PairableInstUserPairSet = nullptr);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool pairWillFormCycle(ValuePair P,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<ValuePair, std::vector<ValuePair> > &PairableInstUsers,
DenseSet<ValuePair> &CurrentPairs);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
void pruneDAGFor(
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
std::vector<Value *> &PairableInsts,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairs,
DenseSet<ValuePair> &PairableInstUsers,
DenseMap<ValuePair, std::vector<ValuePair> > &PairableInstUserMap,
DenseSet<VPPair> &PairableInstUserPairSet,
DenseMap<Value *, Value *> &ChosenPairs,
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
DenseMap<ValuePair, size_t> &DAG,
DenseSet<ValuePair> &PrunedDAG, ValuePair J,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
bool UseCycleCheck);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
void buildInitialDAGFor(
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
DenseSet<ValuePair> &CandidatePairsSet,
std::vector<Value *> &PairableInsts,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairs,
DenseSet<ValuePair> &PairableInstUsers,
DenseMap<Value *, Value *> &ChosenPairs,
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
DenseMap<ValuePair, size_t> &DAG, ValuePair J);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
void findBestDAGFor(
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
DenseSet<ValuePair> &CandidatePairsSet,
DenseMap<ValuePair, int> &CandidatePairCostSavings,
std::vector<Value *> &PairableInsts,
DenseSet<ValuePair> &FixedOrderPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairs,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairDeps,
DenseSet<ValuePair> &PairableInstUsers,
DenseMap<ValuePair, std::vector<ValuePair> > &PairableInstUserMap,
DenseSet<VPPair> &PairableInstUserPairSet,
DenseMap<Value *, Value *> &ChosenPairs,
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
DenseSet<ValuePair> &BestDAG, size_t &BestMaxDepth,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
int &BestEffSize, Value *II, std::vector<Value *>&JJ,
bool UseCycleCheck);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Value *getReplacementPointerInput(LLVMContext& Context, Instruction *I,
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
Instruction *J, unsigned o);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
void fillNewShuffleMask(LLVMContext& Context, Instruction *J,
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
unsigned MaskOffset, unsigned NumInElem,
unsigned NumInElem1, unsigned IdxOffset,
std::vector<Constant*> &Mask);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Value *getReplacementShuffleMask(LLVMContext& Context, Instruction *I,
Instruction *J);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
bool expandIEChain(LLVMContext& Context, Instruction *I, Instruction *J,
unsigned o, Value *&LOp, unsigned numElemL,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
Type *ArgTypeL, Type *ArgTypeR, bool IBeforeJ,
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
unsigned IdxOff = 0);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Value *getReplacementInput(LLVMContext& Context, Instruction *I,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
Instruction *J, unsigned o, bool IBeforeJ);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
void getReplacementInputsForPair(LLVMContext& Context, Instruction *I,
Use SmallVectorImpl& instead of SmallVector to avoid repeating small vector size. llvm-svn: 186274
2013-07-14 12:42:23 +08:00
Instruction *J, SmallVectorImpl<Value *> &ReplacedOperands,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
bool IBeforeJ);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
void replaceOutputsOfPair(LLVMContext& Context, Instruction *I,
Instruction *J, Instruction *K,
Instruction *&InsertionPt, Instruction *&K1,
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
Instruction *&K2);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
void collectPairLoadMoveSet(BasicBlock &BB,
DenseMap<Value *, Value *> &ChosenPairs,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > &LoadMoveSet,
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
DenseSet<ValuePair> &LoadMoveSetPairs,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Instruction *I);
void collectLoadMoveSet(BasicBlock &BB,
std::vector<Value *> &PairableInsts,
DenseMap<Value *, Value *> &ChosenPairs,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > &LoadMoveSet,
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
DenseSet<ValuePair> &LoadMoveSetPairs);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool canMoveUsesOfIAfterJ(BasicBlock &BB,
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
DenseSet<ValuePair> &LoadMoveSetPairs,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Instruction *I, Instruction *J);
void moveUsesOfIAfterJ(BasicBlock &BB,
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
DenseSet<ValuePair> &LoadMoveSetPairs,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Instruction *&InsertionPt,
Instruction *I, Instruction *J);
Add the function "vectorizeBasicBlock" which allow users vectorize a BasicBlock in other passes, e.g. we can call vectorizeBasicBlock in the loop unroll pass right after the loop is unrolled. llvm-svn: 154089
2012-04-05 16:05:16 +08:00
bool vectorizeBB(BasicBlock &BB) {
Re-commit optimization bisect support (r267022) without new pass manager support. The original commit was reverted because of a buildbot problem with LazyCallGraph::SCC handling (not related to the OptBisect handling). Differential Revision: http://reviews.llvm.org/D19172 llvm-svn: 267231
2016-04-23 06:06:11 +08:00
if (skipBasicBlock(BB))
Disable most IR-level transform passes on functions marked 'optnone'. Ideally only those transform passes that run at -O0 remain enabled, in reality we get as close as we reasonably can. Passes are responsible for disabling themselves, it's not the job of the pass manager to do it for them. llvm-svn: 200892
2014-02-06 08:07:05 +08:00
return false;
BBVectorize should ignore unreachable blocks. Unreachable blocks can have invalid instructions. For example, jump threading can produce self-referential instructions in unreachable blocks. Also, we should not be spending time optimizing unreachable code. Fixes PR14133. llvm-svn: 166423
2012-10-23 02:00:55 +08:00
if (!DT->isReachableFromEntry(&BB)) {
DEBUG(dbgs() << "BBV: skipping unreachable " << BB.getName() <<
" in " << BB.getParent()->getName() << "\n");
return false;
}
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
DEBUG(if (TTI) dbgs() << "BBV: using target information\n");
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool changed = false;
// Iterate a sufficient number of times to merge types of size 1 bit,
// then 2 bits, then 4, etc. up to half of the target vector width of the
// target vector register.
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
unsigned n = 1;
for (unsigned v = 2;
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
(TTI || v <= Config.VectorBits) &&
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
(!Config.MaxIter || n <= Config.MaxIter);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
v *= 2, ++n) {
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
DEBUG(dbgs() << "BBV: fusing loop #" << n <<
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
" for " << BB.getName() << " in " <<
BB.getParent()->getName() << "...\n");
if (vectorizePairs(BB))
changed = true;
else
break;
}
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
if (changed && !Pow2LenOnly) {
++n;
for (; !Config.MaxIter || n <= Config.MaxIter; ++n) {
DEBUG(dbgs() << "BBV: fusing for non-2^n-length vectors loop #: " <<
n << " for " << BB.getName() << " in " <<
BB.getParent()->getName() << "...\n");
if (!vectorizePairs(BB, true)) break;
}
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DEBUG(dbgs() << "BBV: done!\n");
return changed;
}
[C++11] Add 'override' keyword to virtual methods that override their base class. llvm-svn: 202953
2014-03-05 17:10:37 +08:00
bool runOnBasicBlock(BasicBlock &BB) override {
Disable most IR-level transform passes on functions marked 'optnone'. Ideally only those transform passes that run at -O0 remain enabled, in reality we get as close as we reasonably can. Passes are responsible for disabling themselves, it's not the job of the pass manager to do it for them. llvm-svn: 200892
2014-02-06 08:07:05 +08:00
// OptimizeNone check deferred to vectorizeBB().
[PM/AA] Rebuild LLVM's alias analysis infrastructure in a way compatible with the new pass manager, and no longer relying on analysis groups. This builds essentially a ground-up new AA infrastructure stack for LLVM. The core ideas are the same that are used throughout the new pass manager: type erased polymorphism and direct composition. The design is as follows: - FunctionAAResults is a type-erasing alias analysis results aggregation interface to walk a single query across a range of results from different alias analyses. Currently this is function-specific as we always assume that aliasing queries are *within* a function. - AAResultBase is a CRTP utility providing stub implementations of various parts of the alias analysis result concept, notably in several cases in terms of other more general parts of the interface. This can be used to implement only a narrow part of the interface rather than the entire interface. This isn't really ideal, this logic should be hoisted into FunctionAAResults as currently it will cause a significant amount of redundant work, but it faithfully models the behavior of the prior infrastructure. - All the alias analysis passes are ported to be wrapper passes for the legacy PM and new-style analysis passes for the new PM with a shared result object. In some cases (most notably CFL), this is an extremely naive approach that we should revisit when we can specialize for the new pass manager. - BasicAA has been restructured to reflect that it is much more fundamentally a function analysis because it uses dominator trees and loop info that need to be constructed for each function. All of the references to getting alias analysis results have been updated to use the new aggregation interface. All the preservation and other pass management code has been updated accordingly. The way the FunctionAAResultsWrapperPass works is to detect the available alias analyses when run, and add them to the results object. This means that we should be able to continue to respect when various passes are added to the pipeline, for example adding CFL or adding TBAA passes should just cause their results to be available and to get folded into this. The exception to this rule is BasicAA which really needs to be a function pass due to using dominator trees and loop info. As a consequence, the FunctionAAResultsWrapperPass directly depends on BasicAA and always includes it in the aggregation. This has significant implications for preserving analyses. Generally, most passes shouldn't bother preserving FunctionAAResultsWrapperPass because rebuilding the results just updates the set of known AA passes. The exception to this rule are LoopPass instances which need to preserve all the function analyses that the loop pass manager will end up needing. This means preserving both BasicAAWrapperPass and the aggregating FunctionAAResultsWrapperPass. Now, when preserving an alias analysis, you do so by directly preserving that analysis. This is only necessary for non-immutable-pass-provided alias analyses though, and there are only three of interest: BasicAA, GlobalsAA (formerly GlobalsModRef), and SCEVAA. Usually BasicAA is preserved when needed because it (like DominatorTree and LoopInfo) is marked as a CFG-only pass. I've expanded GlobalsAA into the preserved set everywhere we previously were preserving all of AliasAnalysis, and I've added SCEVAA in the intersection of that with where we preserve SCEV itself. One significant challenge to all of this is that the CGSCC passes were actually using the alias analysis implementations by taking advantage of a pretty amazing set of loop holes in the old pass manager's analysis management code which allowed analysis groups to slide through in many cases. Moving away from analysis groups makes this problem much more obvious. To fix it, I've leveraged the flexibility the design of the new PM components provides to just directly construct the relevant alias analyses for the relevant functions in the IPO passes that need them. This is a bit hacky, but should go away with the new pass manager, and is already in many ways cleaner than the prior state. Another significant challenge is that various facilities of the old alias analysis infrastructure just don't fit any more. The most significant of these is the alias analysis 'counter' pass. That pass relied on the ability to snoop on AA queries at different points in the analysis group chain. Instead, I'm planning to build printing functionality directly into the aggregation layer. I've not included that in this patch merely to keep it smaller. Note that all of this needs a nearly complete rewrite of the AA documentation. I'm planning to do that, but I'd like to make sure the new design settles, and to flesh out a bit more of what it looks like in the new pass manager first. Differential Revision: http://reviews.llvm.org/D12080 llvm-svn: 247167
2015-09-10 01:55:00 +08:00
AA = &getAnalysis<AAResultsWrapperPass>().getAAResults();
[PM] Split DominatorTree into a concrete analysis result object which can be used by both the new pass manager and the old. This removes it from any of the virtual mess of the pass interfaces and lets it derive cleanly from the DominatorTreeBase<> template. In turn, tons of boilerplate interface can be nuked and it turns into a very straightforward extension of the base DominatorTree interface. The old analysis pass is now a simple wrapper. The names and style of this split should match the split between CallGraph and CallGraphWrapperPass. All of the users of DominatorTree have been updated to match using many of the same tricks as with CallGraph. The goal is that the common type remains the resulting DominatorTree rather than the pass. This will make subsequent work toward the new pass manager significantly easier. Also in numerous places things became cleaner because I switched from re-running the pass (!!! mid way through some other passes run!!!) to directly recomputing the domtree. llvm-svn: 199104
2014-01-13 21:07:17 +08:00
DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
[PM] Port ScalarEvolution to the new pass manager. This change makes ScalarEvolution a stand-alone object and just produces one from a pass as needed. Making this work well requires making the object movable, using references instead of overwritten pointers in a number of places, and other refactorings. I've also wired it up to the new pass manager and added a RUN line to a test to exercise it under the new pass manager. This includes basic printing support much like with other analyses. But there is a big and somewhat scary change here. Prior to this patch ScalarEvolution was never *actually* invalidated!!! Re-running the pass just re-wired up the various other analyses and didn't remove any of the existing entries in the SCEV caches or clear out anything at all. This might seem OK as everything in SCEV that can uses ValueHandles to track updates to the values that serve as SCEV keys. However, this still means that as we ran SCEV over each function in the module, we kept accumulating more and more SCEVs into the cache. At the end, we would have a SCEV cache with every value that we ever needed a SCEV for in the entire module!!! Yowzers. The releaseMemory routine would dump all of this, but that isn't realy called during normal runs of the pipeline as far as I can see. To make matters worse, there *is* actually a key that we don't update with value handles -- there is a map keyed off of Loop*s. Because LoopInfo *does* release its memory from run to run, it is entirely possible to run SCEV over one function, then over another function, and then lookup a Loop* from the second function but find an entry inserted for the first function! Ouch. To make matters still worse, there are plenty of updates that *don't* trip a value handle. It seems incredibly unlikely that today GVN or another pass that invalidates SCEV can update values in *just* such a way that a subsequent run of SCEV will incorrectly find lookups in a cache, but it is theoretically possible and would be a nightmare to debug. With this refactoring, I've fixed all this by actually destroying and recreating the ScalarEvolution object from run to run. Technically, this could increase the amount of malloc traffic we see, but then again it is also technically correct. ;] I don't actually think we're suffering from tons of malloc traffic from SCEV because if we were, the fact that we never clear the memory would seem more likely to have come up as an actual problem before now. So, I've made the simple fix here. If in fact there are serious issues with too much allocation and deallocation, I can work on a clever fix that preserves the allocations (while clearing the data) between each run, but I'd prefer to do that kind of optimization with a test case / benchmark that shows why we need such cleverness (and that can test that we actually make it faster). It's possible that this will make some things faster by making the SCEV caches have higher locality (due to being significantly smaller) so until there is a clear benchmark, I think the simple change is best. Differential Revision: http://reviews.llvm.org/D12063 llvm-svn: 245193
2015-08-17 10:08:17 +08:00
SE = &getAnalysis<ScalarEvolutionWrapperPass>().getSE();
[PM/AA] Explicitly depend on TLI rather than getting it out of the AliasAnalysis. Same as the other commits, the TLI access from an alias analysis is going away and isn't very clean -- it is better to explicitly mark the dependencies. llvm-svn: 244785
2015-08-13 02:06:08 +08:00
TLI = &getAnalysis<TargetLibraryInfoWrapperPass>().getTLI();
[PM] Change the core design of the TTI analysis to use a polymorphic type erased interface and a single analysis pass rather than an extremely complex analysis group. The end result is that the TTI analysis can contain a type erased implementation that supports the polymorphic TTI interface. We can build one from a target-specific implementation or from a dummy one in the IR. I've also factored all of the code into "mix-in"-able base classes, including CRTP base classes to facilitate calling back up to the most specialized form when delegating horizontally across the surface. These aren't as clean as I would like and I'm planning to work on cleaning some of this up, but I wanted to start by putting into the right form. There are a number of reasons for this change, and this particular design. The first and foremost reason is that an analysis group is complete overkill, and the chaining delegation strategy was so opaque, confusing, and high overhead that TTI was suffering greatly for it. Several of the TTI functions had failed to be implemented in all places because of the chaining-based delegation making there be no checking of this. A few other functions were implemented with incorrect delegation. The message to me was very clear working on this -- the delegation and analysis group structure was too confusing to be useful here. The other reason of course is that this is *much* more natural fit for the new pass manager. This will lay the ground work for a type-erased per-function info object that can look up the correct subtarget and even cache it. Yet another benefit is that this will significantly simplify the interaction of the pass managers and the TargetMachine. See the future work below. The downside of this change is that it is very, very verbose. I'm going to work to improve that, but it is somewhat an implementation necessity in C++ to do type erasure. =/ I discussed this design really extensively with Eric and Hal prior to going down this path, and afterward showed them the result. No one was really thrilled with it, but there doesn't seem to be a substantially better alternative. Using a base class and virtual method dispatch would make the code much shorter, but as discussed in the update to the programmer's manual and elsewhere, a polymorphic interface feels like the more principled approach even if this is perhaps the least compelling example of it. ;] Ultimately, there is still a lot more to be done here, but this was the huge chunk that I couldn't really split things out of because this was the interface change to TTI. I've tried to minimize all the other parts of this. The follow up work should include at least: 1) Improving the TargetMachine interface by having it directly return a TTI object. Because we have a non-pass object with value semantics and an internal type erasure mechanism, we can narrow the interface of the TargetMachine to *just* do what we need: build and return a TTI object that we can then insert into the pass pipeline. 2) Make the TTI object be fully specialized for a particular function. This will include splitting off a minimal form of it which is sufficient for the inliner and the old pass manager. 3) Add a new pass manager analysis which produces TTI objects from the target machine for each function. This may actually be done as part of #2 in order to use the new analysis to implement #2. 4) Work on narrowing the API between TTI and the targets so that it is easier to understand and less verbose to type erase. 5) Work on narrowing the API between TTI and its clients so that it is easier to understand and less verbose to forward. 6) Try to improve the CRTP-based delegation. I feel like this code is just a bit messy and exacerbating the complexity of implementing the TTI in each target. Many thanks to Eric and Hal for their help here. I ended up blocked on this somewhat more abruptly than I expected, and so I appreciate getting it sorted out very quickly. Differential Revision: http://reviews.llvm.org/D7293 llvm-svn: 227669
2015-01-31 11:43:40 +08:00
TTI = IgnoreTargetInfo
? nullptr
[multiversion] Thread a function argument through all the callers of the getTTI method used to get an actual TTI object. No functionality changed. This just threads the argument and ensures code like the inliner can correctly look up the callee's TTI rather than using a fixed one. The next change will use this to implement per-function subtarget usage by TTI. The changes after that should eliminate the need for FTTI as that will have become the default. llvm-svn: 227730
2015-02-01 20:01:35 +08:00
: &getAnalysis<TargetTransformInfoWrapperPass>().getTTI(
*BB.getParent());
Add the function "vectorizeBasicBlock" which allow users vectorize a BasicBlock in other passes, e.g. we can call vectorizeBasicBlock in the loop unroll pass right after the loop is unrolled. llvm-svn: 154089
2012-04-05 16:05:16 +08:00
return vectorizeBB(BB);
}
[C++11] Add 'override' keyword to virtual methods that override their base class. llvm-svn: 202953
2014-03-05 17:10:37 +08:00
void getAnalysisUsage(AnalysisUsage &AU) const override {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BasicBlockPass::getAnalysisUsage(AU);
[PM/AA] Rebuild LLVM's alias analysis infrastructure in a way compatible with the new pass manager, and no longer relying on analysis groups. This builds essentially a ground-up new AA infrastructure stack for LLVM. The core ideas are the same that are used throughout the new pass manager: type erased polymorphism and direct composition. The design is as follows: - FunctionAAResults is a type-erasing alias analysis results aggregation interface to walk a single query across a range of results from different alias analyses. Currently this is function-specific as we always assume that aliasing queries are *within* a function. - AAResultBase is a CRTP utility providing stub implementations of various parts of the alias analysis result concept, notably in several cases in terms of other more general parts of the interface. This can be used to implement only a narrow part of the interface rather than the entire interface. This isn't really ideal, this logic should be hoisted into FunctionAAResults as currently it will cause a significant amount of redundant work, but it faithfully models the behavior of the prior infrastructure. - All the alias analysis passes are ported to be wrapper passes for the legacy PM and new-style analysis passes for the new PM with a shared result object. In some cases (most notably CFL), this is an extremely naive approach that we should revisit when we can specialize for the new pass manager. - BasicAA has been restructured to reflect that it is much more fundamentally a function analysis because it uses dominator trees and loop info that need to be constructed for each function. All of the references to getting alias analysis results have been updated to use the new aggregation interface. All the preservation and other pass management code has been updated accordingly. The way the FunctionAAResultsWrapperPass works is to detect the available alias analyses when run, and add them to the results object. This means that we should be able to continue to respect when various passes are added to the pipeline, for example adding CFL or adding TBAA passes should just cause their results to be available and to get folded into this. The exception to this rule is BasicAA which really needs to be a function pass due to using dominator trees and loop info. As a consequence, the FunctionAAResultsWrapperPass directly depends on BasicAA and always includes it in the aggregation. This has significant implications for preserving analyses. Generally, most passes shouldn't bother preserving FunctionAAResultsWrapperPass because rebuilding the results just updates the set of known AA passes. The exception to this rule are LoopPass instances which need to preserve all the function analyses that the loop pass manager will end up needing. This means preserving both BasicAAWrapperPass and the aggregating FunctionAAResultsWrapperPass. Now, when preserving an alias analysis, you do so by directly preserving that analysis. This is only necessary for non-immutable-pass-provided alias analyses though, and there are only three of interest: BasicAA, GlobalsAA (formerly GlobalsModRef), and SCEVAA. Usually BasicAA is preserved when needed because it (like DominatorTree and LoopInfo) is marked as a CFG-only pass. I've expanded GlobalsAA into the preserved set everywhere we previously were preserving all of AliasAnalysis, and I've added SCEVAA in the intersection of that with where we preserve SCEV itself. One significant challenge to all of this is that the CGSCC passes were actually using the alias analysis implementations by taking advantage of a pretty amazing set of loop holes in the old pass manager's analysis management code which allowed analysis groups to slide through in many cases. Moving away from analysis groups makes this problem much more obvious. To fix it, I've leveraged the flexibility the design of the new PM components provides to just directly construct the relevant alias analyses for the relevant functions in the IPO passes that need them. This is a bit hacky, but should go away with the new pass manager, and is already in many ways cleaner than the prior state. Another significant challenge is that various facilities of the old alias analysis infrastructure just don't fit any more. The most significant of these is the alias analysis 'counter' pass. That pass relied on the ability to snoop on AA queries at different points in the analysis group chain. Instead, I'm planning to build printing functionality directly into the aggregation layer. I've not included that in this patch merely to keep it smaller. Note that all of this needs a nearly complete rewrite of the AA documentation. I'm planning to do that, but I'd like to make sure the new design settles, and to flesh out a bit more of what it looks like in the new pass manager first. Differential Revision: http://reviews.llvm.org/D12080 llvm-svn: 247167
2015-09-10 01:55:00 +08:00
AU.addRequired<AAResultsWrapperPass>();
[PM] Split DominatorTree into a concrete analysis result object which can be used by both the new pass manager and the old. This removes it from any of the virtual mess of the pass interfaces and lets it derive cleanly from the DominatorTreeBase<> template. In turn, tons of boilerplate interface can be nuked and it turns into a very straightforward extension of the base DominatorTree interface. The old analysis pass is now a simple wrapper. The names and style of this split should match the split between CallGraph and CallGraphWrapperPass. All of the users of DominatorTree have been updated to match using many of the same tricks as with CallGraph. The goal is that the common type remains the resulting DominatorTree rather than the pass. This will make subsequent work toward the new pass manager significantly easier. Also in numerous places things became cleaner because I switched from re-running the pass (!!! mid way through some other passes run!!!) to directly recomputing the domtree. llvm-svn: 199104
2014-01-13 21:07:17 +08:00
AU.addRequired<DominatorTreeWrapperPass>();
[PM] Port ScalarEvolution to the new pass manager. This change makes ScalarEvolution a stand-alone object and just produces one from a pass as needed. Making this work well requires making the object movable, using references instead of overwritten pointers in a number of places, and other refactorings. I've also wired it up to the new pass manager and added a RUN line to a test to exercise it under the new pass manager. This includes basic printing support much like with other analyses. But there is a big and somewhat scary change here. Prior to this patch ScalarEvolution was never *actually* invalidated!!! Re-running the pass just re-wired up the various other analyses and didn't remove any of the existing entries in the SCEV caches or clear out anything at all. This might seem OK as everything in SCEV that can uses ValueHandles to track updates to the values that serve as SCEV keys. However, this still means that as we ran SCEV over each function in the module, we kept accumulating more and more SCEVs into the cache. At the end, we would have a SCEV cache with every value that we ever needed a SCEV for in the entire module!!! Yowzers. The releaseMemory routine would dump all of this, but that isn't realy called during normal runs of the pipeline as far as I can see. To make matters worse, there *is* actually a key that we don't update with value handles -- there is a map keyed off of Loop*s. Because LoopInfo *does* release its memory from run to run, it is entirely possible to run SCEV over one function, then over another function, and then lookup a Loop* from the second function but find an entry inserted for the first function! Ouch. To make matters still worse, there are plenty of updates that *don't* trip a value handle. It seems incredibly unlikely that today GVN or another pass that invalidates SCEV can update values in *just* such a way that a subsequent run of SCEV will incorrectly find lookups in a cache, but it is theoretically possible and would be a nightmare to debug. With this refactoring, I've fixed all this by actually destroying and recreating the ScalarEvolution object from run to run. Technically, this could increase the amount of malloc traffic we see, but then again it is also technically correct. ;] I don't actually think we're suffering from tons of malloc traffic from SCEV because if we were, the fact that we never clear the memory would seem more likely to have come up as an actual problem before now. So, I've made the simple fix here. If in fact there are serious issues with too much allocation and deallocation, I can work on a clever fix that preserves the allocations (while clearing the data) between each run, but I'd prefer to do that kind of optimization with a test case / benchmark that shows why we need such cleverness (and that can test that we actually make it faster). It's possible that this will make some things faster by making the SCEV caches have higher locality (due to being significantly smaller) so until there is a clear benchmark, I think the simple change is best. Differential Revision: http://reviews.llvm.org/D12063 llvm-svn: 245193
2015-08-17 10:08:17 +08:00
AU.addRequired<ScalarEvolutionWrapperPass>();
[PM/AA] Explicitly depend on TLI rather than getting it out of the AliasAnalysis. Same as the other commits, the TLI access from an alias analysis is going away and isn't very clean -- it is better to explicitly mark the dependencies. llvm-svn: 244785
2015-08-13 02:06:08 +08:00
AU.addRequired<TargetLibraryInfoWrapperPass>();
[PM] Change the core design of the TTI analysis to use a polymorphic type erased interface and a single analysis pass rather than an extremely complex analysis group. The end result is that the TTI analysis can contain a type erased implementation that supports the polymorphic TTI interface. We can build one from a target-specific implementation or from a dummy one in the IR. I've also factored all of the code into "mix-in"-able base classes, including CRTP base classes to facilitate calling back up to the most specialized form when delegating horizontally across the surface. These aren't as clean as I would like and I'm planning to work on cleaning some of this up, but I wanted to start by putting into the right form. There are a number of reasons for this change, and this particular design. The first and foremost reason is that an analysis group is complete overkill, and the chaining delegation strategy was so opaque, confusing, and high overhead that TTI was suffering greatly for it. Several of the TTI functions had failed to be implemented in all places because of the chaining-based delegation making there be no checking of this. A few other functions were implemented with incorrect delegation. The message to me was very clear working on this -- the delegation and analysis group structure was too confusing to be useful here. The other reason of course is that this is *much* more natural fit for the new pass manager. This will lay the ground work for a type-erased per-function info object that can look up the correct subtarget and even cache it. Yet another benefit is that this will significantly simplify the interaction of the pass managers and the TargetMachine. See the future work below. The downside of this change is that it is very, very verbose. I'm going to work to improve that, but it is somewhat an implementation necessity in C++ to do type erasure. =/ I discussed this design really extensively with Eric and Hal prior to going down this path, and afterward showed them the result. No one was really thrilled with it, but there doesn't seem to be a substantially better alternative. Using a base class and virtual method dispatch would make the code much shorter, but as discussed in the update to the programmer's manual and elsewhere, a polymorphic interface feels like the more principled approach even if this is perhaps the least compelling example of it. ;] Ultimately, there is still a lot more to be done here, but this was the huge chunk that I couldn't really split things out of because this was the interface change to TTI. I've tried to minimize all the other parts of this. The follow up work should include at least: 1) Improving the TargetMachine interface by having it directly return a TTI object. Because we have a non-pass object with value semantics and an internal type erasure mechanism, we can narrow the interface of the TargetMachine to *just* do what we need: build and return a TTI object that we can then insert into the pass pipeline. 2) Make the TTI object be fully specialized for a particular function. This will include splitting off a minimal form of it which is sufficient for the inliner and the old pass manager. 3) Add a new pass manager analysis which produces TTI objects from the target machine for each function. This may actually be done as part of #2 in order to use the new analysis to implement #2. 4) Work on narrowing the API between TTI and the targets so that it is easier to understand and less verbose to type erase. 5) Work on narrowing the API between TTI and its clients so that it is easier to understand and less verbose to forward. 6) Try to improve the CRTP-based delegation. I feel like this code is just a bit messy and exacerbating the complexity of implementing the TTI in each target. Many thanks to Eric and Hal for their help here. I ended up blocked on this somewhat more abruptly than I expected, and so I appreciate getting it sorted out very quickly. Differential Revision: http://reviews.llvm.org/D7293 llvm-svn: 227669
2015-01-31 11:43:40 +08:00
AU.addRequired<TargetTransformInfoWrapperPass>();
[PM] Split DominatorTree into a concrete analysis result object which can be used by both the new pass manager and the old. This removes it from any of the virtual mess of the pass interfaces and lets it derive cleanly from the DominatorTreeBase<> template. In turn, tons of boilerplate interface can be nuked and it turns into a very straightforward extension of the base DominatorTree interface. The old analysis pass is now a simple wrapper. The names and style of this split should match the split between CallGraph and CallGraphWrapperPass. All of the users of DominatorTree have been updated to match using many of the same tricks as with CallGraph. The goal is that the common type remains the resulting DominatorTree rather than the pass. This will make subsequent work toward the new pass manager significantly easier. Also in numerous places things became cleaner because I switched from re-running the pass (!!! mid way through some other passes run!!!) to directly recomputing the domtree. llvm-svn: 199104
2014-01-13 21:07:17 +08:00
AU.addPreserved<DominatorTreeWrapperPass>();
[PM/AA] Rebuild LLVM's alias analysis infrastructure in a way compatible with the new pass manager, and no longer relying on analysis groups. This builds essentially a ground-up new AA infrastructure stack for LLVM. The core ideas are the same that are used throughout the new pass manager: type erased polymorphism and direct composition. The design is as follows: - FunctionAAResults is a type-erasing alias analysis results aggregation interface to walk a single query across a range of results from different alias analyses. Currently this is function-specific as we always assume that aliasing queries are *within* a function. - AAResultBase is a CRTP utility providing stub implementations of various parts of the alias analysis result concept, notably in several cases in terms of other more general parts of the interface. This can be used to implement only a narrow part of the interface rather than the entire interface. This isn't really ideal, this logic should be hoisted into FunctionAAResults as currently it will cause a significant amount of redundant work, but it faithfully models the behavior of the prior infrastructure. - All the alias analysis passes are ported to be wrapper passes for the legacy PM and new-style analysis passes for the new PM with a shared result object. In some cases (most notably CFL), this is an extremely naive approach that we should revisit when we can specialize for the new pass manager. - BasicAA has been restructured to reflect that it is much more fundamentally a function analysis because it uses dominator trees and loop info that need to be constructed for each function. All of the references to getting alias analysis results have been updated to use the new aggregation interface. All the preservation and other pass management code has been updated accordingly. The way the FunctionAAResultsWrapperPass works is to detect the available alias analyses when run, and add them to the results object. This means that we should be able to continue to respect when various passes are added to the pipeline, for example adding CFL or adding TBAA passes should just cause their results to be available and to get folded into this. The exception to this rule is BasicAA which really needs to be a function pass due to using dominator trees and loop info. As a consequence, the FunctionAAResultsWrapperPass directly depends on BasicAA and always includes it in the aggregation. This has significant implications for preserving analyses. Generally, most passes shouldn't bother preserving FunctionAAResultsWrapperPass because rebuilding the results just updates the set of known AA passes. The exception to this rule are LoopPass instances which need to preserve all the function analyses that the loop pass manager will end up needing. This means preserving both BasicAAWrapperPass and the aggregating FunctionAAResultsWrapperPass. Now, when preserving an alias analysis, you do so by directly preserving that analysis. This is only necessary for non-immutable-pass-provided alias analyses though, and there are only three of interest: BasicAA, GlobalsAA (formerly GlobalsModRef), and SCEVAA. Usually BasicAA is preserved when needed because it (like DominatorTree and LoopInfo) is marked as a CFG-only pass. I've expanded GlobalsAA into the preserved set everywhere we previously were preserving all of AliasAnalysis, and I've added SCEVAA in the intersection of that with where we preserve SCEV itself. One significant challenge to all of this is that the CGSCC passes were actually using the alias analysis implementations by taking advantage of a pretty amazing set of loop holes in the old pass manager's analysis management code which allowed analysis groups to slide through in many cases. Moving away from analysis groups makes this problem much more obvious. To fix it, I've leveraged the flexibility the design of the new PM components provides to just directly construct the relevant alias analyses for the relevant functions in the IPO passes that need them. This is a bit hacky, but should go away with the new pass manager, and is already in many ways cleaner than the prior state. Another significant challenge is that various facilities of the old alias analysis infrastructure just don't fit any more. The most significant of these is the alias analysis 'counter' pass. That pass relied on the ability to snoop on AA queries at different points in the analysis group chain. Instead, I'm planning to build printing functionality directly into the aggregation layer. I've not included that in this patch merely to keep it smaller. Note that all of this needs a nearly complete rewrite of the AA documentation. I'm planning to do that, but I'd like to make sure the new design settles, and to flesh out a bit more of what it looks like in the new pass manager first. Differential Revision: http://reviews.llvm.org/D12080 llvm-svn: 247167
2015-09-10 01:55:00 +08:00
AU.addPreserved<GlobalsAAWrapperPass>();
[PM] Port ScalarEvolution to the new pass manager. This change makes ScalarEvolution a stand-alone object and just produces one from a pass as needed. Making this work well requires making the object movable, using references instead of overwritten pointers in a number of places, and other refactorings. I've also wired it up to the new pass manager and added a RUN line to a test to exercise it under the new pass manager. This includes basic printing support much like with other analyses. But there is a big and somewhat scary change here. Prior to this patch ScalarEvolution was never *actually* invalidated!!! Re-running the pass just re-wired up the various other analyses and didn't remove any of the existing entries in the SCEV caches or clear out anything at all. This might seem OK as everything in SCEV that can uses ValueHandles to track updates to the values that serve as SCEV keys. However, this still means that as we ran SCEV over each function in the module, we kept accumulating more and more SCEVs into the cache. At the end, we would have a SCEV cache with every value that we ever needed a SCEV for in the entire module!!! Yowzers. The releaseMemory routine would dump all of this, but that isn't realy called during normal runs of the pipeline as far as I can see. To make matters worse, there *is* actually a key that we don't update with value handles -- there is a map keyed off of Loop*s. Because LoopInfo *does* release its memory from run to run, it is entirely possible to run SCEV over one function, then over another function, and then lookup a Loop* from the second function but find an entry inserted for the first function! Ouch. To make matters still worse, there are plenty of updates that *don't* trip a value handle. It seems incredibly unlikely that today GVN or another pass that invalidates SCEV can update values in *just* such a way that a subsequent run of SCEV will incorrectly find lookups in a cache, but it is theoretically possible and would be a nightmare to debug. With this refactoring, I've fixed all this by actually destroying and recreating the ScalarEvolution object from run to run. Technically, this could increase the amount of malloc traffic we see, but then again it is also technically correct. ;] I don't actually think we're suffering from tons of malloc traffic from SCEV because if we were, the fact that we never clear the memory would seem more likely to have come up as an actual problem before now. So, I've made the simple fix here. If in fact there are serious issues with too much allocation and deallocation, I can work on a clever fix that preserves the allocations (while clearing the data) between each run, but I'd prefer to do that kind of optimization with a test case / benchmark that shows why we need such cleverness (and that can test that we actually make it faster). It's possible that this will make some things faster by making the SCEV caches have higher locality (due to being significantly smaller) so until there is a clear benchmark, I think the simple change is best. Differential Revision: http://reviews.llvm.org/D12063 llvm-svn: 245193
2015-08-17 10:08:17 +08:00
AU.addPreserved<ScalarEvolutionWrapperPass>();
[PM/AA] Rebuild LLVM's alias analysis infrastructure in a way compatible with the new pass manager, and no longer relying on analysis groups. This builds essentially a ground-up new AA infrastructure stack for LLVM. The core ideas are the same that are used throughout the new pass manager: type erased polymorphism and direct composition. The design is as follows: - FunctionAAResults is a type-erasing alias analysis results aggregation interface to walk a single query across a range of results from different alias analyses. Currently this is function-specific as we always assume that aliasing queries are *within* a function. - AAResultBase is a CRTP utility providing stub implementations of various parts of the alias analysis result concept, notably in several cases in terms of other more general parts of the interface. This can be used to implement only a narrow part of the interface rather than the entire interface. This isn't really ideal, this logic should be hoisted into FunctionAAResults as currently it will cause a significant amount of redundant work, but it faithfully models the behavior of the prior infrastructure. - All the alias analysis passes are ported to be wrapper passes for the legacy PM and new-style analysis passes for the new PM with a shared result object. In some cases (most notably CFL), this is an extremely naive approach that we should revisit when we can specialize for the new pass manager. - BasicAA has been restructured to reflect that it is much more fundamentally a function analysis because it uses dominator trees and loop info that need to be constructed for each function. All of the references to getting alias analysis results have been updated to use the new aggregation interface. All the preservation and other pass management code has been updated accordingly. The way the FunctionAAResultsWrapperPass works is to detect the available alias analyses when run, and add them to the results object. This means that we should be able to continue to respect when various passes are added to the pipeline, for example adding CFL or adding TBAA passes should just cause their results to be available and to get folded into this. The exception to this rule is BasicAA which really needs to be a function pass due to using dominator trees and loop info. As a consequence, the FunctionAAResultsWrapperPass directly depends on BasicAA and always includes it in the aggregation. This has significant implications for preserving analyses. Generally, most passes shouldn't bother preserving FunctionAAResultsWrapperPass because rebuilding the results just updates the set of known AA passes. The exception to this rule are LoopPass instances which need to preserve all the function analyses that the loop pass manager will end up needing. This means preserving both BasicAAWrapperPass and the aggregating FunctionAAResultsWrapperPass. Now, when preserving an alias analysis, you do so by directly preserving that analysis. This is only necessary for non-immutable-pass-provided alias analyses though, and there are only three of interest: BasicAA, GlobalsAA (formerly GlobalsModRef), and SCEVAA. Usually BasicAA is preserved when needed because it (like DominatorTree and LoopInfo) is marked as a CFG-only pass. I've expanded GlobalsAA into the preserved set everywhere we previously were preserving all of AliasAnalysis, and I've added SCEVAA in the intersection of that with where we preserve SCEV itself. One significant challenge to all of this is that the CGSCC passes were actually using the alias analysis implementations by taking advantage of a pretty amazing set of loop holes in the old pass manager's analysis management code which allowed analysis groups to slide through in many cases. Moving away from analysis groups makes this problem much more obvious. To fix it, I've leveraged the flexibility the design of the new PM components provides to just directly construct the relevant alias analyses for the relevant functions in the IPO passes that need them. This is a bit hacky, but should go away with the new pass manager, and is already in many ways cleaner than the prior state. Another significant challenge is that various facilities of the old alias analysis infrastructure just don't fit any more. The most significant of these is the alias analysis 'counter' pass. That pass relied on the ability to snoop on AA queries at different points in the analysis group chain. Instead, I'm planning to build printing functionality directly into the aggregation layer. I've not included that in this patch merely to keep it smaller. Note that all of this needs a nearly complete rewrite of the AA documentation. I'm planning to do that, but I'd like to make sure the new design settles, and to flesh out a bit more of what it looks like in the new pass manager first. Differential Revision: http://reviews.llvm.org/D12080 llvm-svn: 247167
2015-09-10 01:55:00 +08:00
AU.addPreserved<SCEVAAWrapperPass>();
A few of the changes suggested in code review (by Nick Lewycky) llvm-svn: 149472
2012-02-01 13:51:45 +08:00
AU.setPreservesCFG();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
static inline VectorType *getVecTypeForPair(Type *ElemTy, Type *Elem2Ty) {
assert(ElemTy->getScalarType() == Elem2Ty->getScalarType() &&
"Cannot form vector from incompatible scalar types");
Type *STy = ElemTy->getScalarType();
unsigned numElem;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (VectorType *VTy = dyn_cast<VectorType>(ElemTy)) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
numElem = VTy->getNumElements();
} else {
numElem = 1;
}
if (VectorType *VTy = dyn_cast<VectorType>(Elem2Ty)) {
numElem += VTy->getNumElements();
} else {
numElem += 1;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
A few of the changes suggested in code review (by Nick Lewycky) llvm-svn: 149472
2012-02-01 13:51:45 +08:00
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
return VectorType::get(STy, numElem);
}
static inline void getInstructionTypes(Instruction *I,
Type *&T1, Type *&T2) {
BBVectorize: isa/cast cleanup in getInstructionTypes Profiling suggests that getInstructionTypes is performance-sensitive, this cleans up some double-casting in that function in favor of using dyn_cast. No functionality change intended. llvm-svn: 174857
2013-02-11 13:29:48 +08:00
if (StoreInst *SI = dyn_cast<StoreInst>(I)) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
// For stores, it is the value type, not the pointer type that matters
// because the value is what will come from a vector register.
Test commit access Remove some trailing whitespaces. llvm-svn: 298379
2017-03-21 22:49:19 +08:00
BBVectorize: isa/cast cleanup in getInstructionTypes Profiling suggests that getInstructionTypes is performance-sensitive, this cleans up some double-casting in that function in favor of using dyn_cast. No functionality change intended. llvm-svn: 174857
2013-02-11 13:29:48 +08:00
Value *IVal = SI->getValueOperand();
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
T1 = IVal->getType();
} else {
T1 = I->getType();
}
Test commit access Remove some trailing whitespaces. llvm-svn: 298379
2017-03-21 22:49:19 +08:00
BBVectorize: isa/cast cleanup in getInstructionTypes Profiling suggests that getInstructionTypes is performance-sensitive, this cleans up some double-casting in that function in favor of using dyn_cast. No functionality change intended. llvm-svn: 174857
2013-02-11 13:29:48 +08:00
if (CastInst *CI = dyn_cast<CastInst>(I))
T2 = CI->getSrcTy();
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
else
T2 = T1;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (SelectInst *SI = dyn_cast<SelectInst>(I)) {
T2 = SI->getCondition()->getType();
BBVectorize: Check the input types of shuffles for legality This fixes a bug where shuffles were being fused such that the resulting input types were not legal on the target. This would occur only when both inputs and dependencies were also foldable operations (such as other shuffles) and there were other connected pairs in the same block. llvm-svn: 167731
2012-11-12 22:50:59 +08:00
} else if (ShuffleVectorInst *SI = dyn_cast<ShuffleVectorInst>(I)) {
T2 = SI->getOperand(0)->getType();
BBVectorize: Check the types of compare instructions The pass would previously assert when trying to compute the cost of compare instructions with illegal vector types (like struct pointers). llvm-svn: 167743
2012-11-13 03:41:38 +08:00
} else if (CmpInst *CI = dyn_cast<CmpInst>(I)) {
T2 = CI->getOperand(0)->getType();
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
// Returns the weight associated with the provided value. A chain of
// candidate pairs has a length given by the sum of the weights of its
// members (one weight per pair; the weight of each member of the pair
// is assumed to be the same). This length is then compared to the
// chain-length threshold to determine if a given chain is significant
// enough to be vectorized. The length is also used in comparing
// candidate chains where longer chains are considered to be better.
// Note: when this function returns 0, the resulting instructions are
// not actually fused.
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
inline size_t getDepthFactor(Value *V) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// InsertElement and ExtractElement have a depth factor of zero. This is
// for two reasons: First, they cannot be usefully fused. Second, because
// the pass generates a lot of these, they can confuse the simple metric
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// used to compare the dags in the next iteration. Thus, giving them a
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// weight of zero allows the pass to essentially ignore them in
// subsequent iterations when looking for vectorization opportunities
// while still tracking dependency chains that flow through those
// instructions.
if (isa<InsertElementInst>(V) || isa<ExtractElementInst>(V))
return 0;
Boost the effective chain depth of loads and stores. By default, boost the chain depth contribution of loads and stores. This will allow a load/store pair to vectorize even when it would not otherwise be long enough to satisfy the chain depth requirement. llvm-svn: 149761
2012-02-04 12:14:04 +08:00
// Give a load or store half of the required depth so that load/store
// pairs will vectorize.
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
if (!Config.NoMemOpBoost && (isa<LoadInst>(V) || isa<StoreInst>(V)))
return Config.ReqChainDepth/2;
Boost the effective chain depth of loads and stores. By default, boost the chain depth contribution of loads and stores. This will allow a load/store pair to vectorize even when it would not otherwise be long enough to satisfy the chain depth requirement. llvm-svn: 149761
2012-02-04 12:14:04 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return 1;
}
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
// Returns the cost of the provided instruction using TTI.
Update BBVectorize to use the new VTTI instr. cost interfaces. The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. llvm-svn: 166865
2012-10-27 12:33:48 +08:00
// This does not handle loads and stores.
[Vectorizer] Add a new 'OperandValueKind' in TargetTransformInfo called 'OK_NonUniformConstValue' to identify operands which are constants but not constant splats. The cost model now allows returning 'OK_NonUniformConstValue' for non splat operands that are instances of ConstantVector or ConstantDataVector. With this change, targets are now able to compute different costs for instructions with non-uniform constant operands. For example, On X86 the cost of a vector shift may vary depending on whether the second operand is a uniform or non-uniform constant. This patch applies the following changes: - The cost model computation now takes into account non-uniform constants; - The cost of vector shift instructions has been improved in X86TargetTransformInfo analysis pass; - BBVectorize, SLPVectorizer and LoopVectorize now know how to distinguish between non-uniform and uniform constant operands. Added a new test to verify that the output of opt '-cost-model -analyze' is valid in the following configurations: SSE2, SSE4.1, AVX, AVX2. llvm-svn: 201272
2014-02-13 07:43:47 +08:00
unsigned getInstrCost(unsigned Opcode, Type *T1, Type *T2,
Test commit access Remove some trailing whitespaces. llvm-svn: 298379
2017-03-21 22:49:19 +08:00
TargetTransformInfo::OperandValueKind Op1VK =
[Vectorizer] Add a new 'OperandValueKind' in TargetTransformInfo called 'OK_NonUniformConstValue' to identify operands which are constants but not constant splats. The cost model now allows returning 'OK_NonUniformConstValue' for non splat operands that are instances of ConstantVector or ConstantDataVector. With this change, targets are now able to compute different costs for instructions with non-uniform constant operands. For example, On X86 the cost of a vector shift may vary depending on whether the second operand is a uniform or non-uniform constant. This patch applies the following changes: - The cost model computation now takes into account non-uniform constants; - The cost of vector shift instructions has been improved in X86TargetTransformInfo analysis pass; - BBVectorize, SLPVectorizer and LoopVectorize now know how to distinguish between non-uniform and uniform constant operands. Added a new test to verify that the output of opt '-cost-model -analyze' is valid in the following configurations: SSE2, SSE4.1, AVX, AVX2. llvm-svn: 201272
2014-02-13 07:43:47 +08:00
TargetTransformInfo::OK_AnyValue,
TargetTransformInfo::OperandValueKind Op2VK =
[SystemZ] TargetTransformInfo cost functions implemented. getArithmeticInstrCost(), getShuffleCost(), getCastInstrCost(), getCmpSelInstrCost(), getVectorInstrCost(), getMemoryOpCost(), getInterleavedMemoryOpCost() implemented. Interleaved access vectorization enabled. BasicTTIImpl::getCastInstrCost() improved to check for legal extending loads, in which case the cost of the z/sext instruction becomes 0. Review: Ulrich Weigand, Renato Golin. https://reviews.llvm.org/D29631 llvm-svn: 300052
2017-04-12 19:49:08 +08:00
TargetTransformInfo::OK_AnyValue,
const Instruction *I = nullptr) {
Update BBVectorize to use the new VTTI instr. cost interfaces. The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. llvm-svn: 166865
2012-10-27 12:33:48 +08:00
switch (Opcode) {
default: break;
case Instruction::GetElementPtr:
// We mark this instruction as zero-cost because scalar GEPs are usually
Fix spelling intruction -> instruction. llvm-svn: 191610
2013-09-28 19:46:15 +08:00
// lowered to the instruction addressing mode. At the moment we don't
Update BBVectorize to use the new VTTI instr. cost interfaces. The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. llvm-svn: 166865
2012-10-27 12:33:48 +08:00
// generate vector GEPs.
return 0;
case Instruction::Br:
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
return TTI->getCFInstrCost(Opcode);
Update BBVectorize to use the new VTTI instr. cost interfaces. The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. llvm-svn: 166865
2012-10-27 12:33:48 +08:00
case Instruction::PHI:
return 0;
case Instruction::Add:
case Instruction::FAdd:
case Instruction::Sub:
case Instruction::FSub:
case Instruction::Mul:
case Instruction::FMul:
case Instruction::UDiv:
case Instruction::SDiv:
case Instruction::FDiv:
case Instruction::URem:
case Instruction::SRem:
case Instruction::FRem:
case Instruction::Shl:
case Instruction::LShr:
case Instruction::AShr:
case Instruction::And:
case Instruction::Or:
case Instruction::Xor:
[Vectorizer] Add a new 'OperandValueKind' in TargetTransformInfo called 'OK_NonUniformConstValue' to identify operands which are constants but not constant splats. The cost model now allows returning 'OK_NonUniformConstValue' for non splat operands that are instances of ConstantVector or ConstantDataVector. With this change, targets are now able to compute different costs for instructions with non-uniform constant operands. For example, On X86 the cost of a vector shift may vary depending on whether the second operand is a uniform or non-uniform constant. This patch applies the following changes: - The cost model computation now takes into account non-uniform constants; - The cost of vector shift instructions has been improved in X86TargetTransformInfo analysis pass; - BBVectorize, SLPVectorizer and LoopVectorize now know how to distinguish between non-uniform and uniform constant operands. Added a new test to verify that the output of opt '-cost-model -analyze' is valid in the following configurations: SSE2, SSE4.1, AVX, AVX2. llvm-svn: 201272
2014-02-13 07:43:47 +08:00
return TTI->getArithmeticInstrCost(Opcode, T1, Op1VK, Op2VK);
Update BBVectorize to use the new VTTI instr. cost interfaces. The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. llvm-svn: 166865
2012-10-27 12:33:48 +08:00
case Instruction::Select:
case Instruction::ICmp:
case Instruction::FCmp:
[SystemZ] TargetTransformInfo cost functions implemented. getArithmeticInstrCost(), getShuffleCost(), getCastInstrCost(), getCmpSelInstrCost(), getVectorInstrCost(), getMemoryOpCost(), getInterleavedMemoryOpCost() implemented. Interleaved access vectorization enabled. BasicTTIImpl::getCastInstrCost() improved to check for legal extending loads, in which case the cost of the z/sext instruction becomes 0. Review: Ulrich Weigand, Renato Golin. https://reviews.llvm.org/D29631 llvm-svn: 300052
2017-04-12 19:49:08 +08:00
return TTI->getCmpSelInstrCost(Opcode, T1, T2, I);
Update BBVectorize to use the new VTTI instr. cost interfaces. The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. llvm-svn: 166865
2012-10-27 12:33:48 +08:00
case Instruction::ZExt:
case Instruction::SExt:
case Instruction::FPToUI:
case Instruction::FPToSI:
case Instruction::FPExt:
case Instruction::PtrToInt:
case Instruction::IntToPtr:
case Instruction::SIToFP:
case Instruction::UIToFP:
case Instruction::Trunc:
case Instruction::FPTrunc:
case Instruction::BitCast:
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
case Instruction::ShuffleVector:
[SystemZ] TargetTransformInfo cost functions implemented. getArithmeticInstrCost(), getShuffleCost(), getCastInstrCost(), getCmpSelInstrCost(), getVectorInstrCost(), getMemoryOpCost(), getInterleavedMemoryOpCost() implemented. Interleaved access vectorization enabled. BasicTTIImpl::getCastInstrCost() improved to check for legal extending loads, in which case the cost of the z/sext instruction becomes 0. Review: Ulrich Weigand, Renato Golin. https://reviews.llvm.org/D29631 llvm-svn: 300052
2017-04-12 19:49:08 +08:00
return TTI->getCastInstrCost(Opcode, T1, T2, I);
Update BBVectorize to use the new VTTI instr. cost interfaces. The monolithic interface for instruction costs has been split into several functions. This is the corresponding change. No functionality change is intended. llvm-svn: 166865
2012-10-27 12:33:48 +08:00
}
return 1;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// This determines the relative offset of two loads or stores, returning
// true if the offset could be determined to be some constant value.
// For example, if OffsetInElmts == 1, then J accesses the memory directly
// after I; if OffsetInElmts == -1 then I accesses the memory
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
// directly after J.
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool getPairPtrInfo(Instruction *I, Instruction *J,
Value *&IPtr, Value *&JPtr, unsigned &IAlignment, unsigned &JAlignment,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
unsigned &IAddressSpace, unsigned &JAddressSpace,
BBVectorize: Don't make calls to SE when the result is unused. SE was being called during the instruction-fusion process (when the result is unreliable, and thus ignored). No functionality change is intended. llvm-svn: 167037
2012-10-31 02:55:49 +08:00
int64_t &OffsetInElmts, bool ComputeOffset = true) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
OffsetInElmts = 0;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (LoadInst *LI = dyn_cast<LoadInst>(I)) {
LoadInst *LJ = cast<LoadInst>(J);
IPtr = LI->getPointerOperand();
JPtr = LJ->getPointerOperand();
IAlignment = LI->getAlignment();
JAlignment = LJ->getAlignment();
IAddressSpace = LI->getPointerAddressSpace();
JAddressSpace = LJ->getPointerAddressSpace();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
} else {
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
StoreInst *SI = cast<StoreInst>(I), *SJ = cast<StoreInst>(J);
IPtr = SI->getPointerOperand();
JPtr = SJ->getPointerOperand();
IAlignment = SI->getAlignment();
JAlignment = SJ->getAlignment();
IAddressSpace = SI->getPointerAddressSpace();
JAddressSpace = SJ->getPointerAddressSpace();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
BBVectorize: Don't make calls to SE when the result is unused. SE was being called during the instruction-fusion process (when the result is unreliable, and thus ignored). No functionality change is intended. llvm-svn: 167037
2012-10-31 02:55:49 +08:00
if (!ComputeOffset)
return true;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
const SCEV *IPtrSCEV = SE->getSCEV(IPtr);
const SCEV *JPtrSCEV = SE->getSCEV(JPtr);
// If this is a trivial offset, then we'll get something like
// 1*sizeof(type). With target data, which we need anyway, this will get
// constant folded into a number.
const SCEV *OffsetSCEV = SE->getMinusSCEV(JPtrSCEV, IPtrSCEV);
if (const SCEVConstant *ConstOffSCEV =
dyn_cast<SCEVConstant>(OffsetSCEV)) {
ConstantInt *IntOff = ConstOffSCEV->getValue();
int64_t Offset = IntOff->getSExtValue();
DataLayout is mandatory, update the API to reflect it with references. Summary: Now that the DataLayout is a mandatory part of the module, let's start cleaning the codebase. This patch is a first attempt at doing that. This patch is not exactly NFC as for instance some places were passing a nullptr instead of the DataLayout, possibly just because there was a default value on the DataLayout argument to many functions in the API. Even though it is not purely NFC, there is no change in the validation. I turned as many pointer to DataLayout to references, this helped figuring out all the places where a nullptr could come up. I had initially a local version of this patch broken into over 30 independant, commits but some later commit were cleaning the API and touching part of the code modified in the previous commits, so it seemed cleaner without the intermediate state. Test Plan: Reviewers: echristo Subscribers: llvm-commits From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 231740
2015-03-10 10:37:25 +08:00
const DataLayout &DL = I->getModule()->getDataLayout();
Use more type helper functions llvm-svn: 193109
2013-10-22 03:43:56 +08:00
Type *VTy = IPtr->getType()->getPointerElementType();
DataLayout is mandatory, update the API to reflect it with references. Summary: Now that the DataLayout is a mandatory part of the module, let's start cleaning the codebase. This patch is a first attempt at doing that. This patch is not exactly NFC as for instance some places were passing a nullptr instead of the DataLayout, possibly just because there was a default value on the DataLayout argument to many functions in the API. Even though it is not purely NFC, there is no change in the validation. I turned as many pointer to DataLayout to references, this helped figuring out all the places where a nullptr could come up. I had initially a local version of this patch broken into over 30 independant, commits but some later commit were cleaning the API and touching part of the code modified in the previous commits, so it seemed cleaner without the intermediate state. Test Plan: Reviewers: echristo Subscribers: llvm-commits From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 231740
2015-03-10 10:37:25 +08:00
int64_t VTyTSS = (int64_t)DL.getTypeStoreSize(VTy);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Use more type helper functions llvm-svn: 193109
2013-10-22 03:43:56 +08:00
Type *VTy2 = JPtr->getType()->getPointerElementType();
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
if (VTy != VTy2 && Offset < 0) {
DataLayout is mandatory, update the API to reflect it with references. Summary: Now that the DataLayout is a mandatory part of the module, let's start cleaning the codebase. This patch is a first attempt at doing that. This patch is not exactly NFC as for instance some places were passing a nullptr instead of the DataLayout, possibly just because there was a default value on the DataLayout argument to many functions in the API. Even though it is not purely NFC, there is no change in the validation. I turned as many pointer to DataLayout to references, this helped figuring out all the places where a nullptr could come up. I had initially a local version of this patch broken into over 30 independant, commits but some later commit were cleaning the API and touching part of the code modified in the previous commits, so it seemed cleaner without the intermediate state. Test Plan: Reviewers: echristo Subscribers: llvm-commits From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 231740
2015-03-10 10:37:25 +08:00
int64_t VTy2TSS = (int64_t)DL.getTypeStoreSize(VTy2);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
OffsetInElmts = Offset/VTy2TSS;
Remove the remaining uses of abs64 and nuke it. std::abs works just fine and we're already using it in many places. NFC intended. llvm-svn: 231696
2015-03-10 04:20:16 +08:00
return (std::abs(Offset) % VTy2TSS) == 0;
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
OffsetInElmts = Offset/VTyTSS;
Remove the remaining uses of abs64 and nuke it. std::abs works just fine and we're already using it in many places. NFC intended. llvm-svn: 231696
2015-03-10 04:20:16 +08:00
return (std::abs(Offset) % VTyTSS) == 0;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
return false;
}
// Returns true if the provided CallInst represents an intrinsic that can
// be vectorized.
bool isVectorizableIntrinsic(CallInst* I) {
Function *F = I->getCalledFunction();
if (!F) return false;
Change Function::getIntrinsicID() to return an Intrinsic::ID. NFC. Now that Intrinsic::ID is a typed enum, we can forward declare it and so return it from this method. This updates all users which were either using an unsigned to store it, or had a now unnecessary cast. llvm-svn: 237810
2015-05-21 01:16:39 +08:00
Intrinsic::ID IID = F->getIntrinsicID();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (!IID) return false;
switch(IID) {
default:
return false;
case Intrinsic::sqrt:
case Intrinsic::powi:
case Intrinsic::sin:
case Intrinsic::cos:
case Intrinsic::log:
case Intrinsic::log2:
case Intrinsic::log10:
case Intrinsic::exp:
case Intrinsic::exp2:
case Intrinsic::pow:
Allow vectorization of few missed llvm intrinsic calls in BBVectorizor by handling them in isVectorizableIntrinsic function. llvm-svn: 207085
2014-04-24 15:29:55 +08:00
case Intrinsic::round:
case Intrinsic::copysign:
case Intrinsic::ceil:
case Intrinsic::nearbyint:
case Intrinsic::rint:
case Intrinsic::trunc:
case Intrinsic::floor:
case Intrinsic::fabs:
Add minnum / maxnum intrinsics These are named following the IEEE-754 names for these functions, rather than the libm fmin / fmax to avoid possible ambiguities. Some languages may implement something resembling fmin / fmax which return NaN if either operand is to propagate errors. These implement the IEEE-754 semantics of returning the other operand if either is a NaN representing missing data. llvm-svn: 220341
2014-10-22 07:00:20 +08:00
case Intrinsic::minnum:
case Intrinsic::maxnum:
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
return Config.VectorizeMath;
Allow vectorization of bit intrinsics in BB Vectorizer. This patch adds support for vectorization of bit intrinsics such as bswap,ctpop,ctlz,cttz. llvm-svn: 207174
2014-04-25 11:33:48 +08:00
case Intrinsic::bswap:
case Intrinsic::ctpop:
case Intrinsic::ctlz:
case Intrinsic::cttz:
return Config.VectorizeBitManipulations;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
case Intrinsic::fma:
BBVectorize: Enable vectorization of the fmuladd intrinsic llvm-svn: 171075
2012-12-26 06:36:08 +08:00
case Intrinsic::fmuladd:
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
return Config.VectorizeFMA;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
BBVectorize: Only some insert element operand pairs are free. This fixes another infinite recursion case when using target costs. We can only replace insert element input chains that are pure (end with inserting into an undef). llvm-svn: 167784
2012-11-13 07:55:36 +08:00
bool isPureIEChain(InsertElementInst *IE) {
InsertElementInst *IENext = IE;
do {
if (!isa<UndefValue>(IENext->getOperand(0)) &&
!isa<InsertElementInst>(IENext->getOperand(0))) {
return false;
}
} while ((IENext =
dyn_cast<InsertElementInst>(IENext->getOperand(0))));
return true;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
};
// This function implements one vectorization iteration on the provided
// basic block. It returns true if the block is changed.
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
bool BBVectorize::vectorizePairs(BasicBlock &BB, bool NonPow2Len) {
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
bool ShouldContinue;
BasicBlock::iterator Start = BB.getFirstInsertionPt();
std::vector<Value *> AllPairableInsts;
DenseMap<Value *, Value *> AllChosenPairs;
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
DenseSet<ValuePair> AllFixedOrderPairs;
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
DenseMap<VPPair, unsigned> AllPairConnectionTypes;
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<ValuePair, std::vector<ValuePair> > AllConnectedPairs,
AllConnectedPairDeps;
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
do {
std::vector<Value *> PairableInsts;
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
DenseMap<Value *, std::vector<Value *> > CandidatePairs;
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
DenseSet<ValuePair> FixedOrderPairs;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
DenseMap<ValuePair, int> CandidatePairCostSavings;
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
ShouldContinue = getCandidatePairs(BB, Start, CandidatePairs,
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
FixedOrderPairs,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
CandidatePairCostSavings,
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
PairableInsts, NonPow2Len);
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
if (PairableInsts.empty()) continue;
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
BBVectorize: Eliminate one more restricted linear search This eliminates one more linear search over a range of std::multimap entries. This gives a 22% speedup on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 174893
2013-02-12 01:19:34 +08:00
// Build the candidate pair set for faster lookups.
DenseSet<ValuePair> CandidatePairsSet;
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
for (DenseMap<Value *, std::vector<Value *> >::iterator I =
CandidatePairs.begin(), E = CandidatePairs.end(); I != E; ++I)
for (std::vector<Value *>::iterator J = I->second.begin(),
JE = I->second.end(); J != JE; ++J)
CandidatePairsSet.insert(ValuePair(I->first, *J));
BBVectorize: Eliminate one more restricted linear search This eliminates one more linear search over a range of std::multimap entries. This gives a 22% speedup on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 174893
2013-02-12 01:19:34 +08:00
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
// Now we have a map of all of the pairable instructions and we need to
// select the best possible pairing. A good pairing is one such that the
// users of the pair are also paired. This defines a (directed) forest
Revert 'Fix a typo 'iff' => 'if''. iff is an abreviation of if and only if. See: http://en.wikipedia.org/wiki/If_and_only_if Commit 164767 llvm-svn: 164768
2012-09-27 18:14:43 +08:00
// over the pairs such that two pairs are connected iff the second pair
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
// uses the first.
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
// Note that it only matters that both members of the second pair use some
// element of the first pair (to allow for splatting).
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<ValuePair, std::vector<ValuePair> > ConnectedPairs,
ConnectedPairDeps;
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
DenseMap<VPPair, unsigned> PairConnectionTypes;
BBVectorize: Eliminate one more restricted linear search This eliminates one more linear search over a range of std::multimap entries. This gives a 22% speedup on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 174893
2013-02-12 01:19:34 +08:00
computeConnectedPairs(CandidatePairs, CandidatePairsSet,
PairableInsts, ConnectedPairs, PairConnectionTypes);
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
if (ConnectedPairs.empty()) continue;
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
for (DenseMap<ValuePair, std::vector<ValuePair> >::iterator
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
I = ConnectedPairs.begin(), IE = ConnectedPairs.end();
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
I != IE; ++I)
for (std::vector<ValuePair>::iterator J = I->second.begin(),
JE = I->second.end(); J != JE; ++J)
ConnectedPairDeps[*J].push_back(I->first);
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
// Build the pairable-instruction dependency map
DenseSet<ValuePair> PairableInstUsers;
buildDepMap(BB, CandidatePairs, PairableInsts, PairableInstUsers);
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
// There is now a graph of the connected pairs. For each variable, pick
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// the pairing with the largest dag meeting the depth requirement on at
// least one branch. Then select all pairings that are part of that dag
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
// and remove them from the list of available pairings and pairable
// variables.
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
DenseMap<Value *, Value *> ChosenPairs;
BBVectorize: Eliminate one more restricted linear search This eliminates one more linear search over a range of std::multimap entries. This gives a 22% speedup on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 174893
2013-02-12 01:19:34 +08:00
choosePairs(CandidatePairs, CandidatePairsSet,
CandidatePairCostSavings,
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
PairableInsts, FixedOrderPairs, PairConnectionTypes,
ConnectedPairs, ConnectedPairDeps,
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
PairableInstUsers, ChosenPairs);
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
if (ChosenPairs.empty()) continue;
AllPairableInsts.insert(AllPairableInsts.end(), PairableInsts.begin(),
PairableInsts.end());
AllChosenPairs.insert(ChosenPairs.begin(), ChosenPairs.end());
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
// Only for the chosen pairs, propagate information on fixed-order pairs,
// pair connections, and their types to the data structures used by the
// pair fusion procedures.
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
for (DenseMap<Value *, Value *>::iterator I = ChosenPairs.begin(),
IE = ChosenPairs.end(); I != IE; ++I) {
if (FixedOrderPairs.count(*I))
AllFixedOrderPairs.insert(*I);
else if (FixedOrderPairs.count(ValuePair(I->second, I->first)))
AllFixedOrderPairs.insert(ValuePair(I->second, I->first));
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
for (DenseMap<Value *, Value *>::iterator J = ChosenPairs.begin();
J != IE; ++J) {
DenseMap<VPPair, unsigned>::iterator K =
PairConnectionTypes.find(VPPair(*I, *J));
if (K != PairConnectionTypes.end()) {
AllPairConnectionTypes.insert(*K);
} else {
K = PairConnectionTypes.find(VPPair(*J, *I));
if (K != PairConnectionTypes.end())
AllPairConnectionTypes.insert(*K);
}
}
}
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
for (DenseMap<ValuePair, std::vector<ValuePair> >::iterator
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
I = ConnectedPairs.begin(), IE = ConnectedPairs.end();
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
I != IE; ++I)
for (std::vector<ValuePair>::iterator J = I->second.begin(),
JE = I->second.end(); J != JE; ++J)
if (AllPairConnectionTypes.count(VPPair(I->first, *J))) {
AllConnectedPairs[I->first].push_back(*J);
AllConnectedPairDeps[*J].push_back(I->first);
}
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
} while (ShouldContinue);
if (AllChosenPairs.empty()) return false;
NumFusedOps += AllChosenPairs.size();
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// A set of pairs has now been selected. It is now necessary to replace the
// paired instructions with vector instructions. For this procedure each
fix typos llvm-svn: 152035
2012-03-06 01:39:47 +08:00
// operand must be replaced with a vector operand. This vector is formed
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// by using build_vector on the old operands. The replaced values are then
// replaced with a vector_extract on the result. Subsequent optimization
// passes should coalesce the build/extract combinations.
remove spaces on empty lines llvm-svn: 152034
2012-03-06 01:39:45 +08:00
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
fuseChosenPairs(BB, AllPairableInsts, AllChosenPairs, AllFixedOrderPairs,
AllPairConnectionTypes,
AllConnectedPairs, AllConnectedPairDeps);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
// It is important to cleanup here so that future iterations of this
// function have less work to do.
[PM/AA] Explicitly depend on TLI rather than getting it out of the AliasAnalysis. Same as the other commits, the TLI access from an alias analysis is going away and isn't very clean -- it is better to explicitly mark the dependencies. llvm-svn: 244785
2015-08-13 02:06:08 +08:00
(void)SimplifyInstructionsInBlock(&BB, TLI);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return true;
}
// This function returns true if the provided instruction is capable of being
// fused into a vector instruction. This determination is based only on the
// type and other attributes of the instruction.
bool BBVectorize::isInstVectorizable(Instruction *I,
bool &IsSimpleLoadStore) {
IsSimpleLoadStore = false;
if (CallInst *C = dyn_cast<CallInst>(I)) {
if (!isVectorizableIntrinsic(C))
return false;
} else if (LoadInst *L = dyn_cast<LoadInst>(I)) {
// Vectorize simple loads if possbile:
IsSimpleLoadStore = L->isSimple();
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
if (!IsSimpleLoadStore || !Config.VectorizeMemOps)
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
} else if (StoreInst *S = dyn_cast<StoreInst>(I)) {
// Vectorize simple stores if possbile:
IsSimpleLoadStore = S->isSimple();
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
if (!IsSimpleLoadStore || !Config.VectorizeMemOps)
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
} else if (CastInst *C = dyn_cast<CastInst>(I)) {
// We can vectorize casts, but not casts of pointer types, etc.
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
if (!Config.VectorizeCasts)
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
Type *SrcTy = C->getSrcTy();
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
if (!SrcTy->isSingleValueType())
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
Type *DestTy = C->getDestTy();
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
if (!DestTy->isSingleValueType())
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
[BBVectorize] Don't vectorize selects with a scalar condition and vector operands. This fixes PR27879. Differential Revision: http://reviews.llvm.org/D20659 llvm-svn: 270888
2016-05-27 02:43:57 +08:00
} else if (SelectInst *SI = dyn_cast<SelectInst>(I)) {
Add support to BBVectorize for vectorizing selects. llvm-svn: 154700
2012-04-14 04:45:45 +08:00
if (!Config.VectorizeSelect)
return false;
[BBVectorize] Don't vectorize selects with a scalar condition and vector operands. This fixes PR27879. Differential Revision: http://reviews.llvm.org/D20659 llvm-svn: 270888
2016-05-27 02:43:57 +08:00
// We can vectorize a select if either all operands are scalars,
// or all operands are vectors. Trying to "widen" a select between
// vectors that has a scalar condition results in a malformed select.
// FIXME: We could probably be smarter about this by rewriting the select
// with different types instead.
Test commit access Remove some trailing whitespaces. llvm-svn: 298379
2017-03-21 22:49:19 +08:00
return (SI->getCondition()->getType()->isVectorTy() ==
[BBVectorize] Don't vectorize selects with a scalar condition and vector operands. This fixes PR27879. Differential Revision: http://reviews.llvm.org/D20659 llvm-svn: 270888
2016-05-27 02:43:57 +08:00
SI->getTrueValue()->getType()->isVectorTy());
Allow BBVectorize to fuse compare instructions. llvm-svn: 159088
2012-06-24 05:52:50 +08:00
} else if (isa<CmpInst>(I)) {
if (!Config.VectorizeCmp)
return false;
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
} else if (GetElementPtrInst *G = dyn_cast<GetElementPtrInst>(I)) {
if (!Config.VectorizeGEP)
return false;
// Currently, vector GEPs exist only with one index.
if (G->getNumIndices() != 1)
return false;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
} else if (!(I->isBinaryOp() || isa<ShuffleVectorInst>(I) ||
isa<ExtractElementInst>(I) || isa<InsertElementInst>(I))) {
return false;
}
Type *T1, *T2;
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
getInstructionTypes(I, T1, T2);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Not every type can be vectorized...
if (!(VectorType::isValidElementType(T1) || T1->isVectorTy()) ||
!(VectorType::isValidElementType(T2) || T2->isVectorTy()))
return false;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (T1->getScalarSizeInBits() == 1) {
Allow controlling vectorization of boolean values separately from other integer types. These are used as the result of comparisons, and often handled differently from larger integer types. llvm-svn: 159111
2012-06-24 21:28:01 +08:00
if (!Config.VectorizeBools)
return false;
} else {
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (!Config.VectorizeInts && T1->isIntOrIntVectorTy())
Allow controlling vectorization of boolean values separately from other integer types. These are used as the result of comparisons, and often handled differently from larger integer types. llvm-svn: 159111
2012-06-24 21:28:01 +08:00
return false;
}
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (T2->getScalarSizeInBits() == 1) {
if (!Config.VectorizeBools)
return false;
} else {
if (!Config.VectorizeInts && T2->isIntOrIntVectorTy())
return false;
}
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
if (!Config.VectorizeFloats
&& (T1->isFPOrFPVectorTy() || T2->isFPOrFPVectorTy()))
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
Don't vectorize target-specific types (ppc_fp128, x86_fp80, etc.). Target specific types should not be vectorized. As a practical matter, these types are already register matched (at least in the x86 case), and codegen does not always work correctly (at least in the ppc case, and this is not worth fixing because ppc_fp128 is currently broken and will probably go away soon). llvm-svn: 155729
2012-04-28 03:34:00 +08:00
// Don't vectorize target-specific types.
if (T1->isX86_FP80Ty() || T1->isPPC_FP128Ty() || T1->isX86_MMXTy())
return false;
if (T2->isX86_FP80Ty() || T2->isPPC_FP128Ty() || T2->isX86_MMXTy())
return false;
DataLayout is mandatory, update the API to reflect it with references. Summary: Now that the DataLayout is a mandatory part of the module, let's start cleaning the codebase. This patch is a first attempt at doing that. This patch is not exactly NFC as for instance some places were passing a nullptr instead of the DataLayout, possibly just because there was a default value on the DataLayout argument to many functions in the API. Even though it is not purely NFC, there is no change in the validation. I turned as many pointer to DataLayout to references, this helped figuring out all the places where a nullptr could come up. I had initially a local version of this patch broken into over 30 independant, commits but some later commit were cleaning the API and touching part of the code modified in the previous commits, so it seemed cleaner without the intermediate state. Test Plan: Reviewers: echristo Subscribers: llvm-commits From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 231740
2015-03-10 10:37:25 +08:00
if (!Config.VectorizePointers && (T1->getScalarType()->isPointerTy() ||
T2->getScalarType()->isPointerTy()))
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
return false;
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
if (!TTI && (T1->getPrimitiveSizeInBits() >= Config.VectorBits ||
T2->getPrimitiveSizeInBits() >= Config.VectorBits))
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
return true;
}
// This function returns true if the two provided instructions are compatible
// (meaning that they can be fused into a vector instruction). This assumes
// that I has already been determined to be vectorizable and that J is not
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// in the use dag of I.
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool BBVectorize::areInstsCompatible(Instruction *I, Instruction *J,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
bool IsSimpleLoadStore, bool NonPow2Len,
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
int &CostSavings, int &FixedOrder) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DEBUG(if (DebugInstructionExamination) dbgs() << "BBV: looking at " << *I <<
" <-> " << *J << "\n");
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
CostSavings = 0;
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
FixedOrder = 0;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Loads and stores can be merged if they have different alignments,
// but are otherwise the same.
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
if (!J->isSameOperationAs(I, Instruction::CompareIgnoringAlignment |
(NonPow2Len ? Instruction::CompareUsingScalarTypes : 0)))
return false;
Type *IT1, *IT2, *JT1, *JT2;
getInstructionTypes(I, IT1, IT2);
getInstructionTypes(J, JT1, JT2);
unsigned MaxTypeBits = std::max(
IT1->getPrimitiveSizeInBits() + JT1->getPrimitiveSizeInBits(),
IT2->getPrimitiveSizeInBits() + JT2->getPrimitiveSizeInBits());
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
if (!TTI && MaxTypeBits > Config.VectorBits)
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
Refactor operation equivalence checking in BBVectorize by extending Instruction::isSameOperationAs. Maintaining this kind of checking in different places is dangerous, extending Instruction::isSameOperationAs consolidates this logic into one place. Here I've added an optional flags parameter and two flags that are important for vectorization: CompareIgnoringAlignment and CompareUsingScalarTypes. llvm-svn: 159329
2012-06-28 13:42:26 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// FIXME: handle addsub-type operations!
if (IsSimpleLoadStore) {
Value *IPtr, *JPtr;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
unsigned IAlignment, JAlignment, IAddressSpace, JAddressSpace;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
int64_t OffsetInElmts = 0;
if (getPairPtrInfo(I, J, IPtr, JPtr, IAlignment, JAlignment,
Remove the remaining uses of abs64 and nuke it. std::abs works just fine and we're already using it in many places. NFC intended. llvm-svn: 231696
2015-03-10 04:20:16 +08:00
IAddressSpace, JAddressSpace, OffsetInElmts) &&
std::abs(OffsetInElmts) == 1) {
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
FixedOrder = (int) OffsetInElmts;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
unsigned BottomAlignment = IAlignment;
if (OffsetInElmts < 0) BottomAlignment = JAlignment;
Type *aTypeI = isa<StoreInst>(I) ?
cast<StoreInst>(I)->getValueOperand()->getType() : I->getType();
Type *aTypeJ = isa<StoreInst>(J) ?
cast<StoreInst>(J)->getValueOperand()->getType() : J->getType();
Type *VType = getVecTypeForPair(aTypeI, aTypeJ);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (Config.AlignedOnly) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// An aligned load or store is possible only if the instruction
// with the lower offset has an alignment suitable for the
// vector type.
DataLayout is mandatory, update the API to reflect it with references. Summary: Now that the DataLayout is a mandatory part of the module, let's start cleaning the codebase. This patch is a first attempt at doing that. This patch is not exactly NFC as for instance some places were passing a nullptr instead of the DataLayout, possibly just because there was a default value on the DataLayout argument to many functions in the API. Even though it is not purely NFC, there is no change in the validation. I turned as many pointer to DataLayout to references, this helped figuring out all the places where a nullptr could come up. I had initially a local version of this patch broken into over 30 independant, commits but some later commit were cleaning the API and touching part of the code modified in the previous commits, so it seemed cleaner without the intermediate state. Test Plan: Reviewers: echristo Subscribers: llvm-commits From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 231740
2015-03-10 10:37:25 +08:00
const DataLayout &DL = I->getModule()->getDataLayout();
unsigned VecAlignment = DL.getPrefTypeAlignment(VType);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (BottomAlignment < VecAlignment)
return false;
}
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
if (TTI) {
unsigned ICost = TTI->getMemoryOpCost(I->getOpcode(), aTypeI,
IAlignment, IAddressSpace);
unsigned JCost = TTI->getMemoryOpCost(J->getOpcode(), aTypeJ,
JAlignment, JAddressSpace);
unsigned VCost = TTI->getMemoryOpCost(I->getOpcode(), VType,
BottomAlignment,
IAddressSpace);
BBVectorize: Use TTI->getAddressComputationCost This is a follow-up to the cost-model change in r174713 which splits the cost of a memory operation between the address computation and the actual memory access. In r174713, this cost is always added to the memory operation cost, and so BBVectorize will do the same. Currently, this new cost function is used only by ARM, and I don't have any ARM test cases for BBVectorize. Assistance in generating some good ARM test cases for BBVectorize would be greatly appreciated! llvm-svn: 174743
2013-02-09 05:13:39 +08:00
ICost += TTI->getAddressComputationCost(aTypeI);
JCost += TTI->getAddressComputationCost(aTypeJ);
VCost += TTI->getAddressComputationCost(VType);
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (VCost > ICost + JCost)
return false;
BBVectorize, when using VTTI, should not form types that will be split. This is needed so that perl's SHA can be compiled (otherwise BBVectorize takes far too long to find its fixed point). I'll try to come up with a reduced test case. llvm-svn: 166738
2012-10-26 07:47:16 +08:00
Use VTTI->getNumberOfParts in BBVectorize. This change reflects VTTI refactoring; no functionality change intended. llvm-svn: 166752
2012-10-26 12:28:06 +08:00
// We don't want to fuse to a type that will be split, even
BBVectorize, when using VTTI, should not form types that will be split. This is needed so that perl's SHA can be compiled (otherwise BBVectorize takes far too long to find its fixed point). I'll try to come up with a reduced test case. llvm-svn: 166738
2012-10-26 07:47:16 +08:00
// if the two input types will also be split and there is no other
Use VTTI->getNumberOfParts in BBVectorize. This change reflects VTTI refactoring; no functionality change intended. llvm-svn: 166752
2012-10-26 12:28:06 +08:00
// associated cost.
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
unsigned VParts = TTI->getNumberOfParts(VType);
Use VTTI->getNumberOfParts in BBVectorize. This change reflects VTTI refactoring; no functionality change intended. llvm-svn: 166752
2012-10-26 12:28:06 +08:00
if (VParts > 1)
return false;
else if (!VParts && VCost == ICost + JCost)
BBVectorize, when using VTTI, should not form types that will be split. This is needed so that perl's SHA can be compiled (otherwise BBVectorize takes far too long to find its fixed point). I'll try to come up with a reduced test case. llvm-svn: 166738
2012-10-26 07:47:16 +08:00
return false;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
CostSavings = ICost + JCost - VCost;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
} else {
return false;
}
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
} else if (TTI) {
[Vectorizer] Add a new 'OperandValueKind' in TargetTransformInfo called 'OK_NonUniformConstValue' to identify operands which are constants but not constant splats. The cost model now allows returning 'OK_NonUniformConstValue' for non splat operands that are instances of ConstantVector or ConstantDataVector. With this change, targets are now able to compute different costs for instructions with non-uniform constant operands. For example, On X86 the cost of a vector shift may vary depending on whether the second operand is a uniform or non-uniform constant. This patch applies the following changes: - The cost model computation now takes into account non-uniform constants; - The cost of vector shift instructions has been improved in X86TargetTransformInfo analysis pass; - BBVectorize, SLPVectorizer and LoopVectorize now know how to distinguish between non-uniform and uniform constant operands. Added a new test to verify that the output of opt '-cost-model -analyze' is valid in the following configurations: SSE2, SSE4.1, AVX, AVX2. llvm-svn: 201272
2014-02-13 07:43:47 +08:00
TargetTransformInfo::OperandValueKind Op1VK =
TargetTransformInfo::OK_AnyValue;
TargetTransformInfo::OperandValueKind Op2VK =
TargetTransformInfo::OK_AnyValue;
[SystemZ] TargetTransformInfo cost functions implemented. getArithmeticInstrCost(), getShuffleCost(), getCastInstrCost(), getCmpSelInstrCost(), getVectorInstrCost(), getMemoryOpCost(), getInterleavedMemoryOpCost() implemented. Interleaved access vectorization enabled. BasicTTIImpl::getCastInstrCost() improved to check for legal extending loads, in which case the cost of the z/sext instruction becomes 0. Review: Ulrich Weigand, Renato Golin. https://reviews.llvm.org/D29631 llvm-svn: 300052
2017-04-12 19:49:08 +08:00
unsigned ICost = getInstrCost(I->getOpcode(), IT1, IT2, Op1VK, Op2VK, I);
unsigned JCost = getInstrCost(J->getOpcode(), JT1, JT2, Op1VK, Op2VK, J);
Type *VT1 = getVecTypeForPair(IT1, JT1),
*VT2 = getVecTypeForPair(IT2, JT2);
[Vectorizer] Add a new 'OperandValueKind' in TargetTransformInfo called 'OK_NonUniformConstValue' to identify operands which are constants but not constant splats. The cost model now allows returning 'OK_NonUniformConstValue' for non splat operands that are instances of ConstantVector or ConstantDataVector. With this change, targets are now able to compute different costs for instructions with non-uniform constant operands. For example, On X86 the cost of a vector shift may vary depending on whether the second operand is a uniform or non-uniform constant. This patch applies the following changes: - The cost model computation now takes into account non-uniform constants; - The cost of vector shift instructions has been improved in X86TargetTransformInfo analysis pass; - BBVectorize, SLPVectorizer and LoopVectorize now know how to distinguish between non-uniform and uniform constant operands. Added a new test to verify that the output of opt '-cost-model -analyze' is valid in the following configurations: SSE2, SSE4.1, AVX, AVX2. llvm-svn: 201272
2014-02-13 07:43:47 +08:00
// On some targets (example X86) the cost of a vector shift may vary
// depending on whether the second operand is a Uniform or
// NonUniform Constant.
switch (I->getOpcode()) {
default : break;
case Instruction::Shl:
case Instruction::LShr:
case Instruction::AShr:
// If both I and J are scalar shifts by constant, then the
// merged vector shift count would be either a constant splat value
// or a non-uniform vector of constants.
if (ConstantInt *CII = dyn_cast<ConstantInt>(I->getOperand(1))) {
if (ConstantInt *CIJ = dyn_cast<ConstantInt>(J->getOperand(1)))
Op2VK = CII == CIJ ? TargetTransformInfo::OK_UniformConstantValue :
TargetTransformInfo::OK_NonUniformConstantValue;
} else {
// Check for a splat of a constant or for a non uniform vector
// of constants.
Value *IOp = I->getOperand(1);
Value *JOp = J->getOperand(1);
Reduce code duplication resulting from the ConstantVector/ConstantDataVector split. No intended functionality change. llvm-svn: 201344
2014-02-14 00:48:38 +08:00
if ((isa<ConstantVector>(IOp) || isa<ConstantDataVector>(IOp)) &&
(isa<ConstantVector>(JOp) || isa<ConstantDataVector>(JOp))) {
Op2VK = TargetTransformInfo::OK_NonUniformConstantValue;
Constant *SplatValue = cast<Constant>(IOp)->getSplatValue();
[C++] Use 'nullptr'. Transforms edition. llvm-svn: 207196
2014-04-25 13:29:35 +08:00
if (SplatValue != nullptr &&
Reduce code duplication resulting from the ConstantVector/ConstantDataVector split. No intended functionality change. llvm-svn: 201344
2014-02-14 00:48:38 +08:00
SplatValue == cast<Constant>(JOp)->getSplatValue())
Op2VK = TargetTransformInfo::OK_UniformConstantValue;
[Vectorizer] Add a new 'OperandValueKind' in TargetTransformInfo called 'OK_NonUniformConstValue' to identify operands which are constants but not constant splats. The cost model now allows returning 'OK_NonUniformConstValue' for non splat operands that are instances of ConstantVector or ConstantDataVector. With this change, targets are now able to compute different costs for instructions with non-uniform constant operands. For example, On X86 the cost of a vector shift may vary depending on whether the second operand is a uniform or non-uniform constant. This patch applies the following changes: - The cost model computation now takes into account non-uniform constants; - The cost of vector shift instructions has been improved in X86TargetTransformInfo analysis pass; - BBVectorize, SLPVectorizer and LoopVectorize now know how to distinguish between non-uniform and uniform constant operands. Added a new test to verify that the output of opt '-cost-model -analyze' is valid in the following configurations: SSE2, SSE4.1, AVX, AVX2. llvm-svn: 201272
2014-02-13 07:43:47 +08:00
}
}
}
BBVectorize: Add a additional comment about the cost computation llvm-svn: 173580
2013-01-27 00:49:04 +08:00
// Note that this procedure is incorrect for insert and extract element
// instructions (because combining these often results in a shuffle),
// but this cost is ignored (because insert and extract element
// instructions are assigned a zero depth factor and are not really
// fused in general).
[SystemZ] TargetTransformInfo cost functions implemented. getArithmeticInstrCost(), getShuffleCost(), getCastInstrCost(), getCmpSelInstrCost(), getVectorInstrCost(), getMemoryOpCost(), getInterleavedMemoryOpCost() implemented. Interleaved access vectorization enabled. BasicTTIImpl::getCastInstrCost() improved to check for legal extending loads, in which case the cost of the z/sext instruction becomes 0. Review: Ulrich Weigand, Renato Golin. https://reviews.llvm.org/D29631 llvm-svn: 300052
2017-04-12 19:49:08 +08:00
unsigned VCost = getInstrCost(I->getOpcode(), VT1, VT2, Op1VK, Op2VK, I);
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
if (VCost > ICost + JCost)
return false;
BBVectorize, when using VTTI, should not form types that will be split. This is needed so that perl's SHA can be compiled (otherwise BBVectorize takes far too long to find its fixed point). I'll try to come up with a reduced test case. llvm-svn: 166738
2012-10-26 07:47:16 +08:00
Use VTTI->getNumberOfParts in BBVectorize. This change reflects VTTI refactoring; no functionality change intended. llvm-svn: 166752
2012-10-26 12:28:06 +08:00
// We don't want to fuse to a type that will be split, even
BBVectorize, when using VTTI, should not form types that will be split. This is needed so that perl's SHA can be compiled (otherwise BBVectorize takes far too long to find its fixed point). I'll try to come up with a reduced test case. llvm-svn: 166738
2012-10-26 07:47:16 +08:00
// if the two input types will also be split and there is no other
Use VTTI->getNumberOfParts in BBVectorize. This change reflects VTTI refactoring; no functionality change intended. llvm-svn: 166752
2012-10-26 12:28:06 +08:00
// associated cost.
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
unsigned VParts1 = TTI->getNumberOfParts(VT1),
VParts2 = TTI->getNumberOfParts(VT2);
BBVectorize: Check the input types of shuffles for legality This fixes a bug where shuffles were being fused such that the resulting input types were not legal on the target. This would occur only when both inputs and dependencies were also foldable operations (such as other shuffles) and there were other connected pairs in the same block. llvm-svn: 167731
2012-11-12 22:50:59 +08:00
if (VParts1 > 1 || VParts2 > 1)
Use VTTI->getNumberOfParts in BBVectorize. This change reflects VTTI refactoring; no functionality change intended. llvm-svn: 166752
2012-10-26 12:28:06 +08:00
return false;
BBVectorize: Check the input types of shuffles for legality This fixes a bug where shuffles were being fused such that the resulting input types were not legal on the target. This would occur only when both inputs and dependencies were also foldable operations (such as other shuffles) and there were other connected pairs in the same block. llvm-svn: 167731
2012-11-12 22:50:59 +08:00
else if ((!VParts1 || !VParts2) && VCost == ICost + JCost)
BBVectorize, when using VTTI, should not form types that will be split. This is needed so that perl's SHA can be compiled (otherwise BBVectorize takes far too long to find its fixed point). I'll try to come up with a reduced test case. llvm-svn: 166738
2012-10-26 07:47:16 +08:00
return false;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
CostSavings = ICost + JCost - VCost;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Allow vectorization of bit intrinsics in BB Vectorizer. This patch adds support for vectorization of bit intrinsics such as bswap,ctpop,ctlz,cttz. llvm-svn: 207174
2014-04-25 11:33:48 +08:00
// The powi,ctlz,cttz intrinsics are special because only the first
// argument is vectorized, the second arguments must be equal.
Correctly vectorize powi. The powi intrinsic requires special handling because it always takes a single integer power regardless of the result type. As a result, we can vectorize only if the powers are equal. Fixes PR12364. llvm-svn: 153797
2012-03-31 11:38:40 +08:00
CallInst *CI = dyn_cast<CallInst>(I);
Function *FI;
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
if (CI && (FI = CI->getCalledFunction())) {
Change Function::getIntrinsicID() to return an Intrinsic::ID. NFC. Now that Intrinsic::ID is a typed enum, we can forward declare it and so return it from this method. This updates all users which were either using an unsigned to store it, or had a now unnecessary cast. llvm-svn: 237810
2015-05-21 01:16:39 +08:00
Intrinsic::ID IID = FI->getIntrinsicID();
Allow vectorization of bit intrinsics in BB Vectorizer. This patch adds support for vectorization of bit intrinsics such as bswap,ctpop,ctlz,cttz. llvm-svn: 207174
2014-04-25 11:33:48 +08:00
if (IID == Intrinsic::powi || IID == Intrinsic::ctlz ||
IID == Intrinsic::cttz) {
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
Value *A1I = CI->getArgOperand(1),
*A1J = cast<CallInst>(J)->getArgOperand(1);
const SCEV *A1ISCEV = SE->getSCEV(A1I),
*A1JSCEV = SE->getSCEV(A1J);
return (A1ISCEV == A1JSCEV);
}
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
if (IID && TTI) {
[CodeGen] Teach LLVM how to lower @llvm.{min,max}num to {MIN,MAX}NAN The behavior of {MIN,MAX}NAN differs from that of {MIN,MAX}NUM when only one of the inputs is NaN: -NUM will return the non-NaN argument while -NAN would return NaN. It is desirable to lower to @llvm.{min,max}num to -NAN if they don't have a native instruction for -NUM. Notably, ARMv7 NEON's vmin has the -NAN semantics. N.B. Of course, it is only safe to do this if the intrinsic call is marked nnan. llvm-svn: 266279
2016-04-14 15:13:24 +08:00
FastMathFlags FMFCI;
if (auto *FPMOCI = dyn_cast<FPMathOperator>(CI))
FMFCI = FPMOCI->getFastMathFlags();
[TargetTransformInfo] getIntrinsicInstrCost() scalarization estimation improved getIntrinsicInstrCost() used to only compute scalarization cost based on types. This patch improves this so that the actual arguments are checked when they are available, in order to handle only unique non-constant operands. Tests updates: Analysis/CostModel/X86/arith-fp.ll Transforms/LoopVectorize/AArch64/interleaved_cost.ll Transforms/LoopVectorize/ARM/interleaved_cost.ll The improvement in getOperandsScalarizationOverhead() to differentiate on constants made it necessary to update the interleaved_cost.ll tests even though they do not relate to intrinsics. Review: Hal Finkel https://reviews.llvm.org/D29540 llvm-svn: 297705
2017-03-14 14:35:36 +08:00
SmallVector<Value *, 4> IArgs(CI->arg_operands());
unsigned ICost = TTI->getIntrinsicInstrCost(IID, IT1, IArgs, FMFCI);
[CodeGen] Teach LLVM how to lower @llvm.{min,max}num to {MIN,MAX}NAN The behavior of {MIN,MAX}NAN differs from that of {MIN,MAX}NUM when only one of the inputs is NaN: -NUM will return the non-NaN argument while -NAN would return NaN. It is desirable to lower to @llvm.{min,max}num to -NAN if they don't have a native instruction for -NUM. Notably, ARMv7 NEON's vmin has the -NAN semantics. N.B. Of course, it is only safe to do this if the intrinsic call is marked nnan. llvm-svn: 266279
2016-04-14 15:13:24 +08:00
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
CallInst *CJ = cast<CallInst>(J);
[CodeGen] Teach LLVM how to lower @llvm.{min,max}num to {MIN,MAX}NAN The behavior of {MIN,MAX}NAN differs from that of {MIN,MAX}NUM when only one of the inputs is NaN: -NUM will return the non-NaN argument while -NAN would return NaN. It is desirable to lower to @llvm.{min,max}num to -NAN if they don't have a native instruction for -NUM. Notably, ARMv7 NEON's vmin has the -NAN semantics. N.B. Of course, it is only safe to do this if the intrinsic call is marked nnan. llvm-svn: 266279
2016-04-14 15:13:24 +08:00
FastMathFlags FMFCJ;
if (auto *FPMOCJ = dyn_cast<FPMathOperator>(CJ))
FMFCJ = FPMOCJ->getFastMathFlags();
[TargetTransformInfo] getIntrinsicInstrCost() scalarization estimation improved getIntrinsicInstrCost() used to only compute scalarization cost based on types. This patch improves this so that the actual arguments are checked when they are available, in order to handle only unique non-constant operands. Tests updates: Analysis/CostModel/X86/arith-fp.ll Transforms/LoopVectorize/AArch64/interleaved_cost.ll Transforms/LoopVectorize/ARM/interleaved_cost.ll The improvement in getOperandsScalarizationOverhead() to differentiate on constants made it necessary to update the interleaved_cost.ll tests even though they do not relate to intrinsics. Review: Hal Finkel https://reviews.llvm.org/D29540 llvm-svn: 297705
2017-03-14 14:35:36 +08:00
SmallVector<Value *, 4> JArgs(CJ->arg_operands());
unsigned JCost = TTI->getIntrinsicInstrCost(IID, JT1, JArgs, FMFCJ);
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
assert(CI->getNumArgOperands() == CJ->getNumArgOperands() &&
"Intrinsic argument counts differ");
[TargetTransformInfo] getIntrinsicInstrCost() scalarization estimation improved getIntrinsicInstrCost() used to only compute scalarization cost based on types. This patch improves this so that the actual arguments are checked when they are available, in order to handle only unique non-constant operands. Tests updates: Analysis/CostModel/X86/arith-fp.ll Transforms/LoopVectorize/AArch64/interleaved_cost.ll Transforms/LoopVectorize/ARM/interleaved_cost.ll The improvement in getOperandsScalarizationOverhead() to differentiate on constants made it necessary to update the interleaved_cost.ll tests even though they do not relate to intrinsics. Review: Hal Finkel https://reviews.llvm.org/D29540 llvm-svn: 297705
2017-03-14 14:35:36 +08:00
SmallVector<Type*, 4> Tys;
SmallVector<Value *, 4> VecArgs;
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
for (unsigned i = 0, ie = CI->getNumArgOperands(); i != ie; ++i) {
Allow vectorization of bit intrinsics in BB Vectorizer. This patch adds support for vectorization of bit intrinsics such as bswap,ctpop,ctlz,cttz. llvm-svn: 207174
2014-04-25 11:33:48 +08:00
if ((IID == Intrinsic::powi || IID == Intrinsic::ctlz ||
[TargetTransformInfo] getIntrinsicInstrCost() scalarization estimation improved getIntrinsicInstrCost() used to only compute scalarization cost based on types. This patch improves this so that the actual arguments are checked when they are available, in order to handle only unique non-constant operands. Tests updates: Analysis/CostModel/X86/arith-fp.ll Transforms/LoopVectorize/AArch64/interleaved_cost.ll Transforms/LoopVectorize/ARM/interleaved_cost.ll The improvement in getOperandsScalarizationOverhead() to differentiate on constants made it necessary to update the interleaved_cost.ll tests even though they do not relate to intrinsics. Review: Hal Finkel https://reviews.llvm.org/D29540 llvm-svn: 297705
2017-03-14 14:35:36 +08:00
IID == Intrinsic::cttz) && i == 1) {
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
Tys.push_back(CI->getArgOperand(i)->getType());
[TargetTransformInfo] getIntrinsicInstrCost() scalarization estimation improved getIntrinsicInstrCost() used to only compute scalarization cost based on types. This patch improves this so that the actual arguments are checked when they are available, in order to handle only unique non-constant operands. Tests updates: Analysis/CostModel/X86/arith-fp.ll Transforms/LoopVectorize/AArch64/interleaved_cost.ll Transforms/LoopVectorize/ARM/interleaved_cost.ll The improvement in getOperandsScalarizationOverhead() to differentiate on constants made it necessary to update the interleaved_cost.ll tests even though they do not relate to intrinsics. Review: Hal Finkel https://reviews.llvm.org/D29540 llvm-svn: 297705
2017-03-14 14:35:36 +08:00
VecArgs.push_back(CI->getArgOperand(i));
}
else {
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
Tys.push_back(getVecTypeForPair(CI->getArgOperand(i)->getType(),
CJ->getArgOperand(i)->getType()));
[TargetTransformInfo] getIntrinsicInstrCost() scalarization estimation improved getIntrinsicInstrCost() used to only compute scalarization cost based on types. This patch improves this so that the actual arguments are checked when they are available, in order to handle only unique non-constant operands. Tests updates: Analysis/CostModel/X86/arith-fp.ll Transforms/LoopVectorize/AArch64/interleaved_cost.ll Transforms/LoopVectorize/ARM/interleaved_cost.ll The improvement in getOperandsScalarizationOverhead() to differentiate on constants made it necessary to update the interleaved_cost.ll tests even though they do not relate to intrinsics. Review: Hal Finkel https://reviews.llvm.org/D29540 llvm-svn: 297705
2017-03-14 14:35:36 +08:00
// Add both operands, and then count their scalarization overhead
// with VF 1.
VecArgs.push_back(CI->getArgOperand(i));
VecArgs.push_back(CJ->getArgOperand(i));
}
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
}
[TargetTransformInfo] getIntrinsicInstrCost() scalarization estimation improved getIntrinsicInstrCost() used to only compute scalarization cost based on types. This patch improves this so that the actual arguments are checked when they are available, in order to handle only unique non-constant operands. Tests updates: Analysis/CostModel/X86/arith-fp.ll Transforms/LoopVectorize/AArch64/interleaved_cost.ll Transforms/LoopVectorize/ARM/interleaved_cost.ll The improvement in getOperandsScalarizationOverhead() to differentiate on constants made it necessary to update the interleaved_cost.ll tests even though they do not relate to intrinsics. Review: Hal Finkel https://reviews.llvm.org/D29540 llvm-svn: 297705
2017-03-14 14:35:36 +08:00
// Compute the scalarization cost here with the original operands (to
// check for uniqueness etc), and then call getIntrinsicInstrCost()
// with the constructed vector types.
Type *RetTy = getVecTypeForPair(IT1, JT1);
unsigned ScalarizationCost = 0;
if (!RetTy->isVoidTy())
ScalarizationCost += TTI->getScalarizationOverhead(RetTy, true, false);
ScalarizationCost += TTI->getOperandsScalarizationOverhead(VecArgs, 1);
[CodeGen] Teach LLVM how to lower @llvm.{min,max}num to {MIN,MAX}NAN The behavior of {MIN,MAX}NAN differs from that of {MIN,MAX}NUM when only one of the inputs is NaN: -NUM will return the non-NaN argument while -NAN would return NaN. It is desirable to lower to @llvm.{min,max}num to -NAN if they don't have a native instruction for -NUM. Notably, ARMv7 NEON's vmin has the -NAN semantics. N.B. Of course, it is only safe to do this if the intrinsic call is marked nnan. llvm-svn: 266279
2016-04-14 15:13:24 +08:00
FastMathFlags FMFV = FMFCI;
FMFV &= FMFCJ;
[TargetTransformInfo] getIntrinsicInstrCost() scalarization estimation improved getIntrinsicInstrCost() used to only compute scalarization cost based on types. This patch improves this so that the actual arguments are checked when they are available, in order to handle only unique non-constant operands. Tests updates: Analysis/CostModel/X86/arith-fp.ll Transforms/LoopVectorize/AArch64/interleaved_cost.ll Transforms/LoopVectorize/ARM/interleaved_cost.ll The improvement in getOperandsScalarizationOverhead() to differentiate on constants made it necessary to update the interleaved_cost.ll tests even though they do not relate to intrinsics. Review: Hal Finkel https://reviews.llvm.org/D29540 llvm-svn: 297705
2017-03-14 14:35:36 +08:00
unsigned VCost = TTI->getIntrinsicInstrCost(IID, RetTy, Tys, FMFV,
ScalarizationCost);
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
if (VCost > ICost + JCost)
return false;
// We don't want to fuse to a type that will be split, even
// if the two input types will also be split and there is no other
// associated cost.
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
unsigned RetParts = TTI->getNumberOfParts(RetTy);
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
if (RetParts > 1)
return false;
else if (!RetParts && VCost == ICost + JCost)
return false;
for (unsigned i = 0, ie = CI->getNumArgOperands(); i != ie; ++i) {
if (!Tys[i]->isVectorTy())
continue;
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
unsigned NumParts = TTI->getNumberOfParts(Tys[i]);
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
if (NumParts > 1)
return false;
else if (!NumParts && VCost == ICost + JCost)
return false;
}
CostSavings = ICost + JCost - VCost;
}
Correctly vectorize powi. The powi intrinsic requires special handling because it always takes a single integer power regardless of the result type. As a result, we can vectorize only if the powers are equal. Fixes PR12364. llvm-svn: 153797
2012-03-31 11:38:40 +08:00
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return true;
}
// Figure out whether or not J uses I and update the users and write-set
// structures associated with I. Specifically, Users represents the set of
// instructions that depend on I. WriteSet represents the set
// of memory locations that are dependent on I. If UpdateUsers is true,
// and J uses I, then Users is updated to contain J and WriteSet is updated
// to contain any memory locations to which J writes. The function returns
// true if J uses I. By default, alias analysis is used to determine
// whether J reads from memory that overlaps with a location in WriteSet.
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
// If LoadMoveSet is not null, then it is a previously-computed map
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// where the key is the memory-based user instruction and the value is
// the instruction to be compared with I. So, if LoadMoveSet is provided,
// then the alias analysis is not used. This is necessary because this
// function is called during the process of moving instructions during
// vectorization and the results of the alias analysis are not stable during
// that process.
bool BBVectorize::trackUsesOfI(DenseSet<Value *> &Users,
AliasSetTracker &WriteSet, Instruction *I,
Instruction *J, bool UpdateUsers,
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
DenseSet<ValuePair> *LoadMoveSetPairs) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool UsesI = false;
// This instruction may already be marked as a user due, for example, to
// being a member of a selected pair.
if (Users.count(J))
UsesI = true;
if (!UsesI)
A few of the changes suggested in code review (by Nick Lewycky) llvm-svn: 149472
2012-02-01 13:51:45 +08:00
for (User::op_iterator JU = J->op_begin(), JE = J->op_end();
JU != JE; ++JU) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Value *V = *JU;
if (I == V || Users.count(V)) {
UsesI = true;
break;
}
}
if (!UsesI && J->mayReadFromMemory()) {
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
if (LoadMoveSetPairs) {
UsesI = LoadMoveSetPairs->count(ValuePair(J, I));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
} else {
for (AliasSetTracker::iterator W = WriteSet.begin(),
WE = WriteSet.end(); W != WE; ++W) {
Update BBVectorize to use aliasesUnknownInst. This allows BBVectorize to check the "unknown instruction" list in the alias sets. This is important to prevent instruction fusing from reordering function calls. Resolves PR11920. llvm-svn: 150250
2012-02-10 23:52:40 +08:00
if (W->aliasesUnknownInst(J, *AA)) {
UsesI = true;
break;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
}
}
if (UsesI && UpdateUsers) {
if (J->mayWriteToMemory()) WriteSet.add(J);
Users.insert(J);
}
return UsesI;
}
// This function iterates over all instruction pairs in the provided
// basic block and collects all candidate pairs for vectorization.
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
bool BBVectorize::getCandidatePairs(BasicBlock &BB,
BasicBlock::iterator &Start,
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
DenseSet<ValuePair> &FixedOrderPairs,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
DenseMap<ValuePair, int> &CandidatePairCostSavings,
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
std::vector<Value *> &PairableInsts, bool NonPow2Len) {
BBVectorize: Cap the number of candidate pairs in each instruction group For some basic blocks, it is possible to generate many candidate pairs for relatively few pairable instructions. When many (tens of thousands) of these pairs are generated for a single instruction group, the time taken to generate and rank the different vectorization plans can become quite large. As a result, we now cap the number of candidate pairs within each instruction group. This is done by closing out the group once the threshold is reached (set now at 3000 pairs). Although this will limit the overall compile-time impact, this may not be the best way to achieve this result. It might be better, for example, to prune excessive candidate pairs after the fact the prevent the generation of short, but highly-connected groups. We can experiment with this in the future. This change reduces the overall compile-time slowdown of the csa.ll test case in PR15222 to ~5x. If 5x is still considered too large, a lower limit can be used as the default. This represents a functionality change, but only for very large inputs (thus, there is no regression test). llvm-svn: 175251
2013-02-15 12:28:42 +08:00
size_t TotalPairs = 0;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BasicBlock::iterator E = BB.end();
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
if (Start == E) return false;
bool ShouldContinue = false, IAfterStart = false;
for (BasicBlock::iterator I = Start++; I != E; ++I) {
if (I == Start) IAfterStart = true;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
bool IsSimpleLoadStore;
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
if (!isInstVectorizable(&*I, IsSimpleLoadStore))
continue;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Look for an instruction with which to pair instruction *I...
DenseSet<Value *> Users;
AliasSetTracker WriteSet(*AA);
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
if (I->mayWriteToMemory())
WriteSet.add(&*I);
BBVectorize: Add initial stores to the write set when tracking uses When computing the use set of a store, we need to add the store to the write set prior to iterating over later instructions. Otherwise, if there is a later aliasing load of that store, that load will not be tagged as a use, and bad things will happen. trackUsesOfI still adds later dependent stores of an instruction to that instruction's write set, but it never sees the original instruction, and so when tracking uses of a store, the store must be added to the write set by the caller. Fixes PR16834. llvm-svn: 188329
2013-08-14 07:34:32 +08:00
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
bool JAfterStart = IAfterStart;
[C++11] Replace llvm::next and llvm::prior with std::next and std::prev. Remove the old functions. llvm-svn: 202636
2014-03-02 20:27:27 +08:00
BasicBlock::iterator J = std::next(I);
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
for (unsigned ss = 0; J != E && ss <= Config.SearchLimit; ++J, ++ss) {
ADT: Remove == and != comparisons between ilist iterators and pointers I missed == and != when I removed implicit conversions between iterators and pointers in r252380 since they were defined outside ilist_iterator. Since they depend on getNodePtrUnchecked(), they indirectly rely on UB. This commit removes all uses of these operators. (I'll delete the operators themselves in a separate commit so that it can be easily reverted if necessary.) There should be NFC here. llvm-svn: 261498
2016-02-22 04:39:50 +08:00
if (J == Start)
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
JAfterStart = true;
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Determine if J uses I, if so, exit the loop.
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
bool UsesI = trackUsesOfI(Users, WriteSet, &*I, &*J, !Config.FastDep);
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
if (Config.FastDep) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Note: For this heuristic to be effective, independent operations
// must tend to be intermixed. This is likely to be true from some
// kinds of grouped loop unrolling (but not the generic LLVM pass),
// but otherwise may require some kind of reordering pass.
// When using fast dependency analysis,
// stop searching after first use:
if (UsesI) break;
} else {
if (UsesI) continue;
}
// J does not use I, and comes before the first use of I, so it can be
// merged with I if the instructions are compatible.
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
int CostSavings, FixedOrder;
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
if (!areInstsCompatible(&*I, &*J, IsSimpleLoadStore, NonPow2Len,
CostSavings, FixedOrder))
continue;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// J is a candidate for merging with I.
Replace size method call of containers to empty method where appropriate This patch was generated by a clang tidy checker that is being open sourced. The documentation of that checker is the following: /// The emptiness of a container should be checked using the empty method /// instead of the size method. It is not guaranteed that size is a /// constant-time function, and it is generally more efficient and also shows /// clearer intent to use empty. Furthermore some containers may implement the /// empty method but not implement the size method. Using empty whenever /// possible makes it easier to switch to another container in the future. Patch by Gábor Horváth! llvm-svn: 226161
2015-01-15 19:41:30 +08:00
if (PairableInsts.empty() ||
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
PairableInsts[PairableInsts.size() - 1] != &*I) {
PairableInsts.push_back(&*I);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
CandidatePairs[&*I].push_back(&*J);
BBVectorize: Cap the number of candidate pairs in each instruction group For some basic blocks, it is possible to generate many candidate pairs for relatively few pairable instructions. When many (tens of thousands) of these pairs are generated for a single instruction group, the time taken to generate and rank the different vectorization plans can become quite large. As a result, we now cap the number of candidate pairs within each instruction group. This is done by closing out the group once the threshold is reached (set now at 3000 pairs). Although this will limit the overall compile-time impact, this may not be the best way to achieve this result. It might be better, for example, to prune excessive candidate pairs after the fact the prevent the generation of short, but highly-connected groups. We can experiment with this in the future. This change reduces the overall compile-time slowdown of the csa.ll test case in PR15222 to ~5x. If 5x is still considered too large, a lower limit can be used as the default. This represents a functionality change, but only for very large inputs (thus, there is no regression test). llvm-svn: 175251
2013-02-15 12:28:42 +08:00
++TotalPairs;
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
if (TTI)
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
CandidatePairCostSavings.insert(
ValuePairWithCost(ValuePair(&*I, &*J), CostSavings));
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
if (FixedOrder == 1)
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
FixedOrderPairs.insert(ValuePair(&*I, &*J));
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
else if (FixedOrder == -1)
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
FixedOrderPairs.insert(ValuePair(&*J, &*I));
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
// The next call to this function must start after the last instruction
// selected during this invocation.
if (JAfterStart) {
[C++11] Replace llvm::next and llvm::prior with std::next and std::prev. Remove the old functions. llvm-svn: 202636
2014-03-02 20:27:27 +08:00
Start = std::next(J);
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
IAfterStart = JAfterStart = false;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DEBUG(if (DebugCandidateSelection) dbgs() << "BBV: candidate pair "
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
<< *I << " <-> " << *J << " (cost savings: " <<
CostSavings << ")\n");
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
// If we have already found too many pairs, break here and this function
// will be called again starting after the last instruction selected
// during this invocation.
BBVectorize: Cap the number of candidate pairs in each instruction group For some basic blocks, it is possible to generate many candidate pairs for relatively few pairable instructions. When many (tens of thousands) of these pairs are generated for a single instruction group, the time taken to generate and rank the different vectorization plans can become quite large. As a result, we now cap the number of candidate pairs within each instruction group. This is done by closing out the group once the threshold is reached (set now at 3000 pairs). Although this will limit the overall compile-time impact, this may not be the best way to achieve this result. It might be better, for example, to prune excessive candidate pairs after the fact the prevent the generation of short, but highly-connected groups. We can experiment with this in the future. This change reduces the overall compile-time slowdown of the csa.ll test case in PR15222 to ~5x. If 5x is still considered too large, a lower limit can be used as the default. This represents a functionality change, but only for very large inputs (thus, there is no regression test). llvm-svn: 175251
2013-02-15 12:28:42 +08:00
if (PairableInsts.size() >= Config.MaxInsts ||
TotalPairs >= Config.MaxPairs) {
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
ShouldContinue = true;
break;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
if (ShouldContinue)
break;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
DEBUG(dbgs() << "BBV: found " << PairableInsts.size()
<< " instructions with candidate pairs\n");
Vectorize long blocks in groups. Long basic blocks with many candidate pairs (such as in the SHA implementation in Perl 5.14; thanks to Roman Divacky for the example) used to take an unacceptably-long time to compile. Instead, break long blocks into groups so that no group has too many candidate pairs. llvm-svn: 149595
2012-02-02 14:14:56 +08:00
return ShouldContinue;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
// Finds candidate pairs connected to the pair P = <PI, PJ>. This means that
// it looks for pairs such that both members have an input which is an
// output of PI or PJ.
void BBVectorize::computePairsConnectedTo(
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
DenseSet<ValuePair> &CandidatePairsSet,
std::vector<Value *> &PairableInsts,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes,
ValuePair P) {
Fix an error in BBVectorize important for vectorizing pointer types. When vectorizing pointer types it is important to realize that potential pairs cannot be connected via the address pointer argument of a load or store. This is because even after vectorization, the address is still a scalar because the address of the higher half of the pair is implicit from the address of the lower half (it need not be, and should not be, explicitly computed). llvm-svn: 154735
2012-04-14 15:32:50 +08:00
StoreInst *SI, *SJ;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// For each possible pairing for this variable, look at the uses of
// the first value...
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
for (Value::user_iterator I = P.first->user_begin(),
E = P.first->user_end();
I != E; ++I) {
User *UI = *I;
if (isa<LoadInst>(UI)) {
Fix an error in BBVectorize important for vectorizing pointer types. When vectorizing pointer types it is important to realize that potential pairs cannot be connected via the address pointer argument of a load or store. This is because even after vectorization, the address is still a scalar because the address of the higher half of the pair is implicit from the address of the lower half (it need not be, and should not be, explicitly computed). llvm-svn: 154735
2012-04-14 15:32:50 +08:00
// A pair cannot be connected to a load because the load only takes one
// operand (the address) and it is a scalar even after vectorization.
continue;
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
} else if ((SI = dyn_cast<StoreInst>(UI)) &&
Fix an error in BBVectorize important for vectorizing pointer types. When vectorizing pointer types it is important to realize that potential pairs cannot be connected via the address pointer argument of a load or store. This is because even after vectorization, the address is still a scalar because the address of the higher half of the pair is implicit from the address of the lower half (it need not be, and should not be, explicitly computed). llvm-svn: 154735
2012-04-14 15:32:50 +08:00
P.first == SI->getPointerOperand()) {
// Similarly, a pair cannot be connected to a store through its
// pointer operand.
continue;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// For each use of the first variable, look for uses of the second
// variable...
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
for (User *UJ : P.second->users()) {
if ((SJ = dyn_cast<StoreInst>(UJ)) &&
Fix an error in BBVectorize important for vectorizing pointer types. When vectorizing pointer types it is important to realize that potential pairs cannot be connected via the address pointer argument of a load or store. This is because even after vectorization, the address is still a scalar because the address of the higher half of the pair is implicit from the address of the lower half (it need not be, and should not be, explicitly computed). llvm-svn: 154735
2012-04-14 15:32:50 +08:00
P.second == SJ->getPointerOperand())
continue;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Look for <I, J>:
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
if (CandidatePairsSet.count(ValuePair(UI, UJ))) {
VPPair VP(P, ValuePair(UI, UJ));
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
ConnectedPairs[VP.first].push_back(VP.second);
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
PairConnectionTypes.insert(VPPairWithType(VP, PairConnectionDirect));
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Look for <J, I>:
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
if (CandidatePairsSet.count(ValuePair(UJ, UI))) {
VPPair VP(P, ValuePair(UJ, UI));
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
ConnectedPairs[VP.first].push_back(VP.second);
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
PairConnectionTypes.insert(VPPairWithType(VP, PairConnectionSwap));
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
if (Config.SplatBreaksChain) continue;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Look for cases where just the first value in the pair is used by
// both members of another pair (splatting).
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
for (Value::user_iterator J = P.first->user_begin(); J != E; ++J) {
User *UJ = *J;
if ((SJ = dyn_cast<StoreInst>(UJ)) &&
Fix an error in BBVectorize important for vectorizing pointer types. When vectorizing pointer types it is important to realize that potential pairs cannot be connected via the address pointer argument of a load or store. This is because even after vectorization, the address is still a scalar because the address of the higher half of the pair is implicit from the address of the lower half (it need not be, and should not be, explicitly computed). llvm-svn: 154735
2012-04-14 15:32:50 +08:00
P.first == SJ->getPointerOperand())
continue;
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
if (CandidatePairsSet.count(ValuePair(UI, UJ))) {
VPPair VP(P, ValuePair(UI, UJ));
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
ConnectedPairs[VP.first].push_back(VP.second);
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
PairConnectionTypes.insert(VPPairWithType(VP, PairConnectionSplat));
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
if (Config.SplatBreaksChain) return;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Look for cases where just the second value in the pair is used by
// both members of another pair (splatting).
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
for (Value::user_iterator I = P.second->user_begin(),
E = P.second->user_end();
I != E; ++I) {
User *UI = *I;
if (isa<LoadInst>(UI))
Fix an error in BBVectorize important for vectorizing pointer types. When vectorizing pointer types it is important to realize that potential pairs cannot be connected via the address pointer argument of a load or store. This is because even after vectorization, the address is still a scalar because the address of the higher half of the pair is implicit from the address of the lower half (it need not be, and should not be, explicitly computed). llvm-svn: 154735
2012-04-14 15:32:50 +08:00
continue;
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
else if ((SI = dyn_cast<StoreInst>(UI)) &&
Fix an error in BBVectorize important for vectorizing pointer types. When vectorizing pointer types it is important to realize that potential pairs cannot be connected via the address pointer argument of a load or store. This is because even after vectorization, the address is still a scalar because the address of the higher half of the pair is implicit from the address of the lower half (it need not be, and should not be, explicitly computed). llvm-svn: 154735
2012-04-14 15:32:50 +08:00
P.second == SI->getPointerOperand())
continue;
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
for (Value::user_iterator J = P.second->user_begin(); J != E; ++J) {
User *UJ = *J;
if ((SJ = dyn_cast<StoreInst>(UJ)) &&
Fix an error in BBVectorize important for vectorizing pointer types. When vectorizing pointer types it is important to realize that potential pairs cannot be connected via the address pointer argument of a load or store. This is because even after vectorization, the address is still a scalar because the address of the higher half of the pair is implicit from the address of the lower half (it need not be, and should not be, explicitly computed). llvm-svn: 154735
2012-04-14 15:32:50 +08:00
P.second == SJ->getPointerOperand())
continue;
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
if (CandidatePairsSet.count(ValuePair(UI, UJ))) {
VPPair VP(P, ValuePair(UI, UJ));
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
ConnectedPairs[VP.first].push_back(VP.second);
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
PairConnectionTypes.insert(VPPairWithType(VP, PairConnectionSplat));
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
}
// This function figures out which pairs are connected. Two pairs are
// connected if some output of the first pair forms an input to both members
// of the second pair.
void BBVectorize::computeConnectedPairs(
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
DenseSet<ValuePair> &CandidatePairsSet,
std::vector<Value *> &PairableInsts,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
for (std::vector<Value *>::iterator PI = PairableInsts.begin(),
PE = PairableInsts.end(); PI != PE; ++PI) {
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
DenseMap<Value *, std::vector<Value *> >::iterator PP =
CandidatePairs.find(*PI);
if (PP == CandidatePairs.end())
continue;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
for (std::vector<Value *>::iterator P = PP->second.begin(),
E = PP->second.end(); P != E; ++P)
BBVectorize: Eliminate one more restricted linear search This eliminates one more linear search over a range of std::multimap entries. This gives a 22% speedup on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 174893
2013-02-12 01:19:34 +08:00
computePairsConnectedTo(CandidatePairs, CandidatePairsSet,
PairableInsts, ConnectedPairs,
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
PairConnectionTypes, ValuePair(*PI, *P));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DEBUG(size_t TotalPairs = 0;
for (DenseMap<ValuePair, std::vector<ValuePair> >::iterator I =
ConnectedPairs.begin(), IE = ConnectedPairs.end(); I != IE; ++I)
TotalPairs += I->second.size();
dbgs() << "BBV: found " << TotalPairs
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
<< " pair connections.\n");
}
// This function builds a set of use tuples such that <A, B> is in the set
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// if B is in the use dag of A. If B is in the use dag of A, then B
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// depends on the output of A.
void BBVectorize::buildDepMap(
BasicBlock &BB,
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
std::vector<Value *> &PairableInsts,
DenseSet<ValuePair> &PairableInstUsers) {
DenseSet<Value *> IsInPair;
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
for (DenseMap<Value *, std::vector<Value *> >::iterator C =
CandidatePairs.begin(), E = CandidatePairs.end(); C != E; ++C) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
IsInPair.insert(C->first);
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
IsInPair.insert(C->second.begin(), C->second.end());
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
BBVectorize: Fix anomalous capital letter in comment llvm-svn: 173579
2013-01-27 00:49:03 +08:00
// Iterate through the basic block, recording all users of each
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// pairable instruction.
BBVectorize: Don't over-search when building the dependency map When building the pairable-instruction dependency map, don't search past the last pairable instruction. For large blocks that have been divided into multiple instruction groups, searching past the last instruction in each group is very wasteful. This gives a 32% speedup on the csa.ll test case from PR15222 (when using 50 instructions in each group). No functionality change intended. llvm-svn: 174915
2013-02-12 07:02:17 +08:00
BasicBlock::iterator E = BB.end(), EL =
BasicBlock::iterator(cast<Instruction>(PairableInsts.back()));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
for (BasicBlock::iterator I = BB.getFirstInsertionPt(); I != E; ++I) {
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
if (IsInPair.find(&*I) == IsInPair.end())
continue;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DenseSet<Value *> Users;
AliasSetTracker WriteSet(*AA);
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
if (I->mayWriteToMemory())
WriteSet.add(&*I);
BBVectorize: Add initial stores to the write set when tracking uses When computing the use set of a store, we need to add the store to the write set prior to iterating over later instructions. Otherwise, if there is a later aliasing load of that store, that load will not be tagged as a use, and bad things will happen. trackUsesOfI still adds later dependent stores of an instruction to that instruction's write set, but it never sees the original instruction, and so when tracking uses of a store, the store must be added to the write set by the caller. Fixes PR16834. llvm-svn: 188329
2013-08-14 07:34:32 +08:00
[C++11] Replace llvm::next and llvm::prior with std::next and std::prev. Remove the old functions. llvm-svn: 202636
2014-03-02 20:27:27 +08:00
for (BasicBlock::iterator J = std::next(I); J != E; ++J) {
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
(void)trackUsesOfI(Users, WriteSet, &*I, &*J);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Don't over-search when building the dependency map When building the pairable-instruction dependency map, don't search past the last pairable instruction. For large blocks that have been divided into multiple instruction groups, searching past the last instruction in each group is very wasteful. This gives a 32% speedup on the csa.ll test case from PR15222 (when using 50 instructions in each group). No functionality change intended. llvm-svn: 174915
2013-02-12 07:02:17 +08:00
if (J == EL)
break;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
for (DenseSet<Value *>::iterator U = Users.begin(), E = Users.end();
BBVectorize: Omit unnecessary entries in PairableInstUsers This map is queried only for instructions in pairs of pairable instructions; so make sure that only pairs of pairable instructions are added to the map. This gives a 3.5% speedup on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 174914
2013-02-12 07:02:09 +08:00
U != E; ++U) {
if (IsInPair.find(*U) == IsInPair.end()) continue;
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
PairableInstUsers.insert(ValuePair(&*I, *U));
BBVectorize: Omit unnecessary entries in PairableInstUsers This map is queried only for instructions in pairs of pairable instructions; so make sure that only pairs of pairable instructions are added to the map. This gives a 3.5% speedup on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 174914
2013-02-12 07:02:09 +08:00
}
BBVectorize: Don't over-search when building the dependency map When building the pairable-instruction dependency map, don't search past the last pairable instruction. For large blocks that have been divided into multiple instruction groups, searching past the last instruction in each group is very wasteful. This gives a 32% speedup on the csa.ll test case from PR15222 (when using 50 instructions in each group). No functionality change intended. llvm-svn: 174915
2013-02-12 07:02:17 +08:00
if (I == EL)
break;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
// Returns true if an input to pair P is an output of pair Q and also an
// input of pair Q is an output of pair P. If this is the case, then these
// two pairs cannot be simultaneously fused.
bool BBVectorize::pairsConflict(ValuePair P, ValuePair Q,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseSet<ValuePair> &PairableInstUsers,
DenseMap<ValuePair, std::vector<ValuePair> > *PairableInstUserMap,
DenseSet<VPPair> *PairableInstUserPairSet) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Two pairs are in conflict if they are mutual Users of eachother.
bool QUsesP = PairableInstUsers.count(ValuePair(P.first, Q.first)) ||
PairableInstUsers.count(ValuePair(P.first, Q.second)) ||
PairableInstUsers.count(ValuePair(P.second, Q.first)) ||
PairableInstUsers.count(ValuePair(P.second, Q.second));
bool PUsesQ = PairableInstUsers.count(ValuePair(Q.first, P.first)) ||
PairableInstUsers.count(ValuePair(Q.first, P.second)) ||
PairableInstUsers.count(ValuePair(Q.second, P.first)) ||
PairableInstUsers.count(ValuePair(Q.second, P.second));
if (PairableInstUserMap) {
// FIXME: The expensive part of the cycle check is not so much the cycle
// check itself but this edge insertion procedure. This needs some
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
// profiling and probably a different data structure.
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (PUsesQ) {
BBVectorize: Make the bookkeeping to support full cycle checking less expensive By itself, this does not have much of an effect, but only because in the default configuration the full cycle checks are used only for small problem sizes. This is part of a general cleanup of uses of iteration over std::multimap ranges only for the purpose of checking membership. No functionality change intended. llvm-svn: 174856
2013-02-11 13:29:41 +08:00
if (PairableInstUserPairSet->insert(VPPair(Q, P)).second)
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
(*PairableInstUserMap)[Q].push_back(P);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
if (QUsesP) {
BBVectorize: Make the bookkeeping to support full cycle checking less expensive By itself, this does not have much of an effect, but only because in the default configuration the full cycle checks are used only for small problem sizes. This is part of a general cleanup of uses of iteration over std::multimap ranges only for the purpose of checking membership. No functionality change intended. llvm-svn: 174856
2013-02-11 13:29:41 +08:00
if (PairableInstUserPairSet->insert(VPPair(P, Q)).second)
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
(*PairableInstUserMap)[P].push_back(Q);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
return (QUsesP && PUsesQ);
}
// This function walks the use graph of current pairs to see if, starting
// from P, the walk returns to P.
bool BBVectorize::pairWillFormCycle(ValuePair P,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<ValuePair, std::vector<ValuePair> > &PairableInstUserMap,
DenseSet<ValuePair> &CurrentPairs) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DEBUG(if (DebugCycleCheck)
dbgs() << "BBV: starting cycle check for : " << *P.first << " <-> "
<< *P.second << "\n");
// A lookup table of visisted pairs is kept because the PairableInstUserMap
// contains non-direct associations.
DenseSet<ValuePair> Visited;
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
SmallVector<ValuePair, 32> Q;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// General depth-first post-order traversal:
Q.push_back(P);
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
do {
ValuePair QTop = Q.pop_back_val();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Visited.insert(QTop);
DEBUG(if (DebugCycleCheck)
dbgs() << "BBV: cycle check visiting: " << *QTop.first << " <-> "
<< *QTop.second << "\n");
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<ValuePair, std::vector<ValuePair> >::iterator QQ =
PairableInstUserMap.find(QTop);
if (QQ == PairableInstUserMap.end())
continue;
for (std::vector<ValuePair>::iterator C = QQ->second.begin(),
CE = QQ->second.end(); C != CE; ++C) {
if (*C == P) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DEBUG(dbgs()
<< "BBV: rejected to prevent non-trivial cycle formation: "
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
<< QTop.first << " <-> " << C->second << "\n");
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return true;
}
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
if (CurrentPairs.count(*C) && !Visited.count(*C))
Q.push_back(*C);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
} while (!Q.empty());
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return false;
}
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// This function builds the initial dag of connected pairs with the
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// pair J at the root.
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
void BBVectorize::buildInitialDAGFor(
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
DenseSet<ValuePair> &CandidatePairsSet,
std::vector<Value *> &PairableInsts,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairs,
DenseSet<ValuePair> &PairableInstUsers,
DenseMap<Value *, Value *> &ChosenPairs,
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
DenseMap<ValuePair, size_t> &DAG, ValuePair J) {
// Each of these pairs is viewed as the root node of a DAG. The DAG
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// is then walked (depth-first). As this happens, we keep track of
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// the pairs that compose the DAG and the maximum depth of the DAG.
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
SmallVector<ValuePairWithDepth, 32> Q;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// General depth-first post-order traversal:
Q.push_back(ValuePairWithDepth(J, getDepthFactor(J.first)));
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
do {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
ValuePairWithDepth QTop = Q.back();
// Push each child onto the queue:
bool MoreChildren = false;
size_t MaxChildDepth = QTop.second;
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<ValuePair, std::vector<ValuePair> >::iterator QQ =
ConnectedPairs.find(QTop.first);
if (QQ != ConnectedPairs.end())
for (std::vector<ValuePair>::iterator k = QQ->second.begin(),
ke = QQ->second.end(); k != ke; ++k) {
// Make sure that this child pair is still a candidate:
if (CandidatePairsSet.count(*k)) {
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
DenseMap<ValuePair, size_t>::iterator C = DAG.find(*k);
if (C == DAG.end()) {
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
size_t d = getDepthFactor(k->first);
Q.push_back(ValuePairWithDepth(*k, QTop.second+d));
MoreChildren = true;
} else {
MaxChildDepth = std::max(MaxChildDepth, C->second);
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
if (!MoreChildren) {
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// Record the current pair as part of the DAG:
DAG.insert(ValuePairWithDepth(QTop.first, MaxChildDepth));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Q.pop_back();
}
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
} while (!Q.empty());
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// Given some initial dag, prune it by removing conflicting pairs (pairs
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// that cannot be simultaneously chosen for vectorization).
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
void BBVectorize::pruneDAGFor(
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
std::vector<Value *> &PairableInsts,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairs,
DenseSet<ValuePair> &PairableInstUsers,
DenseMap<ValuePair, std::vector<ValuePair> > &PairableInstUserMap,
DenseSet<VPPair> &PairableInstUserPairSet,
DenseMap<Value *, Value *> &ChosenPairs,
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
DenseMap<ValuePair, size_t> &DAG,
DenseSet<ValuePair> &PrunedDAG, ValuePair J,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
bool UseCycleCheck) {
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
SmallVector<ValuePairWithDepth, 32> Q;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// General depth-first post-order traversal:
Q.push_back(ValuePairWithDepth(J, getDepthFactor(J.first)));
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
do {
ValuePairWithDepth QTop = Q.pop_back_val();
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
PrunedDAG.insert(QTop.first);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Visit each child, pruning as necessary...
Replace std::vector -> SmallVector in BBVectorize For now, this uses 8 on-stack elements. I'll need to do some profiling to see if this is the best number. Pointed out by Jakob in post-commit review. llvm-svn: 167966
2012-11-15 03:53:27 +08:00
SmallVector<ValuePairWithDepth, 8> BestChildren;
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<ValuePair, std::vector<ValuePair> >::iterator QQ =
ConnectedPairs.find(QTop.first);
if (QQ == ConnectedPairs.end())
continue;
for (std::vector<ValuePair>::iterator K = QQ->second.begin(),
KE = QQ->second.end(); K != KE; ++K) {
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
DenseMap<ValuePair, size_t>::iterator C = DAG.find(*K);
if (C == DAG.end()) continue;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// This child is in the DAG, now we need to make sure it is the
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// best of any conflicting children. There could be multiple
// conflicting children, so first, determine if we're keeping
// this child, then delete conflicting children as necessary.
// It is also necessary to guard against pairing-induced
// dependencies. Consider instructions a .. x .. y .. b
// such that (a,b) are to be fused and (x,y) are to be fused
// but a is an input to x and b is an output from y. This
// means that y cannot be moved after b but x must be moved
// after b for (a,b) to be fused. In other words, after
// fusing (a,b) we have y .. a/b .. x where y is an input
// to a/b and x is an output to a/b: x and y can no longer
// be legally fused. To prevent this condition, we must
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// make sure that a child pair added to the DAG is not
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// both an input and output of an already-selected pair.
// Pairing-induced dependencies can also form from more complicated
// cycles. The pair vs. pair conflicts are easy to check, and so
// that is done explicitly for "fast rejection", and because for
// child vs. child conflicts, we may prefer to keep the current
// pair in preference to the already-selected child.
DenseSet<ValuePair> CurrentPairs;
bool CanAdd = true;
Use SmallVectorImpl::iterator/const_iterator instead of SmallVector to avoid specifying the vector size. llvm-svn: 185606
2013-07-04 09:31:24 +08:00
for (SmallVectorImpl<ValuePairWithDepth>::iterator C2
fix typos llvm-svn: 152035
2012-03-06 01:39:47 +08:00
= BestChildren.begin(), E2 = BestChildren.end();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
C2 != E2; ++C2) {
if (C2->first.first == C->first.first ||
C2->first.first == C->first.second ||
C2->first.second == C->first.first ||
C2->first.second == C->first.second ||
pairsConflict(C2->first, C->first, PairableInstUsers,
[C++] Use 'nullptr'. Transforms edition. llvm-svn: 207196
2014-04-25 13:29:35 +08:00
UseCycleCheck ? &PairableInstUserMap : nullptr,
UseCycleCheck ? &PairableInstUserPairSet
: nullptr)) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (C2->second >= C->second) {
CanAdd = false;
break;
}
CurrentPairs.insert(C2->first);
}
}
if (!CanAdd) continue;
// Even worse, this child could conflict with another node already
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// selected for the DAG. If that is the case, ignore this child.
for (DenseSet<ValuePair>::iterator T = PrunedDAG.begin(),
E2 = PrunedDAG.end(); T != E2; ++T) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (T->first == C->first.first ||
T->first == C->first.second ||
T->second == C->first.first ||
T->second == C->first.second ||
pairsConflict(*T, C->first, PairableInstUsers,
[C++] Use 'nullptr'. Transforms edition. llvm-svn: 207196
2014-04-25 13:29:35 +08:00
UseCycleCheck ? &PairableInstUserMap : nullptr,
UseCycleCheck ? &PairableInstUserPairSet
: nullptr)) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
CanAdd = false;
break;
}
CurrentPairs.insert(*T);
}
if (!CanAdd) continue;
// And check the queue too...
Use SmallVectorImpl::iterator/const_iterator instead of SmallVector to avoid specifying the vector size. llvm-svn: 185606
2013-07-04 09:31:24 +08:00
for (SmallVectorImpl<ValuePairWithDepth>::iterator C2 = Q.begin(),
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
E2 = Q.end(); C2 != E2; ++C2) {
if (C2->first.first == C->first.first ||
C2->first.first == C->first.second ||
C2->first.second == C->first.first ||
C2->first.second == C->first.second ||
pairsConflict(C2->first, C->first, PairableInstUsers,
[C++] Use 'nullptr'. Transforms edition. llvm-svn: 207196
2014-04-25 13:29:35 +08:00
UseCycleCheck ? &PairableInstUserMap : nullptr,
UseCycleCheck ? &PairableInstUserPairSet
: nullptr)) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
CanAdd = false;
break;
}
CurrentPairs.insert(C2->first);
}
if (!CanAdd) continue;
// Last but not least, check for a conflict with any of the
// already-chosen pairs.
for (DenseMap<Value *, Value *>::iterator C2 =
ChosenPairs.begin(), E2 = ChosenPairs.end();
C2 != E2; ++C2) {
if (pairsConflict(*C2, C->first, PairableInstUsers,
[C++] Use 'nullptr'. Transforms edition. llvm-svn: 207196
2014-04-25 13:29:35 +08:00
UseCycleCheck ? &PairableInstUserMap : nullptr,
UseCycleCheck ? &PairableInstUserPairSet
: nullptr)) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
CanAdd = false;
break;
}
CurrentPairs.insert(*C2);
}
if (!CanAdd) continue;
fix indentation llvm-svn: 149857
2012-02-06 13:29:32 +08:00
// To check for non-trivial cycles formed by the addition of the
// current pair we've formed a list of all relevant pairs, now use a
// graph walk to check for a cycle. We start from the current pair and
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// walk the use dag to see if we again reach the current pair. If we
fix indentation llvm-svn: 149857
2012-02-06 13:29:32 +08:00
// do, then the current pair is rejected.
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// FIXME: It may be more efficient to use a topological-ordering
// algorithm to improve the cycle check. This should be investigated.
if (UseCycleCheck &&
pairWillFormCycle(C->first, PairableInstUserMap, CurrentPairs))
continue;
// This child can be added, but we may have chosen it in preference
// to an already-selected child. Check for this here, and if a
// conflict is found, then remove the previously-selected child
// before adding this one in its place.
Use SmallVectorImpl::iterator/const_iterator instead of SmallVector to avoid specifying the vector size. llvm-svn: 185606
2013-07-04 09:31:24 +08:00
for (SmallVectorImpl<ValuePairWithDepth>::iterator C2
fix typos llvm-svn: 152035
2012-03-06 01:39:47 +08:00
= BestChildren.begin(); C2 != BestChildren.end();) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (C2->first.first == C->first.first ||
C2->first.first == C->first.second ||
C2->first.second == C->first.first ||
C2->first.second == C->first.second ||
pairsConflict(C2->first, C->first, PairableInstUsers))
Fix the largest offender of determinism in BBVectorize Iterating over the children of each node in the potential vectorization plan must happen in a deterministic order (because it affects which children are erased when two children conflict). There was no need for this data structure to be a map in the first place, so replacing it with a vector is a small change. I believe that this was the last remaining instance if iterating over the elements of a Dense* container where the iteration order could matter. There are some remaining iterations over std::*map containers where the order might matter, but so long as the Value* for instructions in a block increase with the order of the instructions in the block (or decrease) monotonically, then this will appear to be deterministic. llvm-svn: 167942
2012-11-15 02:38:11 +08:00
C2 = BestChildren.erase(C2);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
else
++C2;
}
Fix the largest offender of determinism in BBVectorize Iterating over the children of each node in the potential vectorization plan must happen in a deterministic order (because it affects which children are erased when two children conflict). There was no need for this data structure to be a map in the first place, so replacing it with a vector is a small change. I believe that this was the last remaining instance if iterating over the elements of a Dense* container where the iteration order could matter. There are some remaining iterations over std::*map containers where the order might matter, but so long as the Value* for instructions in a block increase with the order of the instructions in the block (or decrease) monotonically, then this will appear to be deterministic. llvm-svn: 167942
2012-11-15 02:38:11 +08:00
BestChildren.push_back(ValuePairWithDepth(C->first, C->second));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Use SmallVectorImpl::iterator/const_iterator instead of SmallVector to avoid specifying the vector size. llvm-svn: 185606
2013-07-04 09:31:24 +08:00
for (SmallVectorImpl<ValuePairWithDepth>::iterator C
fix typos llvm-svn: 152035
2012-03-06 01:39:47 +08:00
= BestChildren.begin(), E2 = BestChildren.end();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
C != E2; ++C) {
size_t DepthF = getDepthFactor(C->first.first);
Q.push_back(ValuePairWithDepth(C->first, QTop.second+DepthF));
}
Minor changes from review. As suggested by Nick Lewycky, the tree traversal queues have been changed to SmallVectors and the associated loops have been rotated. Also, an 80-col violation was fixed. llvm-svn: 149607
2012-02-03 01:29:39 +08:00
} while (!Q.empty());
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// This function finds the best dag of mututally-compatible connected
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// pairs, given the choice of root pairs as an iterator range.
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
void BBVectorize::findBestDAGFor(
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
DenseSet<ValuePair> &CandidatePairsSet,
DenseMap<ValuePair, int> &CandidatePairCostSavings,
std::vector<Value *> &PairableInsts,
DenseSet<ValuePair> &FixedOrderPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairs,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairDeps,
DenseSet<ValuePair> &PairableInstUsers,
DenseMap<ValuePair, std::vector<ValuePair> > &PairableInstUserMap,
DenseSet<VPPair> &PairableInstUserPairSet,
DenseMap<Value *, Value *> &ChosenPairs,
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
DenseSet<ValuePair> &BestDAG, size_t &BestMaxDepth,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
int &BestEffSize, Value *II, std::vector<Value *>&JJ,
bool UseCycleCheck) {
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
for (std::vector<Value *>::iterator J = JJ.begin(), JE = JJ.end();
J != JE; ++J) {
ValuePair IJ(II, *J);
if (!CandidatePairsSet.count(IJ))
continue;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Before going any further, make sure that this pair does not
// conflict with any already-selected pairs (see comment below
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// near the DAG pruning for more details).
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DenseSet<ValuePair> ChosenPairSet;
bool DoesConflict = false;
for (DenseMap<Value *, Value *>::iterator C = ChosenPairs.begin(),
E = ChosenPairs.end(); C != E; ++C) {
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
if (pairsConflict(*C, IJ, PairableInstUsers,
[C++] Use 'nullptr'. Transforms edition. llvm-svn: 207196
2014-04-25 13:29:35 +08:00
UseCycleCheck ? &PairableInstUserMap : nullptr,
UseCycleCheck ? &PairableInstUserPairSet : nullptr)) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DoesConflict = true;
break;
}
ChosenPairSet.insert(*C);
}
if (DoesConflict) continue;
if (UseCycleCheck &&
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
pairWillFormCycle(IJ, PairableInstUserMap, ChosenPairSet))
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
continue;
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
DenseMap<ValuePair, size_t> DAG;
buildInitialDAGFor(CandidatePairs, CandidatePairsSet,
BBVectorize: Eliminate one more restricted linear search This eliminates one more linear search over a range of std::multimap entries. This gives a 22% speedup on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 174893
2013-02-12 01:19:34 +08:00
PairableInsts, ConnectedPairs,
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
PairableInstUsers, ChosenPairs, DAG, IJ);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Because we'll keep the child with the largest depth, the largest
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// depth is still the same in the unpruned DAG.
size_t MaxDepth = DAG.lookup(IJ);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
DEBUG(if (DebugPairSelection) dbgs() << "BBV: found DAG for pair {"
BBVectorize: Fixup debugging statements After the recent data-structure improvements, a couple of debugging statements were broken (printing pointer values). llvm-svn: 176791
2013-03-11 04:57:42 +08:00
<< *IJ.first << " <-> " << *IJ.second << "} of depth " <<
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
MaxDepth << " and size " << DAG.size() << "\n");
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// At this point the DAG has been constructed, but, may contain
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// contradictory children (meaning that different children of
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// some dag node may be attempting to fuse the same instruction).
// So now we walk the dag again, in the case of a conflict,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// keep only the child with the largest depth. To break a tie,
// favor the first child.
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
DenseSet<ValuePair> PrunedDAG;
pruneDAGFor(CandidatePairs, PairableInsts, ConnectedPairs,
BBVectorize: Eliminate one more restricted linear search This eliminates one more linear search over a range of std::multimap entries. This gives a 22% speedup on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 174893
2013-02-12 01:19:34 +08:00
PairableInstUsers, PairableInstUserMap,
PairableInstUserPairSet,
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
ChosenPairs, DAG, PrunedDAG, IJ, UseCycleCheck);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
int EffSize = 0;
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
if (TTI) {
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
DenseSet<Value *> PrunedDAGInstrs;
for (DenseSet<ValuePair>::iterator S = PrunedDAG.begin(),
E = PrunedDAG.end(); S != E; ++S) {
PrunedDAGInstrs.insert(S->first);
PrunedDAGInstrs.insert(S->second);
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
}
// The set of pairs that have already contributed to the total cost.
DenseSet<ValuePair> IncomingPairs;
BBVectorize: Don't vectorize vector-manipulation chains Don't choose a vectorization plan containing only shuffles and vector inserts/extracts. Due to inperfections in the cost model, these can lead to infinite recusion. llvm-svn: 167811
2012-11-13 11:12:40 +08:00
// If the cost model were perfect, this might not be necessary; but we
// need to make sure that we don't get stuck vectorizing our own
// shuffle chains.
bool HasNontrivialInsts = false;
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
// The node weights represent the cost savings associated with
// fusing the pair of instructions.
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
for (DenseSet<ValuePair>::iterator S = PrunedDAG.begin(),
E = PrunedDAG.end(); S != E; ++S) {
BBVectorize: Don't vectorize vector-manipulation chains Don't choose a vectorization plan containing only shuffles and vector inserts/extracts. Due to inperfections in the cost model, these can lead to infinite recusion. llvm-svn: 167811
2012-11-13 11:12:40 +08:00
if (!isa<ShuffleVectorInst>(S->first) &&
!isa<InsertElementInst>(S->first) &&
!isa<ExtractElementInst>(S->first))
HasNontrivialInsts = true;
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
bool FlipOrder = false;
if (getDepthFactor(S->first)) {
int ESContrib = CandidatePairCostSavings.find(*S)->second;
DEBUG(if (DebugPairSelection) dbgs() << "\tweight {"
<< *S->first << " <-> " << *S->second << "} = " <<
ESContrib << "\n");
EffSize += ESContrib;
}
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
// The edge weights contribute in a negative sense: they represent
// the cost of shuffles.
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<ValuePair, std::vector<ValuePair> >::iterator SS =
ConnectedPairDeps.find(*S);
if (SS != ConnectedPairDeps.end()) {
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
unsigned NumDepsDirect = 0, NumDepsSwap = 0;
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
for (std::vector<ValuePair>::iterator T = SS->second.begin(),
TE = SS->second.end(); T != TE; ++T) {
VPPair Q(*S, *T);
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
if (!PrunedDAG.count(Q.second))
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
continue;
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
DenseMap<VPPair, unsigned>::iterator R =
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
PairConnectionTypes.find(VPPair(Q.second, Q.first));
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
assert(R != PairConnectionTypes.end() &&
"Cannot find pair connection type");
if (R->second == PairConnectionDirect)
++NumDepsDirect;
else if (R->second == PairConnectionSwap)
++NumDepsSwap;
}
// If there are more swaps than direct connections, then
// the pair order will be flipped during fusion. So the real
// number of swaps is the minimum number.
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
FlipOrder = !FixedOrderPairs.count(*S) &&
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
((NumDepsSwap > NumDepsDirect) ||
FixedOrderPairs.count(ValuePair(S->second, S->first)));
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
for (std::vector<ValuePair>::iterator T = SS->second.begin(),
TE = SS->second.end(); T != TE; ++T) {
VPPair Q(*S, *T);
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
if (!PrunedDAG.count(Q.second))
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
continue;
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
DenseMap<VPPair, unsigned>::iterator R =
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
PairConnectionTypes.find(VPPair(Q.second, Q.first));
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
assert(R != PairConnectionTypes.end() &&
"Cannot find pair connection type");
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
Type *Ty1 = Q.second.first->getType(),
*Ty2 = Q.second.second->getType();
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
Type *VTy = getVecTypeForPair(Ty1, Ty2);
if ((R->second == PairConnectionDirect && FlipOrder) ||
(R->second == PairConnectionSwap && !FlipOrder) ||
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
R->second == PairConnectionSplat) {
int ESContrib = (int) getInstrCost(Instruction::ShuffleVector,
VTy, VTy);
BBVectorize: Better use of TTI->getShuffleCost When flipping the pair of subvectors that form a vector, if the vector length is 2, we can use the SK_Reverse shuffle kind to get more-accurate cost information. Also we can use the SK_ExtractSubvector shuffle kind to get accurate subvector extraction costs. The current cost model implementations don't yet seem complex enough for this to make a difference (thus, there are no test cases with this commit), but it should help in future. Depending on how the various targets optimize and combine shuffles in practice, we might be able to get more-accurate costs by combining the costs of multiple shuffle kinds. For example, the cost of flipping the subvector pairs could be modeled as two extractions and two subvector insertions. These changes, however, should probably be motivated by specific test cases. llvm-svn: 173621
2013-01-28 04:07:01 +08:00
if (VTy->getVectorNumElements() == 2) {
if (R->second == PairConnectionSplat)
ESContrib = std::min(ESContrib, (int) TTI->getShuffleCost(
TargetTransformInfo::SK_Broadcast, VTy));
else
ESContrib = std::min(ESContrib, (int) TTI->getShuffleCost(
TargetTransformInfo::SK_Reverse, VTy));
}
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
DEBUG(if (DebugPairSelection) dbgs() << "\tcost {" <<
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
*Q.second.first << " <-> " << *Q.second.second <<
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
"} -> {" <<
*S->first << " <-> " << *S->second << "} = " <<
ESContrib << "\n");
EffSize -= ESContrib;
}
}
}
// Compute the cost of outgoing edges. We assume that edges outgoing
// to shuffles, inserts or extracts can be merged, and so contribute
// no additional cost.
if (!S->first->getType()->isVoidTy()) {
Type *Ty1 = S->first->getType(),
*Ty2 = S->second->getType();
Type *VTy = getVecTypeForPair(Ty1, Ty2);
bool NeedsExtraction = false;
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
for (User *U : S->first->users()) {
if (ShuffleVectorInst *SI = dyn_cast<ShuffleVectorInst>(U)) {
BBVectorize: Use a more sophisticated check for input cost The old checking code, which assumed that input shuffles and insert-elements could always be folded (and thus were free) is too simple. This can only happen in special circumstances. Using the simple check caused infinite recursion. llvm-svn: 167750
2012-11-13 05:21:02 +08:00
// Shuffle can be folded if it has no other input
if (isa<UndefValue>(SI->getOperand(1)))
continue;
}
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
if (isa<ExtractElementInst>(U))
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
continue;
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
if (PrunedDAGInstrs.count(U))
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
continue;
NeedsExtraction = true;
break;
}
if (NeedsExtraction) {
int ESContrib;
BBVectorize: Better use of TTI->getShuffleCost When flipping the pair of subvectors that form a vector, if the vector length is 2, we can use the SK_Reverse shuffle kind to get more-accurate cost information. Also we can use the SK_ExtractSubvector shuffle kind to get accurate subvector extraction costs. The current cost model implementations don't yet seem complex enough for this to make a difference (thus, there are no test cases with this commit), but it should help in future. Depending on how the various targets optimize and combine shuffles in practice, we might be able to get more-accurate costs by combining the costs of multiple shuffle kinds. For example, the cost of flipping the subvector pairs could be modeled as two extractions and two subvector insertions. These changes, however, should probably be motivated by specific test cases. llvm-svn: 173621
2013-01-28 04:07:01 +08:00
if (Ty1->isVectorTy()) {
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
ESContrib = (int) getInstrCost(Instruction::ShuffleVector,
Ty1, VTy);
BBVectorize: Better use of TTI->getShuffleCost When flipping the pair of subvectors that form a vector, if the vector length is 2, we can use the SK_Reverse shuffle kind to get more-accurate cost information. Also we can use the SK_ExtractSubvector shuffle kind to get accurate subvector extraction costs. The current cost model implementations don't yet seem complex enough for this to make a difference (thus, there are no test cases with this commit), but it should help in future. Depending on how the various targets optimize and combine shuffles in practice, we might be able to get more-accurate costs by combining the costs of multiple shuffle kinds. For example, the cost of flipping the subvector pairs could be modeled as two extractions and two subvector insertions. These changes, however, should probably be motivated by specific test cases. llvm-svn: 173621
2013-01-28 04:07:01 +08:00
ESContrib = std::min(ESContrib, (int) TTI->getShuffleCost(
TargetTransformInfo::SK_ExtractSubvector, VTy, 0, Ty1));
} else
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
ESContrib = (int) TTI->getVectorInstrCost(
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
Instruction::ExtractElement, VTy, 0);
DEBUG(if (DebugPairSelection) dbgs() << "\tcost {" <<
*S->first << "} = " << ESContrib << "\n");
EffSize -= ESContrib;
}
NeedsExtraction = false;
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
for (User *U : S->second->users()) {
if (ShuffleVectorInst *SI = dyn_cast<ShuffleVectorInst>(U)) {
BBVectorize: Use a more sophisticated check for input cost The old checking code, which assumed that input shuffles and insert-elements could always be folded (and thus were free) is too simple. This can only happen in special circumstances. Using the simple check caused infinite recursion. llvm-svn: 167750
2012-11-13 05:21:02 +08:00
// Shuffle can be folded if it has no other input
if (isa<UndefValue>(SI->getOperand(1)))
continue;
}
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
if (isa<ExtractElementInst>(U))
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
continue;
[C++11] Add range based accessors for the Use-Def chain of a Value. This requires a number of steps. 1) Move value_use_iterator into the Value class as an implementation detail 2) Change it to actually be a *Use* iterator rather than a *User* iterator. 3) Add an adaptor which is a User iterator that always looks through the Use to the User. 4) Wrap these in Value::use_iterator and Value::user_iterator typedefs. 5) Add the range adaptors as Value::uses() and Value::users(). 6) Update *all* of the callers to correctly distinguish between whether they wanted a use_iterator (and to explicitly dig out the User when needed), or a user_iterator which makes the Use itself totally opaque. Because #6 requires churning essentially everything that walked the Use-Def chains, I went ahead and added all of the range adaptors and switched them to range-based loops where appropriate. Also because the renaming requires at least churning every line of code, it didn't make any sense to split these up into multiple commits -- all of which would touch all of the same lies of code. The result is still not quite optimal. The Value::use_iterator is a nice regular iterator, but Value::user_iterator is an iterator over User*s rather than over the User objects themselves. As a consequence, it fits a bit awkwardly into the range-based world and it has the weird extra-dereferencing 'operator->' that so many of our iterators have. I think this could be fixed by providing something which transforms a range of T&s into a range of T*s, but that *can* be separated into another patch, and it isn't yet 100% clear whether this is the right move. However, this change gets us most of the benefit and cleans up a substantial amount of code around Use and User. =] llvm-svn: 203364
2014-03-09 11:16:01 +08:00
if (PrunedDAGInstrs.count(U))
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
continue;
NeedsExtraction = true;
break;
}
if (NeedsExtraction) {
int ESContrib;
BBVectorize: Better use of TTI->getShuffleCost When flipping the pair of subvectors that form a vector, if the vector length is 2, we can use the SK_Reverse shuffle kind to get more-accurate cost information. Also we can use the SK_ExtractSubvector shuffle kind to get accurate subvector extraction costs. The current cost model implementations don't yet seem complex enough for this to make a difference (thus, there are no test cases with this commit), but it should help in future. Depending on how the various targets optimize and combine shuffles in practice, we might be able to get more-accurate costs by combining the costs of multiple shuffle kinds. For example, the cost of flipping the subvector pairs could be modeled as two extractions and two subvector insertions. These changes, however, should probably be motivated by specific test cases. llvm-svn: 173621
2013-01-28 04:07:01 +08:00
if (Ty2->isVectorTy()) {
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
ESContrib = (int) getInstrCost(Instruction::ShuffleVector,
Ty2, VTy);
BBVectorize: Better use of TTI->getShuffleCost When flipping the pair of subvectors that form a vector, if the vector length is 2, we can use the SK_Reverse shuffle kind to get more-accurate cost information. Also we can use the SK_ExtractSubvector shuffle kind to get accurate subvector extraction costs. The current cost model implementations don't yet seem complex enough for this to make a difference (thus, there are no test cases with this commit), but it should help in future. Depending on how the various targets optimize and combine shuffles in practice, we might be able to get more-accurate costs by combining the costs of multiple shuffle kinds. For example, the cost of flipping the subvector pairs could be modeled as two extractions and two subvector insertions. These changes, however, should probably be motivated by specific test cases. llvm-svn: 173621
2013-01-28 04:07:01 +08:00
ESContrib = std::min(ESContrib, (int) TTI->getShuffleCost(
TargetTransformInfo::SK_ExtractSubvector, VTy,
Ty1->isVectorTy() ? Ty1->getVectorNumElements() : 1, Ty2));
} else
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
ESContrib = (int) TTI->getVectorInstrCost(
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
Instruction::ExtractElement, VTy, 1);
DEBUG(if (DebugPairSelection) dbgs() << "\tcost {" <<
*S->second << "} = " << ESContrib << "\n");
EffSize -= ESContrib;
}
}
// Compute the cost of incoming edges.
if (!isa<LoadInst>(S->first) && !isa<StoreInst>(S->first)) {
Instruction *S1 = cast<Instruction>(S->first),
*S2 = cast<Instruction>(S->second);
for (unsigned o = 0; o < S1->getNumOperands(); ++o) {
Value *O1 = S1->getOperand(o), *O2 = S2->getOperand(o);
// Combining constants into vector constants (or small vector
// constants into larger ones are assumed free).
if (isa<Constant>(O1) && isa<Constant>(O2))
continue;
if (FlipOrder)
std::swap(O1, O2);
ValuePair VP = ValuePair(O1, O2);
ValuePair VPR = ValuePair(O2, O1);
// Internal edges are not handled here.
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
if (PrunedDAG.count(VP) || PrunedDAG.count(VPR))
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
continue;
Type *Ty1 = O1->getType(),
*Ty2 = O2->getType();
Type *VTy = getVecTypeForPair(Ty1, Ty2);
// Combining vector operations of the same type is also assumed
// folded with other operations.
BBVectorize: Use a more sophisticated check for input cost The old checking code, which assumed that input shuffles and insert-elements could always be folded (and thus were free) is too simple. This can only happen in special circumstances. Using the simple check caused infinite recursion. llvm-svn: 167750
2012-11-13 05:21:02 +08:00
if (Ty1 == Ty2) {
// If both are insert elements, then both can be widened.
BBVectorize: Only some insert element operand pairs are free. This fixes another infinite recursion case when using target costs. We can only replace insert element input chains that are pure (end with inserting into an undef). llvm-svn: 167784
2012-11-13 07:55:36 +08:00
InsertElementInst *IEO1 = dyn_cast<InsertElementInst>(O1),
*IEO2 = dyn_cast<InsertElementInst>(O2);
if (IEO1 && IEO2 && isPureIEChain(IEO1) && isPureIEChain(IEO2))
BBVectorize: Use a more sophisticated check for input cost The old checking code, which assumed that input shuffles and insert-elements could always be folded (and thus were free) is too simple. This can only happen in special circumstances. Using the simple check caused infinite recursion. llvm-svn: 167750
2012-11-13 05:21:02 +08:00
continue;
// If both are extract elements, and both have the same input
// type, then they can be replaced with a shuffle
ExtractElementInst *EIO1 = dyn_cast<ExtractElementInst>(O1),
*EIO2 = dyn_cast<ExtractElementInst>(O2);
if (EIO1 && EIO2 &&
EIO1->getOperand(0)->getType() ==
EIO2->getOperand(0)->getType())
continue;
// If both are a shuffle with equal operand types and only two
// unqiue operands, then they can be replaced with a single
// shuffle
ShuffleVectorInst *SIO1 = dyn_cast<ShuffleVectorInst>(O1),
*SIO2 = dyn_cast<ShuffleVectorInst>(O2);
if (SIO1 && SIO2 &&
SIO1->getOperand(0)->getType() ==
SIO2->getOperand(0)->getType()) {
SmallSet<Value *, 4> SIOps;
SIOps.insert(SIO1->getOperand(0));
SIOps.insert(SIO1->getOperand(1));
SIOps.insert(SIO2->getOperand(0));
SIOps.insert(SIO2->getOperand(1));
if (SIOps.size() <= 2)
continue;
}
}
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
int ESContrib;
// This pair has already been formed.
if (IncomingPairs.count(VP)) {
continue;
} else if (IncomingPairs.count(VPR)) {
ESContrib = (int) getInstrCost(Instruction::ShuffleVector,
VTy, VTy);
BBVectorize: Better use of TTI->getShuffleCost When flipping the pair of subvectors that form a vector, if the vector length is 2, we can use the SK_Reverse shuffle kind to get more-accurate cost information. Also we can use the SK_ExtractSubvector shuffle kind to get accurate subvector extraction costs. The current cost model implementations don't yet seem complex enough for this to make a difference (thus, there are no test cases with this commit), but it should help in future. Depending on how the various targets optimize and combine shuffles in practice, we might be able to get more-accurate costs by combining the costs of multiple shuffle kinds. For example, the cost of flipping the subvector pairs could be modeled as two extractions and two subvector insertions. These changes, however, should probably be motivated by specific test cases. llvm-svn: 173621
2013-01-28 04:07:01 +08:00
if (VTy->getVectorNumElements() == 2)
ESContrib = std::min(ESContrib, (int) TTI->getShuffleCost(
TargetTransformInfo::SK_Reverse, VTy));
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
} else if (!Ty1->isVectorTy() && !Ty2->isVectorTy()) {
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
ESContrib = (int) TTI->getVectorInstrCost(
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
Instruction::InsertElement, VTy, 0);
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
ESContrib += (int) TTI->getVectorInstrCost(
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
Instruction::InsertElement, VTy, 1);
} else if (!Ty1->isVectorTy()) {
// O1 needs to be inserted into a vector of size O2, and then
// both need to be shuffled together.
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
ESContrib = (int) TTI->getVectorInstrCost(
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
Instruction::InsertElement, Ty2, 0);
ESContrib += (int) getInstrCost(Instruction::ShuffleVector,
VTy, Ty2);
} else if (!Ty2->isVectorTy()) {
// O2 needs to be inserted into a vector of size O1, and then
// both need to be shuffled together.
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
ESContrib = (int) TTI->getVectorInstrCost(
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
Instruction::InsertElement, Ty1, 0);
ESContrib += (int) getInstrCost(Instruction::ShuffleVector,
VTy, Ty1);
} else {
Type *TyBig = Ty1, *TySmall = Ty2;
if (Ty2->getVectorNumElements() > Ty1->getVectorNumElements())
std::swap(TyBig, TySmall);
ESContrib = (int) getInstrCost(Instruction::ShuffleVector,
VTy, TyBig);
if (TyBig != TySmall)
ESContrib += (int) getInstrCost(Instruction::ShuffleVector,
TyBig, TySmall);
}
DEBUG(if (DebugPairSelection) dbgs() << "\tcost {"
<< *O1 << " <-> " << *O2 << "} = " <<
ESContrib << "\n");
EffSize -= ESContrib;
IncomingPairs.insert(VP);
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
}
}
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
}
BBVectorize: Don't vectorize vector-manipulation chains Don't choose a vectorization plan containing only shuffles and vector inserts/extracts. Due to inperfections in the cost model, these can lead to infinite recusion. llvm-svn: 167811
2012-11-13 11:12:40 +08:00
if (!HasNontrivialInsts) {
DEBUG(if (DebugPairSelection) dbgs() <<
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
"\tNo non-trivial instructions in DAG;"
BBVectorize: Don't vectorize vector-manipulation chains Don't choose a vectorization plan containing only shuffles and vector inserts/extracts. Due to inperfections in the cost model, these can lead to infinite recusion. llvm-svn: 167811
2012-11-13 11:12:40 +08:00
" override to zero effective size\n");
EffSize = 0;
}
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
} else {
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
for (DenseSet<ValuePair>::iterator S = PrunedDAG.begin(),
E = PrunedDAG.end(); S != E; ++S)
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
EffSize += (int) getDepthFactor(S->first);
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DEBUG(if (DebugPairSelection)
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
dbgs() << "BBV: found pruned DAG for pair {"
BBVectorize: Fixup debugging statements After the recent data-structure improvements, a couple of debugging statements were broken (printing pointer values). llvm-svn: 176791
2013-03-11 04:57:42 +08:00
<< *IJ.first << " <-> " << *IJ.second << "} of depth " <<
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
MaxDepth << " and size " << PrunedDAG.size() <<
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
" (effective size: " << EffSize << ")\n");
Switch the BB vectorizer from the VTTI interface to the simple TTI interface. llvm-svn: 171618
2013-01-05 18:05:28 +08:00
if (((TTI && !UseChainDepthWithTI) ||
BBVectorize: Use target costs for incoming and outgoing values instead of the depth heuristic. When target cost information is available, compute explicit costs of inserting and extracting values from vectors. At this point, all costs are estimated using the target information, and the chain-depth heuristic is not needed. As a result, it is now, by default, disabled when using target costs. llvm-svn: 167256
2012-11-02 05:50:12 +08:00
MaxDepth >= Config.ReqChainDepth) &&
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
EffSize > 0 && EffSize > BestEffSize) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BestMaxDepth = MaxDepth;
BestEffSize = EffSize;
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
BestDAG = PrunedDAG;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
}
// Given the list of candidate pairs, this function selects those
// that will be fused into vector instructions.
void BBVectorize::choosePairs(
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > &CandidatePairs,
DenseSet<ValuePair> &CandidatePairsSet,
DenseMap<ValuePair, int> &CandidatePairCostSavings,
std::vector<Value *> &PairableInsts,
DenseSet<ValuePair> &FixedOrderPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairs,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairDeps,
DenseSet<ValuePair> &PairableInstUsers,
DenseMap<Value *, Value *>& ChosenPairs) {
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
bool UseCycleCheck =
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
CandidatePairsSet.size() <= Config.MaxCandPairsForCycleCheck;
DenseMap<Value *, std::vector<Value *> > CandidatePairs2;
for (DenseSet<ValuePair>::iterator I = CandidatePairsSet.begin(),
E = CandidatePairsSet.end(); I != E; ++I) {
std::vector<Value *> &JJ = CandidatePairs2[I->second];
if (JJ.empty()) JJ.reserve(32);
JJ.push_back(I->first);
}
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<ValuePair, std::vector<ValuePair> > PairableInstUserMap;
BBVectorize: Make the bookkeeping to support full cycle checking less expensive By itself, this does not have much of an effect, but only because in the default configuration the full cycle checks are used only for small problem sizes. This is part of a general cleanup of uses of iteration over std::multimap ranges only for the purpose of checking membership. No functionality change intended. llvm-svn: 174856
2013-02-11 13:29:41 +08:00
DenseSet<VPPair> PairableInstUserPairSet;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
for (std::vector<Value *>::iterator I = PairableInsts.begin(),
E = PairableInsts.end(); I != E; ++I) {
// The number of possible pairings for this variable:
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
size_t NumChoices = CandidatePairs.lookup(*I).size();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (!NumChoices) continue;
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
std::vector<Value *> &JJ = CandidatePairs[*I];
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// The best pair to choose and its dag:
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
size_t BestMaxDepth = 0;
int BestEffSize = 0;
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
DenseSet<ValuePair> BestDAG;
findBestDAGFor(CandidatePairs, CandidatePairsSet,
BBVectorize: Eliminate one more restricted linear search This eliminates one more linear search over a range of std::multimap entries. This gives a 22% speedup on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 174893
2013-02-12 01:19:34 +08:00
CandidatePairCostSavings,
BBVectorize: Account for internal shuffle costs When target costs are available, use them to account for the costs of shuffles on internal edges of the DAG of candidate pairs. Because the shuffle costs here are currently for only the internal edges, the current target cost model is trivial, and the chain depth requirement is still in place, I don't yet have an easy test case. Nevertheless, by looking at the debug output, it does seem to do the right think to the effective "size" of each DAG of candidate pairs. llvm-svn: 167217
2012-11-01 14:26:34 +08:00
PairableInsts, FixedOrderPairs, PairConnectionTypes,
ConnectedPairs, ConnectedPairDeps,
BBVectorize: Make the bookkeeping to support full cycle checking less expensive By itself, this does not have much of an effect, but only because in the default configuration the full cycle checks are used only for small problem sizes. This is part of a general cleanup of uses of iteration over std::multimap ranges only for the purpose of checking membership. No functionality change intended. llvm-svn: 174856
2013-02-11 13:29:41 +08:00
PairableInstUsers, PairableInstUserMap,
PairableInstUserPairSet, ChosenPairs,
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
BestDAG, BestMaxDepth, BestEffSize, *I, JJ,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
UseCycleCheck);
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
if (BestDAG.empty())
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
continue;
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// A dag has been chosen (or not) at this point. If no dag was
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// chosen, then this instruction, I, cannot be paired (and is no longer
// considered).
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
DEBUG(dbgs() << "BBV: selected pairs in the best DAG for: "
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
<< *cast<Instruction>(*I) << "\n");
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
for (DenseSet<ValuePair>::iterator S = BestDAG.begin(),
SE2 = BestDAG.end(); S != SE2; ++S) {
// Insert the members of this dag into the list of chosen pairs.
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
ChosenPairs.insert(ValuePair(S->first, S->second));
DEBUG(dbgs() << "BBV: selected pair: " << *S->first << " <-> " <<
*S->second << "\n");
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// Remove all candidate pairs that have values in the chosen dag.
BBVectorize: Fix an invalid reference bug This fixes PR15289. This bug was introduced (recently) in r175215; collecting all std::vector references for candidate pairs to delete at once is invalid because subsequent lookups in the owning DenseMap could invalidate the references. bugpoint was able to reduce a useful test case. Unfortunately, because whether or not this asserts depends on memory layout, this test case will sometimes appear to produce valid output. Nevertheless, running under valgrind will reveal the error. llvm-svn: 175397
2013-02-17 23:59:26 +08:00
std::vector<Value *> &KK = CandidatePairs[S->first];
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
for (std::vector<Value *>::iterator K = KK.begin(), KE = KK.end();
K != KE; ++K) {
if (*K == S->second)
continue;
CandidatePairsSet.erase(ValuePair(S->first, *K));
}
BBVectorize: Fix an invalid reference bug This fixes PR15289. This bug was introduced (recently) in r175215; collecting all std::vector references for candidate pairs to delete at once is invalid because subsequent lookups in the owning DenseMap could invalidate the references. bugpoint was able to reduce a useful test case. Unfortunately, because whether or not this asserts depends on memory layout, this test case will sometimes appear to produce valid output. Nevertheless, running under valgrind will reveal the error. llvm-svn: 175397
2013-02-17 23:59:26 +08:00
std::vector<Value *> &LL = CandidatePairs2[S->second];
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
for (std::vector<Value *>::iterator L = LL.begin(), LE = LL.end();
L != LE; ++L) {
if (*L == S->first)
continue;
CandidatePairsSet.erase(ValuePair(*L, S->second));
}
BBVectorize: Fix an invalid reference bug This fixes PR15289. This bug was introduced (recently) in r175215; collecting all std::vector references for candidate pairs to delete at once is invalid because subsequent lookups in the owning DenseMap could invalidate the references. bugpoint was able to reduce a useful test case. Unfortunately, because whether or not this asserts depends on memory layout, this test case will sometimes appear to produce valid output. Nevertheless, running under valgrind will reveal the error. llvm-svn: 175397
2013-02-17 23:59:26 +08:00
std::vector<Value *> &MM = CandidatePairs[S->second];
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
for (std::vector<Value *>::iterator M = MM.begin(), ME = MM.end();
M != ME; ++M) {
assert(*M != S->first && "Flipped pair in candidate list?");
CandidatePairsSet.erase(ValuePair(S->second, *M));
}
BBVectorize: Fix an invalid reference bug This fixes PR15289. This bug was introduced (recently) in r175215; collecting all std::vector references for candidate pairs to delete at once is invalid because subsequent lookups in the owning DenseMap could invalidate the references. bugpoint was able to reduce a useful test case. Unfortunately, because whether or not this asserts depends on memory layout, this test case will sometimes appear to produce valid output. Nevertheless, running under valgrind will reveal the error. llvm-svn: 175397
2013-02-17 23:59:26 +08:00
std::vector<Value *> &NN = CandidatePairs2[S->first];
BBVectorize: Don't store candidate pairs in a std::multimap This is another commit on the road to removing std::multimap from BBVectorize. This gives an ~1% speedup on the csa.ll test case in PR15222. No functionality change intended. llvm-svn: 175215
2013-02-15 06:37:09 +08:00
for (std::vector<Value *>::iterator N = NN.begin(), NE = NN.end();
N != NE; ++N) {
assert(*N != S->second && "Flipped pair in candidate list?");
CandidatePairsSet.erase(ValuePair(*N, S->first));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
}
DEBUG(dbgs() << "BBV: selected " << ChosenPairs.size() << " pairs.\n");
}
std::string getReplacementName(Instruction *I, bool IsInput, unsigned o,
unsigned n = 0) {
if (!I->hasName())
return "";
return (I->getName() + (IsInput ? ".v.i" : ".v.r") + utostr(o) +
(n > 0 ? "." + utostr(n) : "")).str();
}
// Returns the value that is to be used as the pointer input to the vector
// instruction that fuses I with J.
Value *BBVectorize::getReplacementPointerInput(LLVMContext& Context,
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
Instruction *I, Instruction *J, unsigned o) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Value *IPtr, *JPtr;
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
unsigned IAlignment, JAlignment, IAddressSpace, JAddressSpace;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
int64_t OffsetInElmts;
Precompute SCEV pointer analysis prior to instruction fusion in BBVectorize. When both a load/store and its address computation are being vectorized, it can happen that the address-computation vectorization destroys SCEV's ability to analyize the relative pointer offsets. As a result (like with the aliasing analysis info), we need to precompute the necessary information prior to instruction fusing. This was found during stress testing (running through the test suite with a very low required chain length); unfortunately, I don't have a small test case. llvm-svn: 159332
2012-06-28 13:42:45 +08:00
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
// Note: the analysis might fail here, that is why the pair order has
Precompute SCEV pointer analysis prior to instruction fusion in BBVectorize. When both a load/store and its address computation are being vectorized, it can happen that the address-computation vectorization destroys SCEV's ability to analyize the relative pointer offsets. As a result (like with the aliasing analysis info), we need to precompute the necessary information prior to instruction fusing. This was found during stress testing (running through the test suite with a very low required chain length); unfortunately, I don't have a small test case. llvm-svn: 159332
2012-06-28 13:42:45 +08:00
// been precomputed (OffsetInElmts must be unused here).
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
(void) getPairPtrInfo(I, J, IPtr, JPtr, IAlignment, JAlignment,
Begin incorporating target information into BBVectorize. This is the first of several steps to incorporate information from the new TargetTransformInfo infrastructure into BBVectorize. Two things are done here: 1. Target information is used to determine if it is profitable to fuse two instructions. This means that the cost of the vector operation must not be more expensive than the cost of the two original operations. Pairs that are not profitable are no longer considered (because current cost information is incomplete, for intrinsics for example, equal-cost pairs are still considered). 2. The 'cost savings' computed for the profitability check are also used to rank the DAGs that represent the potential vectorization plans. Specifically, for nodes of non-trivial depth, the cost savings is used as the node weight. The next step will be to incorporate the shuffle costs into the DAG weighting; this will give the edges of the DAG weights as well. Once that is done, when target information is available, we should be able to dispense with the depth heuristic. llvm-svn: 166716
2012-10-26 05:12:23 +08:00
IAddressSpace, JAddressSpace,
BBVectorize: Don't make calls to SE when the result is unused. SE was being called during the instruction-fusion process (when the result is unreliable, and thus ignored). No functionality change is intended. llvm-svn: 167037
2012-10-31 02:55:49 +08:00
OffsetInElmts, false);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// The pointer value is taken to be the one with the lowest offset.
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
Value *VPtr = IPtr;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Use more type helper functions llvm-svn: 193109
2013-10-22 03:43:56 +08:00
Type *ArgTypeI = IPtr->getType()->getPointerElementType();
Type *ArgTypeJ = JPtr->getType()->getPointerElementType();
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *VArgType = getVecTypeForPair(ArgTypeI, ArgTypeJ);
Use more type helper functions llvm-svn: 193109
2013-10-22 03:43:56 +08:00
Type *VArgPtrType
= PointerType::get(VArgType,
IPtr->getType()->getPointerAddressSpace());
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return new BitCastInst(VPtr, VArgPtrType, getReplacementName(I, true, o),
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
/* insert before */ I);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
void BBVectorize::fillNewShuffleMask(LLVMContext& Context, Instruction *J,
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
unsigned MaskOffset, unsigned NumInElem,
unsigned NumInElem1, unsigned IdxOffset,
std::vector<Constant*> &Mask) {
Use more type helper functions llvm-svn: 193109
2013-10-22 03:43:56 +08:00
unsigned NumElem1 = J->getType()->getVectorNumElements();
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
for (unsigned v = 0; v < NumElem1; ++v) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
int m = cast<ShuffleVectorInst>(J)->getMaskValue(v);
if (m < 0) {
Mask[v+MaskOffset] = UndefValue::get(Type::getInt32Ty(Context));
} else {
unsigned mm = m + (int) IdxOffset;
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
if (m >= (int) NumInElem1)
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
mm += (int) NumInElem;
Mask[v+MaskOffset] =
ConstantInt::get(Type::getInt32Ty(Context), mm);
}
}
}
// Returns the value that is to be used as the vector-shuffle mask to the
// vector instruction that fuses I with J.
Value *BBVectorize::getReplacementShuffleMask(LLVMContext& Context,
Instruction *I, Instruction *J) {
// This is the shuffle mask. We need to append the second
// mask to the first, and the numbers need to be adjusted.
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *ArgTypeI = I->getType();
Type *ArgTypeJ = J->getType();
Type *VArgType = getVecTypeForPair(ArgTypeI, ArgTypeJ);
Use more type helper functions llvm-svn: 193109
2013-10-22 03:43:56 +08:00
unsigned NumElemI = ArgTypeI->getVectorNumElements();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Get the total number of elements in the fused vector type.
// By definition, this must equal the number of elements in
// the final mask.
Use more type helper functions llvm-svn: 193109
2013-10-22 03:43:56 +08:00
unsigned NumElem = VArgType->getVectorNumElements();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
std::vector<Constant*> Mask(NumElem);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *OpTypeI = I->getOperand(0)->getType();
Use more type helper functions llvm-svn: 193109
2013-10-22 03:43:56 +08:00
unsigned NumInElemI = OpTypeI->getVectorNumElements();
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *OpTypeJ = J->getOperand(0)->getType();
Use more type helper functions llvm-svn: 193109
2013-10-22 03:43:56 +08:00
unsigned NumInElemJ = OpTypeJ->getVectorNumElements();
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
// The fused vector will be:
// -----------------------------------------------------
// | NumInElemI | NumInElemJ | NumInElemI | NumInElemJ |
// -----------------------------------------------------
// from which we'll extract NumElem total elements (where the first NumElemI
// of them come from the mask in I and the remainder come from the mask
// in J.
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// For the mask from the first pair...
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
fillNewShuffleMask(Context, I, 0, NumInElemJ, NumInElemI,
0, Mask);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// For the mask from the second pair...
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
fillNewShuffleMask(Context, J, NumElemI, NumInElemI, NumInElemJ,
NumInElemI, Mask);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return ConstantVector::get(Mask);
}
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
bool BBVectorize::expandIEChain(LLVMContext& Context, Instruction *I,
Instruction *J, unsigned o, Value *&LOp,
unsigned numElemL,
Type *ArgTypeL, Type *ArgTypeH,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
bool IBeforeJ, unsigned IdxOff) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
bool ExpandedIEChain = false;
if (InsertElementInst *LIE = dyn_cast<InsertElementInst>(LOp)) {
// If we have a pure insertelement chain, then this can be rewritten
// into a chain that directly builds the larger type.
BBVectorize: Only some insert element operand pairs are free. This fixes another infinite recursion case when using target costs. We can only replace insert element input chains that are pure (end with inserting into an undef). llvm-svn: 167784
2012-11-13 07:55:36 +08:00
if (isPureIEChain(LIE)) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
SmallVector<Value *, 8> VectElemts(numElemL,
UndefValue::get(ArgTypeL->getScalarType()));
InsertElementInst *LIENext = LIE;
do {
unsigned Idx =
cast<ConstantInt>(LIENext->getOperand(2))->getSExtValue();
VectElemts[Idx] = LIENext->getOperand(1);
} while ((LIENext =
dyn_cast<InsertElementInst>(LIENext->getOperand(0))));
[C++] Use 'nullptr'. Transforms edition. llvm-svn: 207196
2014-04-25 13:29:35 +08:00
LIENext = nullptr;
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Value *LIEPrev = UndefValue::get(ArgTypeH);
for (unsigned i = 0; i < numElemL; ++i) {
if (isa<UndefValue>(VectElemts[i])) continue;
LIENext = InsertElementInst::Create(LIEPrev, VectElemts[i],
ConstantInt::get(Type::getInt32Ty(Context),
i + IdxOff),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, i+1));
LIENext->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
LIEPrev = LIENext;
}
LOp = LIENext ? (Value*) LIENext : UndefValue::get(ArgTypeH);
ExpandedIEChain = true;
}
}
return ExpandedIEChain;
}
Fix code duplication llvm-svn: 191716
2013-10-01 08:01:14 +08:00
static unsigned getNumScalarElements(Type *Ty) {
if (VectorType *VecTy = dyn_cast<VectorType>(Ty))
return VecTy->getNumElements();
return 1;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Returns the value to be used as the specified operand of the vector
// instruction that fuses I with J.
Value *BBVectorize::getReplacementInput(LLVMContext& Context, Instruction *I,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
Instruction *J, unsigned o, bool IBeforeJ) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Value *CV0 = ConstantInt::get(Type::getInt32Ty(Context), 0);
Value *CV1 = ConstantInt::get(Type::getInt32Ty(Context), 1);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
// Compute the fused vector type for this operand
Type *ArgTypeI = I->getOperand(o)->getType();
Type *ArgTypeJ = J->getOperand(o)->getType();
VectorType *VArgType = getVecTypeForPair(ArgTypeI, ArgTypeJ);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Instruction *L = I, *H = J;
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *ArgTypeL = ArgTypeI, *ArgTypeH = ArgTypeJ;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Fix code duplication llvm-svn: 191716
2013-10-01 08:01:14 +08:00
unsigned numElemL = getNumScalarElements(ArgTypeL);
unsigned numElemH = getNumScalarElements(ArgTypeH);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Value *LOp = L->getOperand(o);
Value *HOp = H->getOperand(o);
unsigned numElem = VArgType->getNumElements();
// First, we check if we can reuse the "original" vector outputs (if these
// exist). We might need a shuffle.
ExtractElementInst *LEE = dyn_cast<ExtractElementInst>(LOp);
ExtractElementInst *HEE = dyn_cast<ExtractElementInst>(HOp);
ShuffleVectorInst *LSV = dyn_cast<ShuffleVectorInst>(LOp);
ShuffleVectorInst *HSV = dyn_cast<ShuffleVectorInst>(HOp);
// FIXME: If we're fusing shuffle instructions, then we can't apply this
// optimization. The input vectors to the shuffle might be a different
// length from the shuffle outputs. Unfortunately, the replacement
// shuffle mask has already been formed, and the mask entries are sensitive
// to the sizes of the inputs.
bool IsSizeChangeShuffle =
isa<ShuffleVectorInst>(L) &&
(LOp->getType() != L->getType() || HOp->getType() != H->getType());
if ((LEE || LSV) && (HEE || HSV) && !IsSizeChangeShuffle) {
// We can have at most two unique vector inputs.
bool CanUseInputs = true;
[C++] Use 'nullptr'. Transforms edition. llvm-svn: 207196
2014-04-25 13:29:35 +08:00
Value *I1, *I2 = nullptr;
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
if (LEE) {
I1 = LEE->getOperand(0);
} else {
I1 = LSV->getOperand(0);
I2 = LSV->getOperand(1);
if (I2 == I1 || isa<UndefValue>(I2))
[C++] Use 'nullptr'. Transforms edition. llvm-svn: 207196
2014-04-25 13:29:35 +08:00
I2 = nullptr;
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
}
Test commit access Remove some trailing whitespaces. llvm-svn: 298379
2017-03-21 22:49:19 +08:00
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
if (HEE) {
Value *I3 = HEE->getOperand(0);
if (!I2 && I3 != I1)
I2 = I3;
else if (I3 != I1 && I3 != I2)
CanUseInputs = false;
} else {
Value *I3 = HSV->getOperand(0);
if (!I2 && I3 != I1)
I2 = I3;
else if (I3 != I1 && I3 != I2)
CanUseInputs = false;
if (CanUseInputs) {
Value *I4 = HSV->getOperand(1);
if (!isa<UndefValue>(I4)) {
if (!I2 && I4 != I1)
I2 = I4;
else if (I4 != I1 && I4 != I2)
CanUseInputs = false;
}
}
}
if (CanUseInputs) {
unsigned LOpElem =
Use more type helper functions llvm-svn: 193109
2013-10-22 03:43:56 +08:00
cast<Instruction>(LOp)->getOperand(0)->getType()
->getVectorNumElements();
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
unsigned HOpElem =
Use more type helper functions llvm-svn: 193109
2013-10-22 03:43:56 +08:00
cast<Instruction>(HOp)->getOperand(0)->getType()
->getVectorNumElements();
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
// We have one or two input vectors. We need to map each index of the
// operands to the index of the original vector.
SmallVector<std::pair<int, int>, 8> II(numElem);
for (unsigned i = 0; i < numElemL; ++i) {
int Idx, INum;
if (LEE) {
Idx =
cast<ConstantInt>(LEE->getOperand(1))->getSExtValue();
INum = LEE->getOperand(0) == I1 ? 0 : 1;
} else {
Idx = LSV->getMaskValue(i);
if (Idx < (int) LOpElem) {
INum = LSV->getOperand(0) == I1 ? 0 : 1;
} else {
Idx -= LOpElem;
INum = LSV->getOperand(1) == I1 ? 0 : 1;
}
}
II[i] = std::pair<int, int>(Idx, INum);
}
for (unsigned i = 0; i < numElemH; ++i) {
int Idx, INum;
if (HEE) {
Idx =
cast<ConstantInt>(HEE->getOperand(1))->getSExtValue();
INum = HEE->getOperand(0) == I1 ? 0 : 1;
} else {
Idx = HSV->getMaskValue(i);
if (Idx < (int) HOpElem) {
INum = HSV->getOperand(0) == I1 ? 0 : 1;
} else {
Idx -= HOpElem;
INum = HSV->getOperand(1) == I1 ? 0 : 1;
}
}
II[i + numElemL] = std::pair<int, int>(Idx, INum);
}
// We now have an array which tells us from which index of which
// input vector each element of the operand comes.
VectorType *I1T = cast<VectorType>(I1->getType());
unsigned I1Elem = I1T->getNumElements();
if (!I2) {
// In this case there is only one underlying vector input. Check for
// the trivial case where we can use the input directly.
if (I1Elem == numElem) {
bool ElemInOrder = true;
for (unsigned i = 0; i < numElem; ++i) {
if (II[i].first != (int) i && II[i].first != -1) {
ElemInOrder = false;
break;
}
}
if (ElemInOrder)
return I1;
}
// A shuffle is needed.
std::vector<Constant *> Mask(numElem);
for (unsigned i = 0; i < numElem; ++i) {
int Idx = II[i].first;
if (Idx == -1)
Mask[i] = UndefValue::get(Type::getInt32Ty(Context));
else
Mask[i] = ConstantInt::get(Type::getInt32Ty(Context), Idx);
}
Instruction *S =
new ShuffleVectorInst(I1, UndefValue::get(I1T),
ConstantVector::get(Mask),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o));
S->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
return S;
}
VectorType *I2T = cast<VectorType>(I2->getType());
unsigned I2Elem = I2T->getNumElements();
// This input comes from two distinct vectors. The first step is to
// make sure that both vectors are the same length. If not, the
// smaller one will need to grow before they can be shuffled together.
if (I1Elem < I2Elem) {
std::vector<Constant *> Mask(I2Elem);
unsigned v = 0;
for (; v < I1Elem; ++v)
Mask[v] = ConstantInt::get(Type::getInt32Ty(Context), v);
for (; v < I2Elem; ++v)
Mask[v] = UndefValue::get(Type::getInt32Ty(Context));
Instruction *NewI1 =
new ShuffleVectorInst(I1, UndefValue::get(I1T),
ConstantVector::get(Mask),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 1));
NewI1->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
I1 = NewI1;
I1Elem = I2Elem;
} else if (I1Elem > I2Elem) {
std::vector<Constant *> Mask(I1Elem);
unsigned v = 0;
for (; v < I2Elem; ++v)
Mask[v] = ConstantInt::get(Type::getInt32Ty(Context), v);
for (; v < I1Elem; ++v)
Mask[v] = UndefValue::get(Type::getInt32Ty(Context));
Instruction *NewI2 =
new ShuffleVectorInst(I2, UndefValue::get(I2T),
ConstantVector::get(Mask),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 1));
NewI2->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
I2 = NewI2;
}
// Now that both I1 and I2 are the same length we can shuffle them
// together (and use the result).
std::vector<Constant *> Mask(numElem);
for (unsigned v = 0; v < numElem; ++v) {
if (II[v].first == -1) {
Mask[v] = UndefValue::get(Type::getInt32Ty(Context));
} else {
int Idx = II[v].first + II[v].second * I1Elem;
Mask[v] = ConstantInt::get(Type::getInt32Ty(Context), Idx);
}
}
Instruction *NewOp =
new ShuffleVectorInst(I1, I2, ConstantVector::get(Mask),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J, true, o));
NewOp->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
return NewOp;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *ArgType = ArgTypeL;
if (numElemL < numElemH) {
if (numElemL == 1 && expandIEChain(Context, I, J, o, HOp, numElemH,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
ArgTypeL, VArgType, IBeforeJ, 1)) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
// This is another short-circuit case: we're combining a scalar into
// a vector that is formed by an IE chain. We've just expanded the IE
// chain, now insert the scalar and we're done.
Instruction *S = InsertElementInst::Create(HOp, LOp, CV0,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J, true, o));
S->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
return S;
} else if (!expandIEChain(Context, I, J, o, LOp, numElemL, ArgTypeL,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
ArgTypeH, IBeforeJ)) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
// The two vector inputs to the shuffle must be the same length,
// so extend the smaller vector to be the same length as the larger one.
Instruction *NLOp;
if (numElemL > 1) {
Test commit access Remove some trailing whitespaces. llvm-svn: 298379
2017-03-21 22:49:19 +08:00
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
std::vector<Constant *> Mask(numElemH);
unsigned v = 0;
for (; v < numElemL; ++v)
Mask[v] = ConstantInt::get(Type::getInt32Ty(Context), v);
for (; v < numElemH; ++v)
Mask[v] = UndefValue::get(Type::getInt32Ty(Context));
Test commit access Remove some trailing whitespaces. llvm-svn: 298379
2017-03-21 22:49:19 +08:00
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
NLOp = new ShuffleVectorInst(LOp, UndefValue::get(ArgTypeL),
ConstantVector::get(Mask),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 1));
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
} else {
NLOp = InsertElementInst::Create(UndefValue::get(ArgTypeH), LOp, CV0,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 1));
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
}
Test commit access Remove some trailing whitespaces. llvm-svn: 298379
2017-03-21 22:49:19 +08:00
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
NLOp->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
LOp = NLOp;
}
ArgType = ArgTypeH;
} else if (numElemL > numElemH) {
if (numElemH == 1 && expandIEChain(Context, I, J, o, LOp, numElemL,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
ArgTypeH, VArgType, IBeforeJ)) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Instruction *S =
Test commit access Remove some trailing whitespaces. llvm-svn: 298379
2017-03-21 22:49:19 +08:00
InsertElementInst::Create(LOp, HOp,
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
ConstantInt::get(Type::getInt32Ty(Context),
numElemL),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o));
S->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
return S;
} else if (!expandIEChain(Context, I, J, o, HOp, numElemH, ArgTypeH,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
ArgTypeL, IBeforeJ)) {
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Instruction *NHOp;
if (numElemH > 1) {
std::vector<Constant *> Mask(numElemL);
unsigned v = 0;
for (; v < numElemH; ++v)
Mask[v] = ConstantInt::get(Type::getInt32Ty(Context), v);
for (; v < numElemL; ++v)
Mask[v] = UndefValue::get(Type::getInt32Ty(Context));
Test commit access Remove some trailing whitespaces. llvm-svn: 298379
2017-03-21 22:49:19 +08:00
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
NHOp = new ShuffleVectorInst(HOp, UndefValue::get(ArgTypeH),
ConstantVector::get(Mask),
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 1));
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
} else {
NHOp = InsertElementInst::Create(UndefValue::get(ArgTypeL), HOp, CV0,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 1));
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
}
Use more type helper functions llvm-svn: 193109
2013-10-22 03:43:56 +08:00
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
NHOp->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
HOp = NHOp;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
if (ArgType->isVectorTy()) {
Use more type helper functions llvm-svn: 193109
2013-10-22 03:43:56 +08:00
unsigned numElem = VArgType->getVectorNumElements();
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
std::vector<Constant*> Mask(numElem);
for (unsigned v = 0; v < numElem; ++v) {
unsigned Idx = v;
// If the low vector was expanded, we need to skip the extra
// undefined entries.
if (v >= numElemL && numElemH > numElemL)
Idx += (numElemH - numElemL);
Mask[v] = ConstantInt::get(Type::getInt32Ty(Context), Idx);
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Instruction *BV = new ShuffleVectorInst(LOp, HOp,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
ConstantVector::get(Mask),
getReplacementName(IBeforeJ ? I : J, true, o));
BV->insertBefore(IBeforeJ ? J : I);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return BV;
}
Instruction *BV1 = InsertElementInst::Create(
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
UndefValue::get(VArgType), LOp, CV0,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 1));
BV1->insertBefore(IBeforeJ ? J : I);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Instruction *BV2 = InsertElementInst::Create(BV1, HOp, CV1,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementName(IBeforeJ ? I : J,
true, o, 2));
BV2->insertBefore(IBeforeJ ? J : I);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
return BV2;
}
// This function creates an array of values that will be used as the inputs
// to the vector instruction that fuses I with J.
void BBVectorize::getReplacementInputsForPair(LLVMContext& Context,
Instruction *I, Instruction *J,
Use SmallVectorImpl& instead of SmallVector to avoid repeating small vector size. llvm-svn: 186274
2013-07-14 12:42:23 +08:00
SmallVectorImpl<Value *> &ReplacedOperands,
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
bool IBeforeJ) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
unsigned NumOperands = I->getNumOperands();
for (unsigned p = 0, o = NumOperands-1; p < NumOperands; ++p, --o) {
// Iterate backward so that we look at the store pointer
// first and know whether or not we need to flip the inputs.
if (isa<LoadInst>(I) || (o == 1 && isa<StoreInst>(I))) {
// This is the pointer for a load/store instruction.
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
ReplacedOperands[o] = getReplacementPointerInput(Context, I, J, o);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
continue;
Correctly vectorize powi. The powi intrinsic requires special handling because it always takes a single integer power regardless of the result type. As a result, we can vectorize only if the powers are equal. Fixes PR12364. llvm-svn: 153797
2012-03-31 11:38:40 +08:00
} else if (isa<CallInst>(I)) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Function *F = cast<CallInst>(I)->getCalledFunction();
Change Function::getIntrinsicID() to return an Intrinsic::ID. NFC. Now that Intrinsic::ID is a typed enum, we can forward declare it and so return it from this method. This updates all users which were either using an unsigned to store it, or had a now unnecessary cast. llvm-svn: 237810
2015-05-21 01:16:39 +08:00
Intrinsic::ID IID = F->getIntrinsicID();
Correctly vectorize powi. The powi intrinsic requires special handling because it always takes a single integer power regardless of the result type. As a result, we can vectorize only if the powers are equal. Fixes PR12364. llvm-svn: 153797
2012-03-31 11:38:40 +08:00
if (o == NumOperands-1) {
BasicBlock &BB = *I->getParent();
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
Correctly vectorize powi. The powi intrinsic requires special handling because it always takes a single integer power regardless of the result type. As a result, we can vectorize only if the powers are equal. Fixes PR12364. llvm-svn: 153797
2012-03-31 11:38:40 +08:00
Module *M = BB.getParent()->getParent();
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Type *ArgTypeI = I->getType();
Type *ArgTypeJ = J->getType();
Type *VArgType = getVecTypeForPair(ArgTypeI, ArgTypeJ);
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
BBVectorize: Use VTTI to compute costs for intrinsics vectorization For the time being this includes only some dummy test cases. Once the generic implementation of the intrinsics cost function does something other than assuming scalarization in all cases, or some target specializes the interface, some real test cases can be added. Also, for consistency, I changed the type of IID from unsigned to Intrinsic::ID in a few other places. llvm-svn: 171079
2012-12-26 09:36:57 +08:00
ReplacedOperands[o] = Intrinsic::getDeclaration(M, IID, VArgType);
Correctly vectorize powi. The powi intrinsic requires special handling because it always takes a single integer power regardless of the result type. As a result, we can vectorize only if the powers are equal. Fixes PR12364. llvm-svn: 153797
2012-03-31 11:38:40 +08:00
continue;
Allow vectorization of bit intrinsics in BB Vectorizer. This patch adds support for vectorization of bit intrinsics such as bswap,ctpop,ctlz,cttz. llvm-svn: 207174
2014-04-25 11:33:48 +08:00
} else if ((IID == Intrinsic::powi || IID == Intrinsic::ctlz ||
IID == Intrinsic::cttz) && o == 1) {
// The second argument of powi/ctlz/cttz is a single integer/constant
// and we've already checked that both arguments are equal.
// As a result, we just keep I's second argument.
Correctly vectorize powi. The powi intrinsic requires special handling because it always takes a single integer power regardless of the result type. As a result, we can vectorize only if the powers are equal. Fixes PR12364. llvm-svn: 153797
2012-03-31 11:38:40 +08:00
ReplacedOperands[o] = I->getOperand(o);
continue;
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
} else if (isa<ShuffleVectorInst>(I) && o == NumOperands-1) {
ReplacedOperands[o] = getReplacementShuffleMask(Context, I, J);
continue;
}
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
ReplacedOperands[o] = getReplacementInput(Context, I, J, o, IBeforeJ);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
// This function creates two values that represent the outputs of the
// original I and J instructions. These are generally vector shuffles
// or extracts. In many cases, these will end up being unused and, thus,
// eliminated by later passes.
void BBVectorize::replaceOutputsOfPair(LLVMContext& Context, Instruction *I,
Instruction *J, Instruction *K,
Instruction *&InsertionPt,
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
Instruction *&K1, Instruction *&K2) {
[PM/AA] Remove the last of the legacy update API from AliasAnalysis as part of simplifying its interface and usage in preparation for porting to work with the new pass manager. Note that this will likely expose that we have dead arguments, members, and maybe even pass requirements for AA. I'll be cleaning those up in seperate patches. This just zaps the actual update API. Differential Revision: http://reviews.llvm.org/D11325 llvm-svn: 242881
2015-07-22 17:49:59 +08:00
if (isa<StoreInst>(I))
[PM/AA] Switch to an early-exit. NFC. This was split out of another change because the diff is *useless*. I assure you, I just switched to early-return in this function. Cleanup in preparation for my next commit, as requested in code review! llvm-svn: 242880
2015-07-22 17:44:54 +08:00
return;
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
[PM/AA] Switch to an early-exit. NFC. This was split out of another change because the diff is *useless*. I assure you, I just switched to early-return in this function. Cleanup in preparation for my next commit, as requested in code review! llvm-svn: 242880
2015-07-22 17:44:54 +08:00
Type *IType = I->getType();
Type *JType = J->getType();
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
[PM/AA] Switch to an early-exit. NFC. This was split out of another change because the diff is *useless*. I assure you, I just switched to early-return in this function. Cleanup in preparation for my next commit, as requested in code review! llvm-svn: 242880
2015-07-22 17:44:54 +08:00
VectorType *VType = getVecTypeForPair(IType, JType);
unsigned numElem = VType->getNumElements();
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
[PM/AA] Switch to an early-exit. NFC. This was split out of another change because the diff is *useless*. I assure you, I just switched to early-return in this function. Cleanup in preparation for my next commit, as requested in code review! llvm-svn: 242880
2015-07-22 17:44:54 +08:00
unsigned numElemI = getNumScalarElements(IType);
unsigned numElemJ = getNumScalarElements(JType);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
[PM/AA] Switch to an early-exit. NFC. This was split out of another change because the diff is *useless*. I assure you, I just switched to early-return in this function. Cleanup in preparation for my next commit, as requested in code review! llvm-svn: 242880
2015-07-22 17:44:54 +08:00
if (IType->isVectorTy()) {
std::vector<Constant *> Mask1(numElemI), Mask2(numElemI);
for (unsigned v = 0; v < numElemI; ++v) {
Mask1[v] = ConstantInt::get(Type::getInt32Ty(Context), v);
Mask2[v] = ConstantInt::get(Type::getInt32Ty(Context), numElemJ + v);
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
}
[PM/AA] Switch to an early-exit. NFC. This was split out of another change because the diff is *useless*. I assure you, I just switched to early-return in this function. Cleanup in preparation for my next commit, as requested in code review! llvm-svn: 242880
2015-07-22 17:44:54 +08:00
K1 = new ShuffleVectorInst(K, UndefValue::get(VType),
ConstantVector::get(Mask1),
getReplacementName(K, false, 1));
} else {
Value *CV0 = ConstantInt::get(Type::getInt32Ty(Context), 0);
K1 = ExtractElementInst::Create(K, CV0, getReplacementName(K, false, 1));
}
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
[PM/AA] Switch to an early-exit. NFC. This was split out of another change because the diff is *useless*. I assure you, I just switched to early-return in this function. Cleanup in preparation for my next commit, as requested in code review! llvm-svn: 242880
2015-07-22 17:44:54 +08:00
if (JType->isVectorTy()) {
std::vector<Constant *> Mask1(numElemJ), Mask2(numElemJ);
for (unsigned v = 0; v < numElemJ; ++v) {
Mask1[v] = ConstantInt::get(Type::getInt32Ty(Context), v);
Mask2[v] = ConstantInt::get(Type::getInt32Ty(Context), numElemI + v);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
[PM/AA] Switch to an early-exit. NFC. This was split out of another change because the diff is *useless*. I assure you, I just switched to early-return in this function. Cleanup in preparation for my next commit, as requested in code review! llvm-svn: 242880
2015-07-22 17:44:54 +08:00
K2 = new ShuffleVectorInst(K, UndefValue::get(VType),
ConstantVector::get(Mask2),
getReplacementName(K, false, 2));
} else {
Value *CV1 = ConstantInt::get(Type::getInt32Ty(Context), numElem - 1);
K2 = ExtractElementInst::Create(K, CV1, getReplacementName(K, false, 2));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
[PM/AA] Switch to an early-exit. NFC. This was split out of another change because the diff is *useless*. I assure you, I just switched to early-return in this function. Cleanup in preparation for my next commit, as requested in code review! llvm-svn: 242880
2015-07-22 17:44:54 +08:00
K1->insertAfter(K);
K2->insertAfter(K1);
InsertionPt = K2;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
// Move all uses of the function I (including pairing-induced uses) after J.
bool BBVectorize::canMoveUsesOfIAfterJ(BasicBlock &BB,
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
DenseSet<ValuePair> &LoadMoveSetPairs,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Instruction *I, Instruction *J) {
// Skip to the first instruction past I.
[C++11] Replace llvm::next and llvm::prior with std::next and std::prev. Remove the old functions. llvm-svn: 202636
2014-03-02 20:27:27 +08:00
BasicBlock::iterator L = std::next(BasicBlock::iterator(I));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DenseSet<Value *> Users;
AliasSetTracker WriteSet(*AA);
BBVectorize: Add initial stores to the write set when tracking uses When computing the use set of a store, we need to add the store to the write set prior to iterating over later instructions. Otherwise, if there is a later aliasing load of that store, that load will not be tagged as a use, and bad things will happen. trackUsesOfI still adds later dependent stores of an instruction to that instruction's write set, but it never sees the original instruction, and so when tracking uses of a store, the store must be added to the write set by the caller. Fixes PR16834. llvm-svn: 188329
2013-08-14 07:34:32 +08:00
if (I->mayWriteToMemory()) WriteSet.add(I);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
for (; cast<Instruction>(L) != J; ++L)
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
(void)trackUsesOfI(Users, WriteSet, I, &*L, true, &LoadMoveSetPairs);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
assert(cast<Instruction>(L) == J &&
"Tracking has not proceeded far enough to check for dependencies");
// If J is now in the use set of I, then trackUsesOfI will return true
// and we have a dependency cycle (and the fusing operation must abort).
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
return !trackUsesOfI(Users, WriteSet, I, J, true, &LoadMoveSetPairs);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
// Move all uses of the function I (including pairing-induced uses) after J.
void BBVectorize::moveUsesOfIAfterJ(BasicBlock &BB,
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
DenseSet<ValuePair> &LoadMoveSetPairs,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Instruction *&InsertionPt,
Instruction *I, Instruction *J) {
// Skip to the first instruction past I.
[C++11] Replace llvm::next and llvm::prior with std::next and std::prev. Remove the old functions. llvm-svn: 202636
2014-03-02 20:27:27 +08:00
BasicBlock::iterator L = std::next(BasicBlock::iterator(I));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DenseSet<Value *> Users;
AliasSetTracker WriteSet(*AA);
BBVectorize: Add initial stores to the write set when tracking uses When computing the use set of a store, we need to add the store to the write set prior to iterating over later instructions. Otherwise, if there is a later aliasing load of that store, that load will not be tagged as a use, and bad things will happen. trackUsesOfI still adds later dependent stores of an instruction to that instruction's write set, but it never sees the original instruction, and so when tracking uses of a store, the store must be added to the write set by the caller. Fixes PR16834. llvm-svn: 188329
2013-08-14 07:34:32 +08:00
if (I->mayWriteToMemory()) WriteSet.add(I);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
for (; cast<Instruction>(L) != J;) {
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
if (trackUsesOfI(Users, WriteSet, I, &*L, true, &LoadMoveSetPairs)) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Move this instruction
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
Instruction *InstToMove = &*L++;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DEBUG(dbgs() << "BBV: moving: " << *InstToMove <<
" to after " << *InsertionPt << "\n");
InstToMove->removeFromParent();
InstToMove->insertAfter(InsertionPt);
InsertionPt = InstToMove;
} else {
++L;
}
}
}
// Collect all load instruction that are in the move set of a given first
// pair member. These loads depend on the first instruction, I, and so need
// to be moved after J (the second instruction) when the pair is fused.
void BBVectorize::collectPairLoadMoveSet(BasicBlock &BB,
DenseMap<Value *, Value *> &ChosenPairs,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > &LoadMoveSet,
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
DenseSet<ValuePair> &LoadMoveSetPairs,
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Instruction *I) {
// Skip to the first instruction past I.
[C++11] Replace llvm::next and llvm::prior with std::next and std::prev. Remove the old functions. llvm-svn: 202636
2014-03-02 20:27:27 +08:00
BasicBlock::iterator L = std::next(BasicBlock::iterator(I));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DenseSet<Value *> Users;
AliasSetTracker WriteSet(*AA);
BBVectorize: Add initial stores to the write set when tracking uses When computing the use set of a store, we need to add the store to the write set prior to iterating over later instructions. Otherwise, if there is a later aliasing load of that store, that load will not be tagged as a use, and bad things will happen. trackUsesOfI still adds later dependent stores of an instruction to that instruction's write set, but it never sees the original instruction, and so when tracking uses of a store, the store must be added to the write set by the caller. Fixes PR16834. llvm-svn: 188329
2013-08-14 07:34:32 +08:00
if (I->mayWriteToMemory()) WriteSet.add(I);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Note: We cannot end the loop when we reach J because J could be moved
// farther down the use chain by another instruction pairing. Also, J
// could be before I if this is an inverted input.
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
for (BasicBlock::iterator E = BB.end(); L != E; ++L) {
if (trackUsesOfI(Users, WriteSet, I, &*L)) {
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
if (L->mayReadFromMemory()) {
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
LoadMoveSet[&*L].push_back(I);
LoadMoveSetPairs.insert(ValuePair(&*L, I));
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
}
// In cases where both load/stores and the computation of their pointers
// are chosen for vectorization, we can end up in a situation where the
// aliasing analysis starts returning different query results as the
// process of fusing instruction pairs continues. Because the algorithm
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// relies on finding the same use dags here as were found earlier, we'll
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// need to precompute the necessary aliasing information here and then
// manually update it during the fusion process.
void BBVectorize::collectLoadMoveSet(BasicBlock &BB,
std::vector<Value *> &PairableInsts,
DenseMap<Value *, Value *> &ChosenPairs,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > &LoadMoveSet,
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
DenseSet<ValuePair> &LoadMoveSetPairs) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
for (std::vector<Value *>::iterator PI = PairableInsts.begin(),
PIE = PairableInsts.end(); PI != PIE; ++PI) {
DenseMap<Value *, Value *>::iterator P = ChosenPairs.find(*PI);
if (P == ChosenPairs.end()) continue;
Instruction *I = cast<Instruction>(P->first);
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
collectPairLoadMoveSet(BB, ChosenPairs, LoadMoveSet,
LoadMoveSetPairs, I);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
}
// This function fuses the chosen instruction pairs into vector instructions,
// taking care preserve any needed scalar outputs and, then, it reorders the
// remaining instructions as needed (users of the first member of the pair
// need to be moved to after the location of the second member of the pair
// because the vector instruction is inserted in the location of the pair's
// second member).
void BBVectorize::fuseChosenPairs(BasicBlock &BB,
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
std::vector<Value *> &PairableInsts,
DenseMap<Value *, Value *> &ChosenPairs,
DenseSet<ValuePair> &FixedOrderPairs,
DenseMap<VPPair, unsigned> &PairConnectionTypes,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairs,
DenseMap<ValuePair, std::vector<ValuePair> > &ConnectedPairDeps) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
LLVMContext& Context = BB.getContext();
// During the vectorization process, the order of the pairs to be fused
// could be flipped. So we'll add each pair, flipped, into the ChosenPairs
// list. After a pair is fused, the flipped pair is removed from the list.
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
DenseSet<ValuePair> FlippedPairs;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
for (DenseMap<Value *, Value *>::iterator P = ChosenPairs.begin(),
E = ChosenPairs.end(); P != E; ++P)
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
FlippedPairs.insert(ValuePair(P->second, P->first));
for (DenseSet<ValuePair>::iterator P = FlippedPairs.begin(),
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
E = FlippedPairs.end(); P != E; ++P)
ChosenPairs.insert(*P);
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> > LoadMoveSet;
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
DenseSet<ValuePair> LoadMoveSetPairs;
collectLoadMoveSet(BB, PairableInsts, ChosenPairs,
LoadMoveSet, LoadMoveSetPairs);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DEBUG(dbgs() << "BBV: initial: \n" << BB << "\n");
for (BasicBlock::iterator PI = BB.getFirstInsertionPt(); PI != BB.end();) {
Vectorize: Remove implicit ilist iterator conversions, NFC Besides the usual, I finally added an overload to `BasicBlock::splitBasicBlock()` that accepts an `Instruction*` instead of `BasicBlock::iterator`. Someone can go back and remove this overload later (after updating the callers I'm going to skip going forward), but the most common call seems to be `BB->splitBasicBlock(BB->getTerminator(), ...)` and I'm not sure it's better to add `->getIterator()` to every one than have the overload. It's pretty hard to get the usage wrong. llvm-svn: 250745
2015-10-20 06:06:09 +08:00
DenseMap<Value *, Value *>::iterator P = ChosenPairs.find(&*PI);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (P == ChosenPairs.end()) {
++PI;
continue;
}
if (getDepthFactor(P->first) == 0) {
// These instructions are not really fused, but are tracked as though
// they are. Any case in which it would be interesting to fuse them
// will be taken care of by InstCombine.
--NumFusedOps;
++PI;
continue;
}
Instruction *I = cast<Instruction>(P->first),
*J = cast<Instruction>(P->second);
DEBUG(dbgs() << "BBV: fusing: " << *I <<
" <-> " << *J << "\n");
// Remove the pair and flipped pair from the list.
DenseMap<Value *, Value *>::iterator FP = ChosenPairs.find(P->second);
assert(FP != ChosenPairs.end() && "Flipped pair not found in list");
ChosenPairs.erase(FP);
ChosenPairs.erase(P);
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
if (!canMoveUsesOfIAfterJ(BB, LoadMoveSetPairs, I, J)) {
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
DEBUG(dbgs() << "BBV: fusion of: " << *I <<
" <-> " << *J <<
" aborted because of non-trivial dependency cycle\n");
--NumFusedOps;
++PI;
continue;
}
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
// If the pair must have the other order, then flip it.
bool FlipPairOrder = FixedOrderPairs.count(ValuePair(J, I));
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
if (!FlipPairOrder && !FixedOrderPairs.count(ValuePair(I, J))) {
// This pair does not have a fixed order, and so we might want to
// flip it if that will yield fewer shuffles. We count the number
// of dependencies connected via swaps, and those directly connected,
// and flip the order if the number of swaps is greater.
bool OrigOrder = true;
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<ValuePair, std::vector<ValuePair> >::iterator IJ =
ConnectedPairDeps.find(ValuePair(I, J));
if (IJ == ConnectedPairDeps.end()) {
IJ = ConnectedPairDeps.find(ValuePair(J, I));
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
OrigOrder = false;
}
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
if (IJ != ConnectedPairDeps.end()) {
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
unsigned NumDepsDirect = 0, NumDepsSwap = 0;
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
for (std::vector<ValuePair>::iterator T = IJ->second.begin(),
TE = IJ->second.end(); T != TE; ++T) {
VPPair Q(IJ->first, *T);
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
DenseMap<VPPair, unsigned>::iterator R =
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
PairConnectionTypes.find(VPPair(Q.second, Q.first));
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
assert(R != PairConnectionTypes.end() &&
"Cannot find pair connection type");
if (R->second == PairConnectionDirect)
++NumDepsDirect;
else if (R->second == PairConnectionSwap)
++NumDepsSwap;
}
if (!OrigOrder)
std::swap(NumDepsDirect, NumDepsSwap);
if (NumDepsSwap > NumDepsDirect) {
FlipPairOrder = true;
DEBUG(dbgs() << "BBV: reordering pair: " << *I <<
" <-> " << *J << "\n");
}
}
}
Precompute SCEV pointer analysis prior to instruction fusion in BBVectorize. When both a load/store and its address computation are being vectorized, it can happen that the address-computation vectorization destroys SCEV's ability to analyize the relative pointer offsets. As a result (like with the aliasing analysis info), we need to precompute the necessary information prior to instruction fusing. This was found during stress testing (running through the test suite with a very low required chain length); unfortunately, I don't have a small test case. llvm-svn: 159332
2012-06-28 13:42:45 +08:00
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
Instruction *L = I, *H = J;
BBVectorize: Cache fixed-order pairs instead of recomputing pointer info. Instead of recomputing relative pointer information just prior to fusing, cache this information (which also needs to be computed during the candidate-pair selection process). This cuts down on the total number of SE queries made, and also is a necessary intermediate step on the road toward including shuffle costs in the pair selection procedure. No functionality change is intended. llvm-svn: 167049
2012-10-31 04:17:37 +08:00
if (FlipPairOrder)
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
std::swap(H, L);
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
// If the pair being fused uses the opposite order from that in the pair
// connection map, then we need to flip the types.
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<ValuePair, std::vector<ValuePair> >::iterator HL =
ConnectedPairs.find(ValuePair(H, L));
if (HL != ConnectedPairs.end())
for (std::vector<ValuePair>::iterator T = HL->second.begin(),
TE = HL->second.end(); T != TE; ++T) {
VPPair Q(HL->first, *T);
DenseMap<VPPair, unsigned>::iterator R = PairConnectionTypes.find(Q);
assert(R != PairConnectionTypes.end() &&
"Cannot find pair connection type");
if (R->second == PairConnectionDirect)
R->second = PairConnectionSwap;
else if (R->second == PairConnectionSwap)
R->second = PairConnectionDirect;
}
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
bool LBeforeH = !FlipPairOrder;
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
unsigned NumOperands = I->getNumOperands();
SmallVector<Value *, 3> ReplacedOperands(NumOperands);
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
getReplacementInputsForPair(Context, L, H, ReplacedOperands,
LBeforeH);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// Make a copy of the original operation, change its type to the vector
// type and replace its operands with the vector operands.
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
Instruction *K = L->clone();
if (L->hasName())
K->takeName(L);
else if (H->hasName())
K->takeName(H);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
Recommit r235458: [opaque pointer type] Avoid using PointerType::getElementType for a few cases of CallInst (reverted in r235533) Original commit message: "Calls to llvm::Value::mutateType are becoming extra-sensitive now that instructions have extra type information that will not be derived from operands or result type (alloca, gep, load, call/invoke, etc... ). The special-handling for mutateType will get more complicated as this work continues - it might be worth making mutateType virtual & pushing the complexity down into the classes that need special handling. But with only two significant uses of mutateType (vectorization and linking) this seems OK for now. Totally open to ideas/suggestions/improvements, of course. With this, and a bunch of exceptions, we can roundtrip an indirect call site through bitcode and IR. (a direct call site is actually trickier... I haven't figured out how to deal with the IR deserializer's lazy construction of Function/GlobalVariable decl's based on the type of the entity which means looking through the "pointer to T" type referring to the global)" The remapping done in ValueMapper for LTO was insufficient as the types weren't correctly mapped (though I was using the post-mapped operands, some of those operands might not have been mapped yet so the type wouldn't be post-mapped yet). Instead use the pre-mapped type and explicitly map all the types. llvm-svn: 235651
2015-04-24 05:36:23 +08:00
if (auto CS = CallSite(K)) {
SmallVector<Type *, 3> Tys;
FunctionType *Old = CS.getFunctionType();
unsigned NumOld = Old->getNumParams();
assert(NumOld <= ReplacedOperands.size());
for (unsigned i = 0; i != NumOld; ++i)
Tys.push_back(ReplacedOperands[i]->getType());
CS.mutateFunctionType(
FunctionType::get(getVecTypeForPair(L->getType(), H->getType()),
Tys, Old->isVarArg()));
} else if (!isa<StoreInst>(K))
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
K->mutateType(getVecTypeForPair(L->getType(), H->getType()));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
inariant.group handling in GVN The most important part required to make clang devirtualization works ( ͡°͜ʖ ͡°). The code is able to find non local dependencies, but unfortunatelly because the caller can only handle local dependencies, I had to add some restrictions to look for dependencies only in the same BB. http://reviews.llvm.org/D12992 llvm-svn: 249196
2015-10-03 06:12:22 +08:00
unsigned KnownIDs[] = {LLVMContext::MD_tbaa, LLVMContext::MD_alias_scope,
LLVMContext::MD_noalias, LLVMContext::MD_fpmath,
LLVMContext::MD_invariant_group};
Introduce a helper to combine instruction metadata. Replace the old code in GVN and BBVectorize with it. Update SimplifyCFG to use it. Patch by Björn Steinbrink! llvm-svn: 215723
2014-08-15 23:46:38 +08:00
combineMetadata(K, H, KnownIDs);
IR: Remove Value::intersectOptionalDataWith, replace all calls with calls to Instruction::andIRFlags. The two functions are functionally equivalent. Differential Revision: https://reviews.llvm.org/D22830 llvm-svn: 280884
2016-09-08 07:39:04 +08:00
K->andIRFlags(H);
Teach BBVectorize to combine, when possible, or discard metadata when fusing instructions. The present implementation handles only TBAA and FP metadata, discarding everything else. For debug metadata, the current behavior is maintained (the debug metadata associated with one of the instructions will be kept, discarding that attached to the other). This should address PR 13040. llvm-svn: 158606
2012-06-17 04:34:06 +08:00
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
for (unsigned o = 0; o < NumOperands; ++o)
K->setOperand(o, ReplacedOperands[o]);
K->insertAfter(J);
// Instruction insertion point:
Instruction *InsertionPt = K;
[C++] Use 'nullptr'. Transforms edition. llvm-svn: 207196
2014-04-25 13:29:35 +08:00
Instruction *K1 = nullptr, *K2 = nullptr;
BBVectorize: Simplify how input swapping is handled. Stop propagating the FlipMemInputs variable into the routines that create the replacement instructions. Instead, just flip the arguments of those routines. This allows for some associated cleanup (not all of which is done here). No functionality change is intended. llvm-svn: 167042
2012-10-31 03:35:29 +08:00
replaceOutputsOfPair(Context, L, H, K, InsertionPt, K1, K2);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
BBVectorize: Call a DAG and DAG instead of a tree Several functions and variable names used the term 'tree' to refer to what is actually a DAG. Correcting this mistake will, hopefully, prevent confusion in the future. No functionality change intended. llvm-svn: 175278
2013-02-16 01:20:54 +08:00
// The use dag of the first original instruction must be moved to after
// the location of the second instruction. The entire use dag of the
// first instruction is disjoint from the input dag of the second
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
// (by definition), and so commutes with it.
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
moveUsesOfIAfterJ(BB, LoadMoveSetPairs, InsertionPt, I, J);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (!isa<StoreInst>(I)) {
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
L->replaceAllUsesWith(K1);
H->replaceAllUsesWith(K2);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
// Instructions that may read from memory may be in the load move set.
// Once an instruction is fused, we no longer need its move set, and so
// the values of the map never need to be updated. However, when a load
// is fused, we need to merge the entries from both instructions in the
// pair in case those instructions were in the move set of some other
// yet-to-be-fused pair. The loads in question are the keys of the map.
if (I->mayReadFromMemory()) {
std::vector<ValuePair> NewSetMembers;
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
DenseMap<Value *, std::vector<Value *> >::iterator II =
LoadMoveSet.find(I);
if (II != LoadMoveSet.end())
for (std::vector<Value *>::iterator N = II->second.begin(),
NE = II->second.end(); N != NE; ++N)
NewSetMembers.push_back(ValuePair(K, *N));
DenseMap<Value *, std::vector<Value *> >::iterator JJ =
LoadMoveSet.find(J);
if (JJ != LoadMoveSet.end())
for (std::vector<Value *>::iterator N = JJ->second.begin(),
NE = JJ->second.end(); N != NE; ++N)
NewSetMembers.push_back(ValuePair(K, *N));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
for (std::vector<ValuePair>::iterator A = NewSetMembers.begin(),
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
AE = NewSetMembers.end(); A != AE; ++A) {
BBVectorize: Remove the remaining instances of std::multimap All instances of std::multimap have now been replaced by DenseMap<K, std::vector<V> >, and this yields a speedup of 5% on the csa.ll test case from PR15222. No functionality change intended. llvm-svn: 175216
2013-02-15 06:38:04 +08:00
LoadMoveSet[A->first].push_back(A->second);
BBVectorize: Avoid linear searches within the load-move set This is another cleanup aimed at eliminating linear searches in ranges of std::multimap. No functionality change intended. llvm-svn: 174858
2013-02-11 13:29:49 +08:00
LoadMoveSetPairs.insert(*A);
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
// Before removing I, set the iterator to the next instruction.
[C++11] Replace llvm::next and llvm::prior with std::next and std::prev. Remove the old functions. llvm-svn: 202636
2014-03-02 20:27:27 +08:00
PI = std::next(BasicBlock::iterator(I));
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
if (cast<Instruction>(PI) == J)
++PI;
SE->forgetValue(I);
SE->forgetValue(J);
I->eraseFromParent();
J->eraseFromParent();
BBVectorize: Choose pair ordering to minimize shuffles BBVectorize would, except for loads and stores, always fuse instructions so that the first instruction (in the current source order) would always represent the low part of the input vectors and the second instruction would always represent the high part. This lead to too many shuffles being produced because sometimes the opposite order produces fewer of them. With this change, BBVectorize tracks the kind of pair connections that form the DAG of candidate pairs, and uses that information to reorder the pairs to avoid excess shuffles. Using this information, a future commit will be able to add VTTI-based shuffle costs to the pair selection procedure. Importantly, the number of remaining shuffles can now be estimated during pair selection. There are some trivial instruction reorderings in the test cases, and one simple additional test where we certainly want to do a reordering to avoid an unnecessary shuffle. llvm-svn: 167122
2012-10-31 23:17:07 +08:00
DEBUG(if (PrintAfterEveryPair) dbgs() << "BBV: block is now: \n" <<
BB << "\n");
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
DEBUG(dbgs() << "BBV: final: \n" << BB << "\n");
}
Revert r240137 (Fixed/added namespace ending comments using clang-tidy. NFC) Apparently, the style needs to be agreed upon first. llvm-svn: 240390
2015-06-23 17:49:53 +08:00
}
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
char BBVectorize::ID = 0;
static const char bb_vectorize_name[] = "Basic-Block Vectorization";
INITIALIZE_PASS_BEGIN(BBVectorize, BBV_NAME, bb_vectorize_name, false, false)
[PM/AA] Rebuild LLVM's alias analysis infrastructure in a way compatible with the new pass manager, and no longer relying on analysis groups. This builds essentially a ground-up new AA infrastructure stack for LLVM. The core ideas are the same that are used throughout the new pass manager: type erased polymorphism and direct composition. The design is as follows: - FunctionAAResults is a type-erasing alias analysis results aggregation interface to walk a single query across a range of results from different alias analyses. Currently this is function-specific as we always assume that aliasing queries are *within* a function. - AAResultBase is a CRTP utility providing stub implementations of various parts of the alias analysis result concept, notably in several cases in terms of other more general parts of the interface. This can be used to implement only a narrow part of the interface rather than the entire interface. This isn't really ideal, this logic should be hoisted into FunctionAAResults as currently it will cause a significant amount of redundant work, but it faithfully models the behavior of the prior infrastructure. - All the alias analysis passes are ported to be wrapper passes for the legacy PM and new-style analysis passes for the new PM with a shared result object. In some cases (most notably CFL), this is an extremely naive approach that we should revisit when we can specialize for the new pass manager. - BasicAA has been restructured to reflect that it is much more fundamentally a function analysis because it uses dominator trees and loop info that need to be constructed for each function. All of the references to getting alias analysis results have been updated to use the new aggregation interface. All the preservation and other pass management code has been updated accordingly. The way the FunctionAAResultsWrapperPass works is to detect the available alias analyses when run, and add them to the results object. This means that we should be able to continue to respect when various passes are added to the pipeline, for example adding CFL or adding TBAA passes should just cause their results to be available and to get folded into this. The exception to this rule is BasicAA which really needs to be a function pass due to using dominator trees and loop info. As a consequence, the FunctionAAResultsWrapperPass directly depends on BasicAA and always includes it in the aggregation. This has significant implications for preserving analyses. Generally, most passes shouldn't bother preserving FunctionAAResultsWrapperPass because rebuilding the results just updates the set of known AA passes. The exception to this rule are LoopPass instances which need to preserve all the function analyses that the loop pass manager will end up needing. This means preserving both BasicAAWrapperPass and the aggregating FunctionAAResultsWrapperPass. Now, when preserving an alias analysis, you do so by directly preserving that analysis. This is only necessary for non-immutable-pass-provided alias analyses though, and there are only three of interest: BasicAA, GlobalsAA (formerly GlobalsModRef), and SCEVAA. Usually BasicAA is preserved when needed because it (like DominatorTree and LoopInfo) is marked as a CFG-only pass. I've expanded GlobalsAA into the preserved set everywhere we previously were preserving all of AliasAnalysis, and I've added SCEVAA in the intersection of that with where we preserve SCEV itself. One significant challenge to all of this is that the CGSCC passes were actually using the alias analysis implementations by taking advantage of a pretty amazing set of loop holes in the old pass manager's analysis management code which allowed analysis groups to slide through in many cases. Moving away from analysis groups makes this problem much more obvious. To fix it, I've leveraged the flexibility the design of the new PM components provides to just directly construct the relevant alias analyses for the relevant functions in the IPO passes that need them. This is a bit hacky, but should go away with the new pass manager, and is already in many ways cleaner than the prior state. Another significant challenge is that various facilities of the old alias analysis infrastructure just don't fit any more. The most significant of these is the alias analysis 'counter' pass. That pass relied on the ability to snoop on AA queries at different points in the analysis group chain. Instead, I'm planning to build printing functionality directly into the aggregation layer. I've not included that in this patch merely to keep it smaller. Note that all of this needs a nearly complete rewrite of the AA documentation. I'm planning to do that, but I'd like to make sure the new design settles, and to flesh out a bit more of what it looks like in the new pass manager first. Differential Revision: http://reviews.llvm.org/D12080 llvm-svn: 247167
2015-09-10 01:55:00 +08:00
INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
INITIALIZE_PASS_DEPENDENCY(AAResultsWrapperPass)
[PM/AA] Explicitly depend on TLI rather than getting it out of the AliasAnalysis. Same as the other commits, the TLI access from an alias analysis is going away and isn't very clean -- it is better to explicitly mark the dependencies. llvm-svn: 244785
2015-08-13 02:06:08 +08:00
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
[PM] Change the core design of the TTI analysis to use a polymorphic type erased interface and a single analysis pass rather than an extremely complex analysis group. The end result is that the TTI analysis can contain a type erased implementation that supports the polymorphic TTI interface. We can build one from a target-specific implementation or from a dummy one in the IR. I've also factored all of the code into "mix-in"-able base classes, including CRTP base classes to facilitate calling back up to the most specialized form when delegating horizontally across the surface. These aren't as clean as I would like and I'm planning to work on cleaning some of this up, but I wanted to start by putting into the right form. There are a number of reasons for this change, and this particular design. The first and foremost reason is that an analysis group is complete overkill, and the chaining delegation strategy was so opaque, confusing, and high overhead that TTI was suffering greatly for it. Several of the TTI functions had failed to be implemented in all places because of the chaining-based delegation making there be no checking of this. A few other functions were implemented with incorrect delegation. The message to me was very clear working on this -- the delegation and analysis group structure was too confusing to be useful here. The other reason of course is that this is *much* more natural fit for the new pass manager. This will lay the ground work for a type-erased per-function info object that can look up the correct subtarget and even cache it. Yet another benefit is that this will significantly simplify the interaction of the pass managers and the TargetMachine. See the future work below. The downside of this change is that it is very, very verbose. I'm going to work to improve that, but it is somewhat an implementation necessity in C++ to do type erasure. =/ I discussed this design really extensively with Eric and Hal prior to going down this path, and afterward showed them the result. No one was really thrilled with it, but there doesn't seem to be a substantially better alternative. Using a base class and virtual method dispatch would make the code much shorter, but as discussed in the update to the programmer's manual and elsewhere, a polymorphic interface feels like the more principled approach even if this is perhaps the least compelling example of it. ;] Ultimately, there is still a lot more to be done here, but this was the huge chunk that I couldn't really split things out of because this was the interface change to TTI. I've tried to minimize all the other parts of this. The follow up work should include at least: 1) Improving the TargetMachine interface by having it directly return a TTI object. Because we have a non-pass object with value semantics and an internal type erasure mechanism, we can narrow the interface of the TargetMachine to *just* do what we need: build and return a TTI object that we can then insert into the pass pipeline. 2) Make the TTI object be fully specialized for a particular function. This will include splitting off a minimal form of it which is sufficient for the inliner and the old pass manager. 3) Add a new pass manager analysis which produces TTI objects from the target machine for each function. This may actually be done as part of #2 in order to use the new analysis to implement #2. 4) Work on narrowing the API between TTI and the targets so that it is easier to understand and less verbose to type erase. 5) Work on narrowing the API between TTI and its clients so that it is easier to understand and less verbose to forward. 6) Try to improve the CRTP-based delegation. I feel like this code is just a bit messy and exacerbating the complexity of implementing the TTI in each target. Many thanks to Eric and Hal for their help here. I ended up blocked on this somewhat more abruptly than I expected, and so I appreciate getting it sorted out very quickly. Differential Revision: http://reviews.llvm.org/D7293 llvm-svn: 227669
2015-01-31 11:43:40 +08:00
INITIALIZE_PASS_DEPENDENCY(TargetTransformInfoWrapperPass)
[PM] Split DominatorTree into a concrete analysis result object which can be used by both the new pass manager and the old. This removes it from any of the virtual mess of the pass interfaces and lets it derive cleanly from the DominatorTreeBase<> template. In turn, tons of boilerplate interface can be nuked and it turns into a very straightforward extension of the base DominatorTree interface. The old analysis pass is now a simple wrapper. The names and style of this split should match the split between CallGraph and CallGraphWrapperPass. All of the users of DominatorTree have been updated to match using many of the same tricks as with CallGraph. The goal is that the common type remains the resulting DominatorTree rather than the pass. This will make subsequent work toward the new pass manager significantly easier. Also in numerous places things became cleaner because I switched from re-running the pass (!!! mid way through some other passes run!!!) to directly recomputing the domtree. llvm-svn: 199104
2014-01-13 21:07:17 +08:00
INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
[PM] Port ScalarEvolution to the new pass manager. This change makes ScalarEvolution a stand-alone object and just produces one from a pass as needed. Making this work well requires making the object movable, using references instead of overwritten pointers in a number of places, and other refactorings. I've also wired it up to the new pass manager and added a RUN line to a test to exercise it under the new pass manager. This includes basic printing support much like with other analyses. But there is a big and somewhat scary change here. Prior to this patch ScalarEvolution was never *actually* invalidated!!! Re-running the pass just re-wired up the various other analyses and didn't remove any of the existing entries in the SCEV caches or clear out anything at all. This might seem OK as everything in SCEV that can uses ValueHandles to track updates to the values that serve as SCEV keys. However, this still means that as we ran SCEV over each function in the module, we kept accumulating more and more SCEVs into the cache. At the end, we would have a SCEV cache with every value that we ever needed a SCEV for in the entire module!!! Yowzers. The releaseMemory routine would dump all of this, but that isn't realy called during normal runs of the pipeline as far as I can see. To make matters worse, there *is* actually a key that we don't update with value handles -- there is a map keyed off of Loop*s. Because LoopInfo *does* release its memory from run to run, it is entirely possible to run SCEV over one function, then over another function, and then lookup a Loop* from the second function but find an entry inserted for the first function! Ouch. To make matters still worse, there are plenty of updates that *don't* trip a value handle. It seems incredibly unlikely that today GVN or another pass that invalidates SCEV can update values in *just* such a way that a subsequent run of SCEV will incorrectly find lookups in a cache, but it is theoretically possible and would be a nightmare to debug. With this refactoring, I've fixed all this by actually destroying and recreating the ScalarEvolution object from run to run. Technically, this could increase the amount of malloc traffic we see, but then again it is also technically correct. ;] I don't actually think we're suffering from tons of malloc traffic from SCEV because if we were, the fact that we never clear the memory would seem more likely to have come up as an actual problem before now. So, I've made the simple fix here. If in fact there are serious issues with too much allocation and deallocation, I can work on a clever fix that preserves the allocations (while clearing the data) between each run, but I'd prefer to do that kind of optimization with a test case / benchmark that shows why we need such cleverness (and that can test that we actually make it faster). It's possible that this will make some things faster by making the SCEV caches have higher locality (due to being significantly smaller) so until there is a clear benchmark, I think the simple change is best. Differential Revision: http://reviews.llvm.org/D12063 llvm-svn: 245193
2015-08-17 10:08:17 +08:00
INITIALIZE_PASS_DEPENDENCY(ScalarEvolutionWrapperPass)
[PM/AA] Rebuild LLVM's alias analysis infrastructure in a way compatible with the new pass manager, and no longer relying on analysis groups. This builds essentially a ground-up new AA infrastructure stack for LLVM. The core ideas are the same that are used throughout the new pass manager: type erased polymorphism and direct composition. The design is as follows: - FunctionAAResults is a type-erasing alias analysis results aggregation interface to walk a single query across a range of results from different alias analyses. Currently this is function-specific as we always assume that aliasing queries are *within* a function. - AAResultBase is a CRTP utility providing stub implementations of various parts of the alias analysis result concept, notably in several cases in terms of other more general parts of the interface. This can be used to implement only a narrow part of the interface rather than the entire interface. This isn't really ideal, this logic should be hoisted into FunctionAAResults as currently it will cause a significant amount of redundant work, but it faithfully models the behavior of the prior infrastructure. - All the alias analysis passes are ported to be wrapper passes for the legacy PM and new-style analysis passes for the new PM with a shared result object. In some cases (most notably CFL), this is an extremely naive approach that we should revisit when we can specialize for the new pass manager. - BasicAA has been restructured to reflect that it is much more fundamentally a function analysis because it uses dominator trees and loop info that need to be constructed for each function. All of the references to getting alias analysis results have been updated to use the new aggregation interface. All the preservation and other pass management code has been updated accordingly. The way the FunctionAAResultsWrapperPass works is to detect the available alias analyses when run, and add them to the results object. This means that we should be able to continue to respect when various passes are added to the pipeline, for example adding CFL or adding TBAA passes should just cause their results to be available and to get folded into this. The exception to this rule is BasicAA which really needs to be a function pass due to using dominator trees and loop info. As a consequence, the FunctionAAResultsWrapperPass directly depends on BasicAA and always includes it in the aggregation. This has significant implications for preserving analyses. Generally, most passes shouldn't bother preserving FunctionAAResultsWrapperPass because rebuilding the results just updates the set of known AA passes. The exception to this rule are LoopPass instances which need to preserve all the function analyses that the loop pass manager will end up needing. This means preserving both BasicAAWrapperPass and the aggregating FunctionAAResultsWrapperPass. Now, when preserving an alias analysis, you do so by directly preserving that analysis. This is only necessary for non-immutable-pass-provided alias analyses though, and there are only three of interest: BasicAA, GlobalsAA (formerly GlobalsModRef), and SCEVAA. Usually BasicAA is preserved when needed because it (like DominatorTree and LoopInfo) is marked as a CFG-only pass. I've expanded GlobalsAA into the preserved set everywhere we previously were preserving all of AliasAnalysis, and I've added SCEVAA in the intersection of that with where we preserve SCEV itself. One significant challenge to all of this is that the CGSCC passes were actually using the alias analysis implementations by taking advantage of a pretty amazing set of loop holes in the old pass manager's analysis management code which allowed analysis groups to slide through in many cases. Moving away from analysis groups makes this problem much more obvious. To fix it, I've leveraged the flexibility the design of the new PM components provides to just directly construct the relevant alias analyses for the relevant functions in the IPO passes that need them. This is a bit hacky, but should go away with the new pass manager, and is already in many ways cleaner than the prior state. Another significant challenge is that various facilities of the old alias analysis infrastructure just don't fit any more. The most significant of these is the alias analysis 'counter' pass. That pass relied on the ability to snoop on AA queries at different points in the analysis group chain. Instead, I'm planning to build printing functionality directly into the aggregation layer. I've not included that in this patch merely to keep it smaller. Note that all of this needs a nearly complete rewrite of the AA documentation. I'm planning to do that, but I'd like to make sure the new design settles, and to flesh out a bit more of what it looks like in the new pass manager first. Differential Revision: http://reviews.llvm.org/D12080 llvm-svn: 247167
2015-09-10 01:55:00 +08:00
INITIALIZE_PASS_DEPENDENCY(GlobalsAAWrapperPass)
INITIALIZE_PASS_DEPENDENCY(SCEVAAWrapperPass)
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
INITIALIZE_PASS_END(BBVectorize, BBV_NAME, bb_vectorize_name, false, false)
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
BasicBlockPass *llvm::createBBVectorizePass(const VectorizeConfig &C) {
return new BBVectorize(C);
Add a basic-block autovectorization pass. This is the initial checkin of the basic-block autovectorization pass along with some supporting vectorization infrastructure. Special thanks to everyone who helped review this code over the last several months (especially Tobias Grosser). llvm-svn: 149468
2012-02-01 11:51:43 +08:00
}
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
bool
llvm::vectorizeBasicBlock(Pass *P, BasicBlock &BB, const VectorizeConfig &C) {
[multiversion] Thread a function argument through all the callers of the getTTI method used to get an actual TTI object. No functionality changed. This just threads the argument and ensures code like the inliner can correctly look up the callee's TTI rather than using a fixed one. The next change will use this to implement per-function subtarget usage by TTI. The changes after that should eliminate the need for FTTI as that will have become the default. llvm-svn: 227730
2015-02-01 20:01:35 +08:00
BBVectorize BBVectorizer(P, *BB.getParent(), C);
Add the function "vectorizeBasicBlock" which allow users vectorize a BasicBlock in other passes, e.g. we can call vectorizeBasicBlock in the loop unroll pass right after the loop is unrolled. llvm-svn: 154089
2012-04-05 16:05:16 +08:00
return BBVectorizer.vectorizeBB(BB);
}
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
//===----------------------------------------------------------------------===//
VectorizeConfig::VectorizeConfig() {
VectorBits = ::VectorBits;
Allow controlling vectorization of boolean values separately from other integer types. These are used as the result of comparisons, and often handled differently from larger integer types. llvm-svn: 159111
2012-06-24 21:28:01 +08:00
VectorizeBools = !::NoBools;
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
VectorizeInts = !::NoInts;
VectorizeFloats = !::NoFloats;
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
VectorizePointers = !::NoPointers;
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
VectorizeCasts = !::NoCasts;
VectorizeMath = !::NoMath;
Allow vectorization of bit intrinsics in BB Vectorizer. This patch adds support for vectorization of bit intrinsics such as bswap,ctpop,ctlz,cttz. llvm-svn: 207174
2014-04-25 11:33:48 +08:00
VectorizeBitManipulations = !::NoBitManipulation;
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
VectorizeFMA = !::NoFMA;
Add support to BBVectorize for vectorizing selects. llvm-svn: 154700
2012-04-14 04:45:45 +08:00
VectorizeSelect = !::NoSelect;
Allow BBVectorize to fuse compare instructions. llvm-svn: 159088
2012-06-24 05:52:50 +08:00
VectorizeCmp = !::NoCmp;
Enhance BBVectorize to more-properly handle pointer values and vectorize GEPs. llvm-svn: 154734
2012-04-14 15:32:43 +08:00
VectorizeGEP = !::NoGEP;
Refactor: Use positive field names in VectorizeConfig. llvm-svn: 154249
2012-04-07 11:56:23 +08:00
VectorizeMemOps = !::NoMemOps;
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
AlignedOnly = ::AlignedOnly;
ReqChainDepth= ::ReqChainDepth;
SearchLimit = ::SearchLimit;
MaxCandPairsForCycleCheck = ::MaxCandPairsForCycleCheck;
SplatBreaksChain = ::SplatBreaksChain;
MaxInsts = ::MaxInsts;
BBVectorize: Cap the number of candidate pairs in each instruction group For some basic blocks, it is possible to generate many candidate pairs for relatively few pairable instructions. When many (tens of thousands) of these pairs are generated for a single instruction group, the time taken to generate and rank the different vectorization plans can become quite large. As a result, we now cap the number of candidate pairs within each instruction group. This is done by closing out the group once the threshold is reached (set now at 3000 pairs). Although this will limit the overall compile-time impact, this may not be the best way to achieve this result. It might be better, for example, to prune excessive candidate pairs after the fact the prevent the generation of short, but highly-connected groups. We can experiment with this in the future. This change reduces the overall compile-time slowdown of the csa.ll test case in PR15222 to ~5x. If 5x is still considered too large, a lower limit can be used as the default. This represents a functionality change, but only for very large inputs (thus, there is no regression test). llvm-svn: 175251
2013-02-15 12:28:42 +08:00
MaxPairs = ::MaxPairs;
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
MaxIter = ::MaxIter;
Allow BBVectorize to form non-2^n-length vectors. The original algorithm only used recursive pair fusion of equal-length types. This is now extended to allow pairing of any types that share the same underlying scalar type. Because we would still generally prefer the 2^n-length types, those are formed first. Then a second set of iterations form the non-2^n-length types. Also, a call to SimplifyInstructionsInBlock has been added after each pairing iteration. This takes care of DCE (and a few other things) that make the following iterations execute somewhat faster. For the same reason, some of the simple shuffle-combination cases are now handled internally. There is some additional refactoring work to be done, but I've had many requests for this feature, so additional refactoring will come soon in future commits (as will additional test cases). llvm-svn: 159330
2012-06-28 13:42:42 +08:00
Pow2LenOnly = ::Pow2LenOnly;
Introduce the VectorizeConfig class, with which we can control the behavior of the BBVectorizePass without using command line option. As pointed out by Hal, we can ask the TargetLoweringInfo for the architecture specific VectorizeConfig to perform vectorizing with architecture specific information. llvm-svn: 154096
2012-04-05 23:46:55 +08:00
NoMemOpBoost = ::NoMemOpBoost;
FastDep = ::FastDep;
}