2015-07-11 02:55:13 +08:00
|
|
|
//===- LoopVersioning.cpp - Utility to version a loop ---------------------===//
|
|
|
|
//
|
2019-01-19 16:50:56 +08:00
|
|
|
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
|
|
|
|
// See https://llvm.org/LICENSE.txt for license information.
|
|
|
|
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
2015-07-11 02:55:13 +08:00
|
|
|
//
|
|
|
|
//===----------------------------------------------------------------------===//
|
|
|
|
//
|
|
|
|
// This file defines a utility class to perform loop versioning. The versioned
|
|
|
|
// loop speculates that otherwise may-aliasing memory accesses don't overlap and
|
|
|
|
// emits checks to prove this.
|
|
|
|
//
|
|
|
|
//===----------------------------------------------------------------------===//
|
|
|
|
|
2015-10-27 02:40:56 +08:00
|
|
|
#include "llvm/Transforms/Utils/LoopVersioning.h"
|
2020-05-01 03:15:30 +08:00
|
|
|
#include "llvm/ADT/ArrayRef.h"
|
2020-08-04 02:50:07 +08:00
|
|
|
#include "llvm/Analysis/AliasAnalysis.h"
|
2015-07-11 02:55:13 +08:00
|
|
|
#include "llvm/Analysis/LoopAccessAnalysis.h"
|
|
|
|
#include "llvm/Analysis/LoopInfo.h"
|
2020-08-04 02:50:07 +08:00
|
|
|
#include "llvm/Analysis/MemorySSA.h"
|
2020-08-01 08:30:30 +08:00
|
|
|
#include "llvm/Analysis/ScalarEvolution.h"
|
2020-08-04 02:50:07 +08:00
|
|
|
#include "llvm/Analysis/TargetLibraryInfo.h"
|
2015-07-11 02:55:13 +08:00
|
|
|
#include "llvm/IR/Dominators.h"
|
[LoopVersioning] Annotate versioned loop with noalias metadata
Summary:
If we decide to version a loop to benefit a transformation, it makes
sense to record the now non-aliasing accesses in the newly versioned
loop. This allows non-aliasing information to be used by subsequent
passes.
One example is 456.hmmer in SPECint2006 where after loop distribution,
we vectorize one of the newly distributed loops. To vectorize we
version this loop to fully disambiguate may-aliasing accesses. If we
add the noalias markers, we can use the same information in a later DSE
pass to eliminate some dead stores which amounts to ~25% of the
instructions of this hot memory-pipeline-bound loop. The overall
performance improves by 18% on our ARM64.
The scoped noalias annotation is added in LoopVersioning. The patch
then enables this for loop distribution. A follow-on patch will enable
it for the vectorizer. Eventually this should be run by default when
versioning the loop but first I'd like to get some feedback whether my
understanding and application of scoped noalias metadata is correct.
Essentially my approach was to have a separate alias domain for each
versioning of the loop. For example, if we first version in loop
distribution and then in vectorization of the distributed loops, we have
a different set of memchecks for each versioning. By keeping the scopes
in different domains they can conveniently be defined independently
since different alias domains don't affect each other.
As written, I also have a separate domain for each loop. This is not
necessary and we could save some metadata here by using the same domain
across the different loops. I don't think it's a big deal either way.
Probably the best is to review the tests first to see if I mapped this
problem correctly to scoped noalias markers. I have plenty of comments
in the tests.
Note that the interface is prepared for the vectorizer which needs the
annotateInstWithNoAlias API. The vectorizer does not use LoopVersioning
so we need a way to pass in the versioned instructions. This is also
why the maps have to become part of the object state.
Also currently, we only have an AA-aware DSE after the vectorizer if we
also run the LTO pipeline. Depending how widely this triggers we may
want to schedule a DSE toward the end of the regular pass pipeline.
Reviewers: hfinkel, nadav, ashutosh.nema
Subscribers: mssimpso, aemerson, llvm-commits, mcrosier
Differential Revision: http://reviews.llvm.org/D16712
llvm-svn: 263743
2016-03-18 04:32:32 +08:00
|
|
|
#include "llvm/IR/MDBuilder.h"
|
2020-08-01 08:30:30 +08:00
|
|
|
#include "llvm/IR/PassManager.h"
|
Sink all InitializePasses.h includes
This file lists every pass in LLVM, and is included by Pass.h, which is
very popular. Every time we add, remove, or rename a pass in LLVM, it
caused lots of recompilation.
I found this fact by looking at this table, which is sorted by the
number of times a file was changed over the last 100,000 git commits
multiplied by the number of object files that depend on it in the
current checkout:
recompiles touches affected_files header
342380 95 3604 llvm/include/llvm/ADT/STLExtras.h
314730 234 1345 llvm/include/llvm/InitializePasses.h
307036 118 2602 llvm/include/llvm/ADT/APInt.h
213049 59 3611 llvm/include/llvm/Support/MathExtras.h
170422 47 3626 llvm/include/llvm/Support/Compiler.h
162225 45 3605 llvm/include/llvm/ADT/Optional.h
158319 63 2513 llvm/include/llvm/ADT/Triple.h
140322 39 3598 llvm/include/llvm/ADT/StringRef.h
137647 59 2333 llvm/include/llvm/Support/Error.h
131619 73 1803 llvm/include/llvm/Support/FileSystem.h
Before this change, touching InitializePasses.h would cause 1345 files
to recompile. After this change, touching it only causes 550 compiles in
an incremental rebuild.
Reviewers: bkramer, asbirlea, bollu, jdoerfert
Differential Revision: https://reviews.llvm.org/D70211
2019-11-14 05:15:01 +08:00
|
|
|
#include "llvm/InitializePasses.h"
|
2019-11-15 07:15:48 +08:00
|
|
|
#include "llvm/Support/CommandLine.h"
|
2015-07-11 02:55:13 +08:00
|
|
|
#include "llvm/Transforms/Utils/BasicBlockUtils.h"
|
|
|
|
#include "llvm/Transforms/Utils/Cloning.h"
|
2020-05-20 17:08:08 +08:00
|
|
|
#include "llvm/Transforms/Utils/ScalarEvolutionExpander.h"
|
2015-07-11 02:55:13 +08:00
|
|
|
|
|
|
|
using namespace llvm;
|
|
|
|
|
[LoopVersioning] Annotate versioned loop with noalias metadata
Summary:
If we decide to version a loop to benefit a transformation, it makes
sense to record the now non-aliasing accesses in the newly versioned
loop. This allows non-aliasing information to be used by subsequent
passes.
One example is 456.hmmer in SPECint2006 where after loop distribution,
we vectorize one of the newly distributed loops. To vectorize we
version this loop to fully disambiguate may-aliasing accesses. If we
add the noalias markers, we can use the same information in a later DSE
pass to eliminate some dead stores which amounts to ~25% of the
instructions of this hot memory-pipeline-bound loop. The overall
performance improves by 18% on our ARM64.
The scoped noalias annotation is added in LoopVersioning. The patch
then enables this for loop distribution. A follow-on patch will enable
it for the vectorizer. Eventually this should be run by default when
versioning the loop but first I'd like to get some feedback whether my
understanding and application of scoped noalias metadata is correct.
Essentially my approach was to have a separate alias domain for each
versioning of the loop. For example, if we first version in loop
distribution and then in vectorization of the distributed loops, we have
a different set of memchecks for each versioning. By keeping the scopes
in different domains they can conveniently be defined independently
since different alias domains don't affect each other.
As written, I also have a separate domain for each loop. This is not
necessary and we could save some metadata here by using the same domain
across the different loops. I don't think it's a big deal either way.
Probably the best is to review the tests first to see if I mapped this
problem correctly to scoped noalias markers. I have plenty of comments
in the tests.
Note that the interface is prepared for the vectorizer which needs the
annotateInstWithNoAlias API. The vectorizer does not use LoopVersioning
so we need a way to pass in the versioned instructions. This is also
why the maps have to become part of the object state.
Also currently, we only have an AA-aware DSE after the vectorizer if we
also run the LTO pipeline. Depending how widely this triggers we may
want to schedule a DSE toward the end of the regular pass pipeline.
Reviewers: hfinkel, nadav, ashutosh.nema
Subscribers: mssimpso, aemerson, llvm-commits, mcrosier
Differential Revision: http://reviews.llvm.org/D16712
llvm-svn: 263743
2016-03-18 04:32:32 +08:00
|
|
|
static cl::opt<bool>
|
|
|
|
AnnotateNoAlias("loop-version-annotate-no-alias", cl::init(true),
|
|
|
|
cl::Hidden,
|
|
|
|
cl::desc("Add no-alias annotation for instructions that "
|
|
|
|
"are disambiguated by memchecks"));
|
|
|
|
|
2020-10-16 04:50:56 +08:00
|
|
|
LoopVersioning::LoopVersioning(const LoopAccessInfo &LAI,
|
|
|
|
ArrayRef<RuntimePointerCheck> Checks, Loop *L,
|
|
|
|
LoopInfo *LI, DominatorTree *DT,
|
|
|
|
ScalarEvolution *SE)
|
|
|
|
: VersionedLoop(L), NonVersionedLoop(nullptr),
|
|
|
|
AliasChecks(Checks.begin(), Checks.end()),
|
|
|
|
Preds(LAI.getPSE().getUnionPredicate()), LAI(LAI), LI(LI), DT(DT),
|
2015-11-09 21:26:09 +08:00
|
|
|
SE(SE) {
|
2015-07-11 02:55:13 +08:00
|
|
|
assert(L->getExitBlock() && "No single exit block");
|
2016-12-20 01:13:37 +08:00
|
|
|
assert(L->isLoopSimplifyForm() && "Loop is not in loop-simplify form");
|
2015-07-11 02:55:13 +08:00
|
|
|
}
|
|
|
|
|
2015-08-21 01:22:29 +08:00
|
|
|
void LoopVersioning::versionLoop(
|
|
|
|
const SmallVectorImpl<Instruction *> &DefsUsedOutside) {
|
2015-07-11 02:55:13 +08:00
|
|
|
Instruction *FirstCheckInst;
|
|
|
|
Instruction *MemRuntimeCheck;
|
2015-11-09 21:26:09 +08:00
|
|
|
Value *SCEVRuntimeCheck;
|
|
|
|
Value *RuntimeCheck = nullptr;
|
|
|
|
|
2015-07-11 02:55:13 +08:00
|
|
|
// Add the memcheck in the original preheader (this is empty initially).
|
2015-11-09 21:26:09 +08:00
|
|
|
BasicBlock *RuntimeCheckBB = VersionedLoop->getLoopPreheader();
|
2020-05-10 23:48:06 +08:00
|
|
|
const auto &RtPtrChecking = *LAI.getRuntimePointerChecking();
|
2015-07-11 02:55:13 +08:00
|
|
|
std::tie(FirstCheckInst, MemRuntimeCheck) =
|
2020-05-10 23:48:06 +08:00
|
|
|
addRuntimeChecks(RuntimeCheckBB->getTerminator(), VersionedLoop,
|
|
|
|
AliasChecks, RtPtrChecking.getSE());
|
2015-07-11 02:55:13 +08:00
|
|
|
|
2015-11-09 21:26:09 +08:00
|
|
|
SCEVExpander Exp(*SE, RuntimeCheckBB->getModule()->getDataLayout(),
|
|
|
|
"scev.check");
|
|
|
|
SCEVRuntimeCheck =
|
2020-10-16 04:50:56 +08:00
|
|
|
Exp.expandCodeForPredicate(&Preds, RuntimeCheckBB->getTerminator());
|
2015-11-09 21:26:09 +08:00
|
|
|
auto *CI = dyn_cast<ConstantInt>(SCEVRuntimeCheck);
|
|
|
|
|
|
|
|
// Discard the SCEV runtime check if it is always true.
|
|
|
|
if (CI && CI->isZero())
|
|
|
|
SCEVRuntimeCheck = nullptr;
|
|
|
|
|
|
|
|
if (MemRuntimeCheck && SCEVRuntimeCheck) {
|
|
|
|
RuntimeCheck = BinaryOperator::Create(Instruction::Or, MemRuntimeCheck,
|
2016-06-13 18:49:28 +08:00
|
|
|
SCEVRuntimeCheck, "lver.safe");
|
2015-11-09 21:26:09 +08:00
|
|
|
if (auto *I = dyn_cast<Instruction>(RuntimeCheck))
|
|
|
|
I->insertBefore(RuntimeCheckBB->getTerminator());
|
|
|
|
} else
|
|
|
|
RuntimeCheck = MemRuntimeCheck ? MemRuntimeCheck : SCEVRuntimeCheck;
|
|
|
|
|
|
|
|
assert(RuntimeCheck && "called even though we don't need "
|
|
|
|
"any runtime checks");
|
|
|
|
|
2015-07-11 02:55:13 +08:00
|
|
|
// Rename the block to make the IR more readable.
|
2015-11-09 21:26:09 +08:00
|
|
|
RuntimeCheckBB->setName(VersionedLoop->getHeader()->getName() +
|
|
|
|
".lver.check");
|
2015-07-11 02:55:13 +08:00
|
|
|
|
|
|
|
// Create empty preheader for the loop (and after cloning for the
|
|
|
|
// non-versioned loop).
|
2015-11-09 21:26:09 +08:00
|
|
|
BasicBlock *PH =
|
2019-09-13 16:03:32 +08:00
|
|
|
SplitBlock(RuntimeCheckBB, RuntimeCheckBB->getTerminator(), DT, LI,
|
|
|
|
nullptr, VersionedLoop->getHeader()->getName() + ".ph");
|
2015-07-11 02:55:13 +08:00
|
|
|
|
|
|
|
// Clone the loop including the preheader.
|
|
|
|
//
|
|
|
|
// FIXME: This does not currently preserve SimplifyLoop because the exit
|
|
|
|
// block is a join between the two loops.
|
|
|
|
SmallVector<BasicBlock *, 8> NonVersionedLoopBlocks;
|
|
|
|
NonVersionedLoop =
|
2015-11-09 21:26:09 +08:00
|
|
|
cloneLoopWithPreheader(PH, RuntimeCheckBB, VersionedLoop, VMap,
|
|
|
|
".lver.orig", LI, DT, NonVersionedLoopBlocks);
|
2015-07-11 02:55:13 +08:00
|
|
|
remapInstructionsInBlocks(NonVersionedLoopBlocks, VMap);
|
|
|
|
|
|
|
|
// Insert the conditional branch based on the result of the memchecks.
|
2015-11-09 21:26:09 +08:00
|
|
|
Instruction *OrigTerm = RuntimeCheckBB->getTerminator();
|
2015-07-11 02:55:13 +08:00
|
|
|
BranchInst::Create(NonVersionedLoop->getLoopPreheader(),
|
2015-11-09 21:26:09 +08:00
|
|
|
VersionedLoop->getLoopPreheader(), RuntimeCheck, OrigTerm);
|
2015-07-11 02:55:13 +08:00
|
|
|
OrigTerm->eraseFromParent();
|
|
|
|
|
|
|
|
// The loops merge in the original exit block. This is now dominated by the
|
|
|
|
// memchecking block.
|
2015-11-09 21:26:09 +08:00
|
|
|
DT->changeImmediateDominator(VersionedLoop->getExitBlock(), RuntimeCheckBB);
|
2015-08-21 01:22:29 +08:00
|
|
|
|
|
|
|
// Adds the necessary PHI nodes for the versioned loops based on the
|
|
|
|
// loop-defined values used outside of the loop.
|
|
|
|
addPHINodes(DefsUsedOutside);
|
2020-10-24 21:39:42 +08:00
|
|
|
formDedicatedExitBlocks(NonVersionedLoop, DT, LI, nullptr, true);
|
|
|
|
formDedicatedExitBlocks(VersionedLoop, DT, LI, nullptr, true);
|
|
|
|
assert(NonVersionedLoop->isLoopSimplifyForm() &&
|
|
|
|
VersionedLoop->isLoopSimplifyForm() &&
|
|
|
|
"The versioned loops should be in simplify form.");
|
2015-07-11 02:55:13 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
void LoopVersioning::addPHINodes(
|
|
|
|
const SmallVectorImpl<Instruction *> &DefsUsedOutside) {
|
|
|
|
BasicBlock *PHIBlock = VersionedLoop->getExitBlock();
|
|
|
|
assert(PHIBlock && "No single successor to loop exit block");
|
2016-06-14 17:38:54 +08:00
|
|
|
PHINode *PN;
|
2015-07-11 02:55:13 +08:00
|
|
|
|
2016-06-14 17:38:54 +08:00
|
|
|
// First add a single-operand PHI for each DefsUsedOutside if one does not
|
|
|
|
// exists yet.
|
2015-07-11 02:55:13 +08:00
|
|
|
for (auto *Inst : DefsUsedOutside) {
|
2016-06-14 17:38:54 +08:00
|
|
|
// See if we have a single-operand PHI with the value defined by the
|
2015-07-11 02:55:13 +08:00
|
|
|
// original loop.
|
|
|
|
for (auto I = PHIBlock->begin(); (PN = dyn_cast<PHINode>(I)); ++I) {
|
2016-06-14 17:39:01 +08:00
|
|
|
if (PN->getIncomingValue(0) == Inst)
|
2015-07-11 02:55:13 +08:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
// If not create it.
|
|
|
|
if (!PN) {
|
|
|
|
PN = PHINode::Create(Inst->getType(), 2, Inst->getName() + ".lver",
|
2015-10-13 10:39:05 +08:00
|
|
|
&PHIBlock->front());
|
2018-05-22 16:33:02 +08:00
|
|
|
SmallVector<User*, 8> UsersToUpdate;
|
|
|
|
for (User *U : Inst->users())
|
|
|
|
if (!VersionedLoop->contains(cast<Instruction>(U)->getParent()))
|
|
|
|
UsersToUpdate.push_back(U);
|
|
|
|
for (User *U : UsersToUpdate)
|
|
|
|
U->replaceUsesOfWith(Inst, PN);
|
2015-07-11 02:55:13 +08:00
|
|
|
PN->addIncoming(Inst, VersionedLoop->getExitingBlock());
|
|
|
|
}
|
2016-06-14 17:38:54 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
// Then for each PHI add the operand for the edge from the cloned loop.
|
|
|
|
for (auto I = PHIBlock->begin(); (PN = dyn_cast<PHINode>(I)); ++I) {
|
|
|
|
assert(PN->getNumOperands() == 1 &&
|
|
|
|
"Exit block should only have on predecessor");
|
|
|
|
|
|
|
|
// If the definition was cloned used that otherwise use the same value.
|
|
|
|
Value *ClonedValue = PN->getIncomingValue(0);
|
|
|
|
auto Mapped = VMap.find(ClonedValue);
|
|
|
|
if (Mapped != VMap.end())
|
|
|
|
ClonedValue = Mapped->second;
|
|
|
|
|
|
|
|
PN->addIncoming(ClonedValue, NonVersionedLoop->getExitingBlock());
|
2015-07-11 02:55:13 +08:00
|
|
|
}
|
|
|
|
}
|
2016-02-03 08:06:10 +08:00
|
|
|
|
[LoopVersioning] Annotate versioned loop with noalias metadata
Summary:
If we decide to version a loop to benefit a transformation, it makes
sense to record the now non-aliasing accesses in the newly versioned
loop. This allows non-aliasing information to be used by subsequent
passes.
One example is 456.hmmer in SPECint2006 where after loop distribution,
we vectorize one of the newly distributed loops. To vectorize we
version this loop to fully disambiguate may-aliasing accesses. If we
add the noalias markers, we can use the same information in a later DSE
pass to eliminate some dead stores which amounts to ~25% of the
instructions of this hot memory-pipeline-bound loop. The overall
performance improves by 18% on our ARM64.
The scoped noalias annotation is added in LoopVersioning. The patch
then enables this for loop distribution. A follow-on patch will enable
it for the vectorizer. Eventually this should be run by default when
versioning the loop but first I'd like to get some feedback whether my
understanding and application of scoped noalias metadata is correct.
Essentially my approach was to have a separate alias domain for each
versioning of the loop. For example, if we first version in loop
distribution and then in vectorization of the distributed loops, we have
a different set of memchecks for each versioning. By keeping the scopes
in different domains they can conveniently be defined independently
since different alias domains don't affect each other.
As written, I also have a separate domain for each loop. This is not
necessary and we could save some metadata here by using the same domain
across the different loops. I don't think it's a big deal either way.
Probably the best is to review the tests first to see if I mapped this
problem correctly to scoped noalias markers. I have plenty of comments
in the tests.
Note that the interface is prepared for the vectorizer which needs the
annotateInstWithNoAlias API. The vectorizer does not use LoopVersioning
so we need a way to pass in the versioned instructions. This is also
why the maps have to become part of the object state.
Also currently, we only have an AA-aware DSE after the vectorizer if we
also run the LTO pipeline. Depending how widely this triggers we may
want to schedule a DSE toward the end of the regular pass pipeline.
Reviewers: hfinkel, nadav, ashutosh.nema
Subscribers: mssimpso, aemerson, llvm-commits, mcrosier
Differential Revision: http://reviews.llvm.org/D16712
llvm-svn: 263743
2016-03-18 04:32:32 +08:00
|
|
|
void LoopVersioning::prepareNoAliasMetadata() {
|
|
|
|
// We need to turn the no-alias relation between pointer checking groups into
|
|
|
|
// no-aliasing annotations between instructions.
|
|
|
|
//
|
|
|
|
// We accomplish this by mapping each pointer checking group (a set of
|
|
|
|
// pointers memchecked together) to an alias scope and then also mapping each
|
|
|
|
// group to the list of scopes it can't alias.
|
|
|
|
|
|
|
|
const RuntimePointerChecking *RtPtrChecking = LAI.getRuntimePointerChecking();
|
|
|
|
LLVMContext &Context = VersionedLoop->getHeader()->getContext();
|
|
|
|
|
|
|
|
// First allocate an aliasing scope for each pointer checking group.
|
|
|
|
//
|
|
|
|
// While traversing through the checking groups in the loop, also create a
|
|
|
|
// reverse map from pointers to the pointer checking group they were assigned
|
|
|
|
// to.
|
|
|
|
MDBuilder MDB(Context);
|
|
|
|
MDNode *Domain = MDB.createAnonymousAliasScopeDomain("LVerDomain");
|
|
|
|
|
|
|
|
for (const auto &Group : RtPtrChecking->CheckingGroups) {
|
|
|
|
GroupToScope[&Group] = MDB.createAnonymousAliasScope(Domain);
|
|
|
|
|
|
|
|
for (unsigned PtrIdx : Group.Members)
|
|
|
|
PtrToGroup[RtPtrChecking->getPointerInfo(PtrIdx).PointerValue] = &Group;
|
|
|
|
}
|
|
|
|
|
|
|
|
// Go through the checks and for each pointer group, collect the scopes for
|
|
|
|
// each non-aliasing pointer group.
|
2020-04-29 04:35:02 +08:00
|
|
|
DenseMap<const RuntimeCheckingPtrGroup *, SmallVector<Metadata *, 4>>
|
[LoopVersioning] Annotate versioned loop with noalias metadata
Summary:
If we decide to version a loop to benefit a transformation, it makes
sense to record the now non-aliasing accesses in the newly versioned
loop. This allows non-aliasing information to be used by subsequent
passes.
One example is 456.hmmer in SPECint2006 where after loop distribution,
we vectorize one of the newly distributed loops. To vectorize we
version this loop to fully disambiguate may-aliasing accesses. If we
add the noalias markers, we can use the same information in a later DSE
pass to eliminate some dead stores which amounts to ~25% of the
instructions of this hot memory-pipeline-bound loop. The overall
performance improves by 18% on our ARM64.
The scoped noalias annotation is added in LoopVersioning. The patch
then enables this for loop distribution. A follow-on patch will enable
it for the vectorizer. Eventually this should be run by default when
versioning the loop but first I'd like to get some feedback whether my
understanding and application of scoped noalias metadata is correct.
Essentially my approach was to have a separate alias domain for each
versioning of the loop. For example, if we first version in loop
distribution and then in vectorization of the distributed loops, we have
a different set of memchecks for each versioning. By keeping the scopes
in different domains they can conveniently be defined independently
since different alias domains don't affect each other.
As written, I also have a separate domain for each loop. This is not
necessary and we could save some metadata here by using the same domain
across the different loops. I don't think it's a big deal either way.
Probably the best is to review the tests first to see if I mapped this
problem correctly to scoped noalias markers. I have plenty of comments
in the tests.
Note that the interface is prepared for the vectorizer which needs the
annotateInstWithNoAlias API. The vectorizer does not use LoopVersioning
so we need a way to pass in the versioned instructions. This is also
why the maps have to become part of the object state.
Also currently, we only have an AA-aware DSE after the vectorizer if we
also run the LTO pipeline. Depending how widely this triggers we may
want to schedule a DSE toward the end of the regular pass pipeline.
Reviewers: hfinkel, nadav, ashutosh.nema
Subscribers: mssimpso, aemerson, llvm-commits, mcrosier
Differential Revision: http://reviews.llvm.org/D16712
llvm-svn: 263743
2016-03-18 04:32:32 +08:00
|
|
|
GroupToNonAliasingScopes;
|
|
|
|
|
|
|
|
for (const auto &Check : AliasChecks)
|
|
|
|
GroupToNonAliasingScopes[Check.first].push_back(GroupToScope[Check.second]);
|
|
|
|
|
|
|
|
// Finally, transform the above to actually map to scope list which is what
|
|
|
|
// the metadata uses.
|
|
|
|
|
|
|
|
for (auto Pair : GroupToNonAliasingScopes)
|
|
|
|
GroupToNonAliasingScopeList[Pair.first] = MDNode::get(Context, Pair.second);
|
|
|
|
}
|
|
|
|
|
|
|
|
void LoopVersioning::annotateLoopWithNoAlias() {
|
|
|
|
if (!AnnotateNoAlias)
|
|
|
|
return;
|
|
|
|
|
|
|
|
// First prepare the maps.
|
|
|
|
prepareNoAliasMetadata();
|
|
|
|
|
|
|
|
// Add the scope and no-alias metadata to the instructions.
|
|
|
|
for (Instruction *I : LAI.getDepChecker().getMemoryInstructions()) {
|
|
|
|
annotateInstWithNoAlias(I);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-03-18 04:32:37 +08:00
|
|
|
void LoopVersioning::annotateInstWithNoAlias(Instruction *VersionedInst,
|
|
|
|
const Instruction *OrigInst) {
|
[LoopVersioning] Annotate versioned loop with noalias metadata
Summary:
If we decide to version a loop to benefit a transformation, it makes
sense to record the now non-aliasing accesses in the newly versioned
loop. This allows non-aliasing information to be used by subsequent
passes.
One example is 456.hmmer in SPECint2006 where after loop distribution,
we vectorize one of the newly distributed loops. To vectorize we
version this loop to fully disambiguate may-aliasing accesses. If we
add the noalias markers, we can use the same information in a later DSE
pass to eliminate some dead stores which amounts to ~25% of the
instructions of this hot memory-pipeline-bound loop. The overall
performance improves by 18% on our ARM64.
The scoped noalias annotation is added in LoopVersioning. The patch
then enables this for loop distribution. A follow-on patch will enable
it for the vectorizer. Eventually this should be run by default when
versioning the loop but first I'd like to get some feedback whether my
understanding and application of scoped noalias metadata is correct.
Essentially my approach was to have a separate alias domain for each
versioning of the loop. For example, if we first version in loop
distribution and then in vectorization of the distributed loops, we have
a different set of memchecks for each versioning. By keeping the scopes
in different domains they can conveniently be defined independently
since different alias domains don't affect each other.
As written, I also have a separate domain for each loop. This is not
necessary and we could save some metadata here by using the same domain
across the different loops. I don't think it's a big deal either way.
Probably the best is to review the tests first to see if I mapped this
problem correctly to scoped noalias markers. I have plenty of comments
in the tests.
Note that the interface is prepared for the vectorizer which needs the
annotateInstWithNoAlias API. The vectorizer does not use LoopVersioning
so we need a way to pass in the versioned instructions. This is also
why the maps have to become part of the object state.
Also currently, we only have an AA-aware DSE after the vectorizer if we
also run the LTO pipeline. Depending how widely this triggers we may
want to schedule a DSE toward the end of the regular pass pipeline.
Reviewers: hfinkel, nadav, ashutosh.nema
Subscribers: mssimpso, aemerson, llvm-commits, mcrosier
Differential Revision: http://reviews.llvm.org/D16712
llvm-svn: 263743
2016-03-18 04:32:32 +08:00
|
|
|
if (!AnnotateNoAlias)
|
|
|
|
return;
|
|
|
|
|
|
|
|
LLVMContext &Context = VersionedLoop->getHeader()->getContext();
|
2016-03-18 04:32:37 +08:00
|
|
|
const Value *Ptr = isa<LoadInst>(OrigInst)
|
|
|
|
? cast<LoadInst>(OrigInst)->getPointerOperand()
|
|
|
|
: cast<StoreInst>(OrigInst)->getPointerOperand();
|
[LoopVersioning] Annotate versioned loop with noalias metadata
Summary:
If we decide to version a loop to benefit a transformation, it makes
sense to record the now non-aliasing accesses in the newly versioned
loop. This allows non-aliasing information to be used by subsequent
passes.
One example is 456.hmmer in SPECint2006 where after loop distribution,
we vectorize one of the newly distributed loops. To vectorize we
version this loop to fully disambiguate may-aliasing accesses. If we
add the noalias markers, we can use the same information in a later DSE
pass to eliminate some dead stores which amounts to ~25% of the
instructions of this hot memory-pipeline-bound loop. The overall
performance improves by 18% on our ARM64.
The scoped noalias annotation is added in LoopVersioning. The patch
then enables this for loop distribution. A follow-on patch will enable
it for the vectorizer. Eventually this should be run by default when
versioning the loop but first I'd like to get some feedback whether my
understanding and application of scoped noalias metadata is correct.
Essentially my approach was to have a separate alias domain for each
versioning of the loop. For example, if we first version in loop
distribution and then in vectorization of the distributed loops, we have
a different set of memchecks for each versioning. By keeping the scopes
in different domains they can conveniently be defined independently
since different alias domains don't affect each other.
As written, I also have a separate domain for each loop. This is not
necessary and we could save some metadata here by using the same domain
across the different loops. I don't think it's a big deal either way.
Probably the best is to review the tests first to see if I mapped this
problem correctly to scoped noalias markers. I have plenty of comments
in the tests.
Note that the interface is prepared for the vectorizer which needs the
annotateInstWithNoAlias API. The vectorizer does not use LoopVersioning
so we need a way to pass in the versioned instructions. This is also
why the maps have to become part of the object state.
Also currently, we only have an AA-aware DSE after the vectorizer if we
also run the LTO pipeline. Depending how widely this triggers we may
want to schedule a DSE toward the end of the regular pass pipeline.
Reviewers: hfinkel, nadav, ashutosh.nema
Subscribers: mssimpso, aemerson, llvm-commits, mcrosier
Differential Revision: http://reviews.llvm.org/D16712
llvm-svn: 263743
2016-03-18 04:32:32 +08:00
|
|
|
|
|
|
|
// Find the group for the pointer and then add the scope metadata.
|
|
|
|
auto Group = PtrToGroup.find(Ptr);
|
|
|
|
if (Group != PtrToGroup.end()) {
|
2016-03-18 04:32:37 +08:00
|
|
|
VersionedInst->setMetadata(
|
[LoopVersioning] Annotate versioned loop with noalias metadata
Summary:
If we decide to version a loop to benefit a transformation, it makes
sense to record the now non-aliasing accesses in the newly versioned
loop. This allows non-aliasing information to be used by subsequent
passes.
One example is 456.hmmer in SPECint2006 where after loop distribution,
we vectorize one of the newly distributed loops. To vectorize we
version this loop to fully disambiguate may-aliasing accesses. If we
add the noalias markers, we can use the same information in a later DSE
pass to eliminate some dead stores which amounts to ~25% of the
instructions of this hot memory-pipeline-bound loop. The overall
performance improves by 18% on our ARM64.
The scoped noalias annotation is added in LoopVersioning. The patch
then enables this for loop distribution. A follow-on patch will enable
it for the vectorizer. Eventually this should be run by default when
versioning the loop but first I'd like to get some feedback whether my
understanding and application of scoped noalias metadata is correct.
Essentially my approach was to have a separate alias domain for each
versioning of the loop. For example, if we first version in loop
distribution and then in vectorization of the distributed loops, we have
a different set of memchecks for each versioning. By keeping the scopes
in different domains they can conveniently be defined independently
since different alias domains don't affect each other.
As written, I also have a separate domain for each loop. This is not
necessary and we could save some metadata here by using the same domain
across the different loops. I don't think it's a big deal either way.
Probably the best is to review the tests first to see if I mapped this
problem correctly to scoped noalias markers. I have plenty of comments
in the tests.
Note that the interface is prepared for the vectorizer which needs the
annotateInstWithNoAlias API. The vectorizer does not use LoopVersioning
so we need a way to pass in the versioned instructions. This is also
why the maps have to become part of the object state.
Also currently, we only have an AA-aware DSE after the vectorizer if we
also run the LTO pipeline. Depending how widely this triggers we may
want to schedule a DSE toward the end of the regular pass pipeline.
Reviewers: hfinkel, nadav, ashutosh.nema
Subscribers: mssimpso, aemerson, llvm-commits, mcrosier
Differential Revision: http://reviews.llvm.org/D16712
llvm-svn: 263743
2016-03-18 04:32:32 +08:00
|
|
|
LLVMContext::MD_alias_scope,
|
2016-03-18 04:32:37 +08:00
|
|
|
MDNode::concatenate(
|
|
|
|
VersionedInst->getMetadata(LLVMContext::MD_alias_scope),
|
|
|
|
MDNode::get(Context, GroupToScope[Group->second])));
|
[LoopVersioning] Annotate versioned loop with noalias metadata
Summary:
If we decide to version a loop to benefit a transformation, it makes
sense to record the now non-aliasing accesses in the newly versioned
loop. This allows non-aliasing information to be used by subsequent
passes.
One example is 456.hmmer in SPECint2006 where after loop distribution,
we vectorize one of the newly distributed loops. To vectorize we
version this loop to fully disambiguate may-aliasing accesses. If we
add the noalias markers, we can use the same information in a later DSE
pass to eliminate some dead stores which amounts to ~25% of the
instructions of this hot memory-pipeline-bound loop. The overall
performance improves by 18% on our ARM64.
The scoped noalias annotation is added in LoopVersioning. The patch
then enables this for loop distribution. A follow-on patch will enable
it for the vectorizer. Eventually this should be run by default when
versioning the loop but first I'd like to get some feedback whether my
understanding and application of scoped noalias metadata is correct.
Essentially my approach was to have a separate alias domain for each
versioning of the loop. For example, if we first version in loop
distribution and then in vectorization of the distributed loops, we have
a different set of memchecks for each versioning. By keeping the scopes
in different domains they can conveniently be defined independently
since different alias domains don't affect each other.
As written, I also have a separate domain for each loop. This is not
necessary and we could save some metadata here by using the same domain
across the different loops. I don't think it's a big deal either way.
Probably the best is to review the tests first to see if I mapped this
problem correctly to scoped noalias markers. I have plenty of comments
in the tests.
Note that the interface is prepared for the vectorizer which needs the
annotateInstWithNoAlias API. The vectorizer does not use LoopVersioning
so we need a way to pass in the versioned instructions. This is also
why the maps have to become part of the object state.
Also currently, we only have an AA-aware DSE after the vectorizer if we
also run the LTO pipeline. Depending how widely this triggers we may
want to schedule a DSE toward the end of the regular pass pipeline.
Reviewers: hfinkel, nadav, ashutosh.nema
Subscribers: mssimpso, aemerson, llvm-commits, mcrosier
Differential Revision: http://reviews.llvm.org/D16712
llvm-svn: 263743
2016-03-18 04:32:32 +08:00
|
|
|
|
|
|
|
// Add the no-alias metadata.
|
|
|
|
auto NonAliasingScopeList = GroupToNonAliasingScopeList.find(Group->second);
|
|
|
|
if (NonAliasingScopeList != GroupToNonAliasingScopeList.end())
|
2016-03-18 04:32:37 +08:00
|
|
|
VersionedInst->setMetadata(
|
[LoopVersioning] Annotate versioned loop with noalias metadata
Summary:
If we decide to version a loop to benefit a transformation, it makes
sense to record the now non-aliasing accesses in the newly versioned
loop. This allows non-aliasing information to be used by subsequent
passes.
One example is 456.hmmer in SPECint2006 where after loop distribution,
we vectorize one of the newly distributed loops. To vectorize we
version this loop to fully disambiguate may-aliasing accesses. If we
add the noalias markers, we can use the same information in a later DSE
pass to eliminate some dead stores which amounts to ~25% of the
instructions of this hot memory-pipeline-bound loop. The overall
performance improves by 18% on our ARM64.
The scoped noalias annotation is added in LoopVersioning. The patch
then enables this for loop distribution. A follow-on patch will enable
it for the vectorizer. Eventually this should be run by default when
versioning the loop but first I'd like to get some feedback whether my
understanding and application of scoped noalias metadata is correct.
Essentially my approach was to have a separate alias domain for each
versioning of the loop. For example, if we first version in loop
distribution and then in vectorization of the distributed loops, we have
a different set of memchecks for each versioning. By keeping the scopes
in different domains they can conveniently be defined independently
since different alias domains don't affect each other.
As written, I also have a separate domain for each loop. This is not
necessary and we could save some metadata here by using the same domain
across the different loops. I don't think it's a big deal either way.
Probably the best is to review the tests first to see if I mapped this
problem correctly to scoped noalias markers. I have plenty of comments
in the tests.
Note that the interface is prepared for the vectorizer which needs the
annotateInstWithNoAlias API. The vectorizer does not use LoopVersioning
so we need a way to pass in the versioned instructions. This is also
why the maps have to become part of the object state.
Also currently, we only have an AA-aware DSE after the vectorizer if we
also run the LTO pipeline. Depending how widely this triggers we may
want to schedule a DSE toward the end of the regular pass pipeline.
Reviewers: hfinkel, nadav, ashutosh.nema
Subscribers: mssimpso, aemerson, llvm-commits, mcrosier
Differential Revision: http://reviews.llvm.org/D16712
llvm-svn: 263743
2016-03-18 04:32:32 +08:00
|
|
|
LLVMContext::MD_noalias,
|
2016-03-18 04:32:37 +08:00
|
|
|
MDNode::concatenate(
|
|
|
|
VersionedInst->getMetadata(LLVMContext::MD_noalias),
|
|
|
|
NonAliasingScopeList->second));
|
[LoopVersioning] Annotate versioned loop with noalias metadata
Summary:
If we decide to version a loop to benefit a transformation, it makes
sense to record the now non-aliasing accesses in the newly versioned
loop. This allows non-aliasing information to be used by subsequent
passes.
One example is 456.hmmer in SPECint2006 where after loop distribution,
we vectorize one of the newly distributed loops. To vectorize we
version this loop to fully disambiguate may-aliasing accesses. If we
add the noalias markers, we can use the same information in a later DSE
pass to eliminate some dead stores which amounts to ~25% of the
instructions of this hot memory-pipeline-bound loop. The overall
performance improves by 18% on our ARM64.
The scoped noalias annotation is added in LoopVersioning. The patch
then enables this for loop distribution. A follow-on patch will enable
it for the vectorizer. Eventually this should be run by default when
versioning the loop but first I'd like to get some feedback whether my
understanding and application of scoped noalias metadata is correct.
Essentially my approach was to have a separate alias domain for each
versioning of the loop. For example, if we first version in loop
distribution and then in vectorization of the distributed loops, we have
a different set of memchecks for each versioning. By keeping the scopes
in different domains they can conveniently be defined independently
since different alias domains don't affect each other.
As written, I also have a separate domain for each loop. This is not
necessary and we could save some metadata here by using the same domain
across the different loops. I don't think it's a big deal either way.
Probably the best is to review the tests first to see if I mapped this
problem correctly to scoped noalias markers. I have plenty of comments
in the tests.
Note that the interface is prepared for the vectorizer which needs the
annotateInstWithNoAlias API. The vectorizer does not use LoopVersioning
so we need a way to pass in the versioned instructions. This is also
why the maps have to become part of the object state.
Also currently, we only have an AA-aware DSE after the vectorizer if we
also run the LTO pipeline. Depending how widely this triggers we may
want to schedule a DSE toward the end of the regular pass pipeline.
Reviewers: hfinkel, nadav, ashutosh.nema
Subscribers: mssimpso, aemerson, llvm-commits, mcrosier
Differential Revision: http://reviews.llvm.org/D16712
llvm-svn: 263743
2016-03-18 04:32:32 +08:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-02-03 08:06:10 +08:00
|
|
|
namespace {
|
2020-08-01 08:30:30 +08:00
|
|
|
bool runImpl(LoopInfo *LI, function_ref<const LoopAccessInfo &(Loop &)> GetLAA,
|
|
|
|
DominatorTree *DT, ScalarEvolution *SE) {
|
|
|
|
// Build up a worklist of inner-loops to version. This is necessary as the
|
|
|
|
// act of versioning a loop creates new loops and can invalidate iterators
|
|
|
|
// across the loops.
|
|
|
|
SmallVector<Loop *, 8> Worklist;
|
|
|
|
|
|
|
|
for (Loop *TopLevelLoop : *LI)
|
|
|
|
for (Loop *L : depth_first(TopLevelLoop))
|
|
|
|
// We only handle inner-most loops.
|
2020-09-23 04:28:00 +08:00
|
|
|
if (L->isInnermost())
|
2020-08-01 08:30:30 +08:00
|
|
|
Worklist.push_back(L);
|
|
|
|
|
|
|
|
// Now walk the identified inner loops.
|
|
|
|
bool Changed = false;
|
|
|
|
for (Loop *L : Worklist) {
|
|
|
|
const LoopAccessInfo &LAI = GetLAA(*L);
|
|
|
|
if (L->isLoopSimplifyForm() && !LAI.hasConvergentOp() &&
|
|
|
|
(LAI.getNumRuntimePointerChecks() ||
|
|
|
|
!LAI.getPSE().getUnionPredicate().isAlwaysTrue())) {
|
2020-10-16 04:50:56 +08:00
|
|
|
LoopVersioning LVer(LAI, LAI.getRuntimePointerChecking()->getChecks(), L,
|
|
|
|
LI, DT, SE);
|
2020-08-01 08:30:30 +08:00
|
|
|
LVer.versionLoop();
|
|
|
|
LVer.annotateLoopWithNoAlias();
|
|
|
|
Changed = true;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return Changed;
|
|
|
|
}
|
|
|
|
|
2018-05-01 23:54:18 +08:00
|
|
|
/// Also expose this is a pass. Currently this is only used for
|
2016-02-03 08:06:10 +08:00
|
|
|
/// unit-testing. It adds all memchecks necessary to remove all may-aliasing
|
|
|
|
/// array accesses from the loop.
|
2020-08-01 08:30:30 +08:00
|
|
|
class LoopVersioningLegacyPass : public FunctionPass {
|
2016-02-03 08:06:10 +08:00
|
|
|
public:
|
2020-08-01 08:30:30 +08:00
|
|
|
LoopVersioningLegacyPass() : FunctionPass(ID) {
|
|
|
|
initializeLoopVersioningLegacyPassPass(*PassRegistry::getPassRegistry());
|
2016-02-03 08:06:10 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
bool runOnFunction(Function &F) override {
|
|
|
|
auto *LI = &getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
|
2020-08-01 08:30:30 +08:00
|
|
|
auto GetLAA = [&](Loop &L) -> const LoopAccessInfo & {
|
|
|
|
return getAnalysis<LoopAccessLegacyAnalysis>().getInfo(&L);
|
|
|
|
};
|
|
|
|
|
2016-02-03 08:06:10 +08:00
|
|
|
auto *DT = &getAnalysis<DominatorTreeWrapperPass>().getDomTree();
|
|
|
|
auto *SE = &getAnalysis<ScalarEvolutionWrapperPass>().getSE();
|
|
|
|
|
2020-08-01 08:30:30 +08:00
|
|
|
return runImpl(LI, GetLAA, DT, SE);
|
2016-02-03 08:06:10 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
void getAnalysisUsage(AnalysisUsage &AU) const override {
|
|
|
|
AU.addRequired<LoopInfoWrapperPass>();
|
|
|
|
AU.addPreserved<LoopInfoWrapperPass>();
|
2016-07-09 04:55:26 +08:00
|
|
|
AU.addRequired<LoopAccessLegacyAnalysis>();
|
2016-02-03 08:06:10 +08:00
|
|
|
AU.addRequired<DominatorTreeWrapperPass>();
|
|
|
|
AU.addPreserved<DominatorTreeWrapperPass>();
|
|
|
|
AU.addRequired<ScalarEvolutionWrapperPass>();
|
|
|
|
}
|
|
|
|
|
|
|
|
static char ID;
|
|
|
|
};
|
|
|
|
}
|
|
|
|
|
|
|
|
#define LVER_OPTION "loop-versioning"
|
|
|
|
#define DEBUG_TYPE LVER_OPTION
|
|
|
|
|
2020-08-01 08:30:30 +08:00
|
|
|
char LoopVersioningLegacyPass::ID;
|
2016-02-03 08:06:10 +08:00
|
|
|
static const char LVer_name[] = "Loop Versioning";
|
|
|
|
|
2020-08-01 08:30:30 +08:00
|
|
|
INITIALIZE_PASS_BEGIN(LoopVersioningLegacyPass, LVER_OPTION, LVer_name, false,
|
|
|
|
false)
|
2016-02-03 08:06:10 +08:00
|
|
|
INITIALIZE_PASS_DEPENDENCY(LoopInfoWrapperPass)
|
2016-07-09 04:55:26 +08:00
|
|
|
INITIALIZE_PASS_DEPENDENCY(LoopAccessLegacyAnalysis)
|
2016-02-03 08:06:10 +08:00
|
|
|
INITIALIZE_PASS_DEPENDENCY(DominatorTreeWrapperPass)
|
|
|
|
INITIALIZE_PASS_DEPENDENCY(ScalarEvolutionWrapperPass)
|
2020-08-01 08:30:30 +08:00
|
|
|
INITIALIZE_PASS_END(LoopVersioningLegacyPass, LVER_OPTION, LVer_name, false,
|
|
|
|
false)
|
2016-02-03 08:06:10 +08:00
|
|
|
|
|
|
|
namespace llvm {
|
2020-08-01 08:30:30 +08:00
|
|
|
FunctionPass *createLoopVersioningLegacyPass() {
|
|
|
|
return new LoopVersioningLegacyPass();
|
2016-02-03 08:06:10 +08:00
|
|
|
}
|
2020-08-01 08:30:30 +08:00
|
|
|
|
|
|
|
PreservedAnalyses LoopVersioningPass::run(Function &F,
|
|
|
|
FunctionAnalysisManager &AM) {
|
|
|
|
auto &SE = AM.getResult<ScalarEvolutionAnalysis>(F);
|
|
|
|
auto &LI = AM.getResult<LoopAnalysis>(F);
|
|
|
|
auto &TTI = AM.getResult<TargetIRAnalysis>(F);
|
|
|
|
auto &DT = AM.getResult<DominatorTreeAnalysis>(F);
|
|
|
|
auto &TLI = AM.getResult<TargetLibraryAnalysis>(F);
|
|
|
|
auto &AA = AM.getResult<AAManager>(F);
|
|
|
|
auto &AC = AM.getResult<AssumptionAnalysis>(F);
|
|
|
|
MemorySSA *MSSA = EnableMSSALoopDependency
|
|
|
|
? &AM.getResult<MemorySSAAnalysis>(F).getMSSA()
|
|
|
|
: nullptr;
|
|
|
|
|
|
|
|
auto &LAM = AM.getResult<LoopAnalysisManagerFunctionProxy>(F).getManager();
|
|
|
|
auto GetLAA = [&](Loop &L) -> const LoopAccessInfo & {
|
2020-09-16 07:09:30 +08:00
|
|
|
LoopStandardAnalysisResults AR = {AA, AC, DT, LI, SE,
|
|
|
|
TLI, TTI, nullptr, MSSA};
|
2020-08-01 08:30:30 +08:00
|
|
|
return LAM.getResult<LoopAccessAnalysis>(L, AR);
|
|
|
|
};
|
|
|
|
|
|
|
|
if (runImpl(&LI, GetLAA, &DT, &SE))
|
|
|
|
return PreservedAnalyses::none();
|
|
|
|
return PreservedAnalyses::all();
|
2016-02-03 08:06:10 +08:00
|
|
|
}
|
2020-08-01 08:30:30 +08:00
|
|
|
} // namespace llvm
|