maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/llvm/lib/Analysis/BranchProbabilityInfo.cpp

847 lines
29 KiB
C++
Raw Normal View History

[Analysis] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC). llvm-svn: 308787
2017-07-22 05:37:46 +08:00
//===- BranchProbabilityInfo.cpp - Branch Probability Analysis ------------===//
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
//
// The LLVM Compiler Infrastructure
//
// This file is distributed under the University of Illinois Open Source
// License. See LICENSE.TXT for details.
//
//===----------------------------------------------------------------------===//
//
// Loops should be simplified before this analysis.
//
//===----------------------------------------------------------------------===//
Use the new script to sort the includes of every file under lib. Sooooo many of these had incorrect or strange main module includes. I have manually inspected all of these, and fixed the main module include to be the nearest plausible thing I could find. If you own or care about any of these source files, I encourage you to take some time and check that these edits were sensible. I can't have broken anything (I strictly added headers, and reordered them, never removed), but they may not be the headers you'd really like to identify as containing the API being implemented. Many forward declarations and missing includes were added to a header files to allow them to parse cleanly when included first. The main module rule does in fact have its merits. =] llvm-svn: 169131
2012-12-04 00:50:05 +08:00
#include "llvm/Analysis/BranchProbabilityInfo.h"
#include "llvm/ADT/PostOrderIterator.h"
[Analysis] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC). llvm-svn: 308787
2017-07-22 05:37:46 +08:00
#include "llvm/ADT/STLExtras.h"
#include "llvm/ADT/SmallVector.h"
Use the new script to sort the includes of every file under lib. Sooooo many of these had incorrect or strange main module includes. I have manually inspected all of these, and fixed the main module include to be the nearest plausible thing I could find. If you own or care about any of these source files, I encourage you to take some time and check that these edits were sensible. I can't have broken anything (I strictly added headers, and reordered them, never removed), but they may not be the headers you'd really like to identify as containing the API being implemented. Many forward declarations and missing includes were added to a header files to allow them to parse cleanly when included first. The main module rule does in fact have its merits. =] llvm-svn: 169131
2012-12-04 00:50:05 +08:00
#include "llvm/Analysis/LoopInfo.h"
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 17:44:40 +08:00
#include "llvm/Analysis/TargetLibraryInfo.h"
[Analysis] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC). llvm-svn: 308787
2017-07-22 05:37:46 +08:00
#include "llvm/IR/Attributes.h"
#include "llvm/IR/BasicBlock.h"
[Modules] Move CFG.h to the IR library as it defines graph traits over IR types. llvm-svn: 202827
2014-03-04 19:45:46 +08:00
#include "llvm/IR/CFG.h"
Move all of the header files which are involved in modelling the LLVM IR into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. llvm-svn: 171366
2013-01-02 19:36:10 +08:00
#include "llvm/IR/Constants.h"
#include "llvm/IR/Function.h"
[Analysis] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC). llvm-svn: 308787
2017-07-22 05:37:46 +08:00
#include "llvm/IR/InstrTypes.h"
#include "llvm/IR/Instruction.h"
Move all of the header files which are involved in modelling the LLVM IR into their new header subdirectory: include/llvm/IR. This matches the directory structure of lib, and begins to correct a long standing point of file layout clutter in LLVM. There are still more header files to move here, but I wanted to handle them in separate commits to make tracking what files make sense at each layer easier. The only really questionable files here are the target intrinsic tablegen files. But that's a battle I'd rather not fight today. I've updated both CMake and Makefile build systems (I think, and my tests think, but I may have missed something). I've also re-sorted the includes throughout the project. I'll be committing updates to Clang, DragonEgg, and Polly momentarily. llvm-svn: 171366
2013-01-02 19:36:10 +08:00
#include "llvm/IR/Instructions.h"
#include "llvm/IR/LLVMContext.h"
#include "llvm/IR/Metadata.h"
[Analysis] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC). llvm-svn: 308787
2017-07-22 05:37:46 +08:00
#include "llvm/IR/PassManager.h"
#include "llvm/IR/Type.h"
#include "llvm/IR/Value.h"
#include "llvm/Pass.h"
#include "llvm/Support/BranchProbability.h"
#include "llvm/Support/Casting.h"
Branch profiling: floating-point avoidance. Patch by: Jakub Staszak! Introduces BranchProbability. Changes unsigned to uint32_t all over and uint64_t only when overflow is expected. llvm-svn: 132867
2011-06-11 09:05:22 +08:00
#include "llvm/Support/Debug.h"
Purge unused includes throughout libSupport. NFC. llvm-svn: 232976
2015-03-24 02:07:13 +08:00
#include "llvm/Support/raw_ostream.h"
[Analysis] Fix some Clang-tidy modernize and Include What You Use warnings; other minor fixes (NFC). llvm-svn: 308787
2017-07-22 05:37:46 +08:00
#include <cassert>
#include <cstdint>
#include <iterator>
#include <utility>
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
using namespace llvm;
[Modules] Fix potential ODR violations by sinking the DEBUG_TYPE definition below all the header #include lines, lib/Analysis/... edition. This one has a bit extra as there were *other* #define's before #include lines in addition to DEBUG_TYPE. I've sunk all of them as a block. llvm-svn: 206843
2014-04-22 10:48:03 +08:00
#define DEBUG_TYPE "branch-prob"
Add options to dump block frequency/branch probability info in text. Summary: Add options -print-bfi/-print-bpi that dump block frequency and branch probability info like -view-block-freq-propagation-dags and -view-machine-block-freq-propagation-dags do but in text. This is useful when the graph is very large and complex (the dot command crashes, lines/edges too close to tell apart, hard to navigate without textual search) or simply when text is preferred. Reviewers: davidxl Reviewed By: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D37165 llvm-svn: 311822
2017-08-26 08:31:00 +08:00
static cl::opt<bool> PrintBranchProb(
"print-bpi", cl::init(false), cl::Hidden,
cl::desc("Print the branch probability info."));
cl::opt<std::string> PrintBranchProbFuncName(
"print-bpi-func-name", cl::Hidden,
cl::desc("The option to specify the name of the function "
"whose branch probability info is printed."));
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 06:48:29 +08:00
INITIALIZE_PASS_BEGIN(BranchProbabilityInfoWrapperPass, "branch-prob",
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
"Branch Probability Analysis", false, true)
[PM] Split the LoopInfo object apart from the legacy pass, creating a LoopInfoWrapperPass to wire the object up to the legacy pass manager. This switches all the clients of LoopInfo over and paves the way to port LoopInfo to the new pass manager. No functionality change is intended with this iteration. llvm-svn: 226373
2015-01-17 22:16:18 +08:00
INITIALIZE_PASS_DEPENDENCY(LoopInfoWrapperPass)
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 17:44:40 +08:00
INITIALIZE_PASS_DEPENDENCY(TargetLibraryInfoWrapperPass)
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 06:48:29 +08:00
INITIALIZE_PASS_END(BranchProbabilityInfoWrapperPass, "branch-prob",
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
"Branch Probability Analysis", false, true)
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 06:48:29 +08:00
char BranchProbabilityInfoWrapperPass::ID = 0;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
Simplify the design of BranchProbabilityInfo by collapsing it into a single class. Previously it was split between two classes, one internal and one external. The concern seemed to center around exposing the weights used, but those can remain confined to the implementation file. Having a single class to maintain the state and analyses in use will also simplify several of the enhancements I want to make to our static heuristics. llvm-svn: 142783
2011-10-24 09:40:45 +08:00
// Weights are for internal use only. They are used by heuristics to help to
// estimate edges' probability. Example:
//
// Using "Loop Branch Heuristics" we predict weights of edges for the
// block BB2.
// ...
// |
// V
// BB1<-+
// | |
// | | (Weight = 124)
// V |
// BB2--+
// |
// | (Weight = 4)
// V
// BB3
//
// Probability of the edge BB2->BB1 = 124 / (124 + 4) = 0.96875
// Probability of the edge BB2->BB3 = 4 / (124 + 4) = 0.03125
static const uint32_t LBH_TAKEN_WEIGHT = 124;
static const uint32_t LBH_NONTAKEN_WEIGHT = 4;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
[BPI] Reduce the probability of unreachable edge to minimal value greater than 0 The probability of edge coming to unreachable block should be as low as possible. The change reduces the probability to minimal value greater than zero. The bug https://bugs.llvm.org/show_bug.cgi?id=32214 show the example when the probability of edge coming to unreachable block is greater than for edge coming to out of the loop and it causes incorrect loop rotation. Please note that with this change the behavior of unreachable heuristic is a bit different than others. Specifically, before this change the sum of probabilities coming to unreachable blocks have the same weight for all branches (it was just split over all edges of this block coming to unreachable blocks). With this change it might be slightly different but not to much due to probability of taken branch to unreachable block is really small. Reviewers: chandlerc, sanjoy, vsk, congh, junbuml, davidxl, dexonsmith Reviewed By: chandlerc, dexonsmith Subscribers: reames, llvm-commits Differential Revision: https://reviews.llvm.org/D30633 llvm-svn: 303327
2017-05-18 14:11:56 +08:00
/// \brief Unreachable-terminating branch taken probability.
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 20:01:08 +08:00
///
[BPI] Reduce the probability of unreachable edge to minimal value greater than 0 The probability of edge coming to unreachable block should be as low as possible. The change reduces the probability to minimal value greater than zero. The bug https://bugs.llvm.org/show_bug.cgi?id=32214 show the example when the probability of edge coming to unreachable block is greater than for edge coming to out of the loop and it causes incorrect loop rotation. Please note that with this change the behavior of unreachable heuristic is a bit different than others. Specifically, before this change the sum of probabilities coming to unreachable blocks have the same weight for all branches (it was just split over all edges of this block coming to unreachable blocks). With this change it might be slightly different but not to much due to probability of taken branch to unreachable block is really small. Reviewers: chandlerc, sanjoy, vsk, congh, junbuml, davidxl, dexonsmith Reviewed By: chandlerc, dexonsmith Subscribers: reames, llvm-commits Differential Revision: https://reviews.llvm.org/D30633 llvm-svn: 303327
2017-05-18 14:11:56 +08:00
/// This is the probability for a branch being taken to a block that terminates
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 20:01:08 +08:00
/// (eventually) in unreachable. These are predicted as unlikely as possible.
[BPI] Reduce the probability of unreachable edge to minimal value greater than 0 The probability of edge coming to unreachable block should be as low as possible. The change reduces the probability to minimal value greater than zero. The bug https://bugs.llvm.org/show_bug.cgi?id=32214 show the example when the probability of edge coming to unreachable block is greater than for edge coming to out of the loop and it causes incorrect loop rotation. Please note that with this change the behavior of unreachable heuristic is a bit different than others. Specifically, before this change the sum of probabilities coming to unreachable blocks have the same weight for all branches (it was just split over all edges of this block coming to unreachable blocks). With this change it might be slightly different but not to much due to probability of taken branch to unreachable block is really small. Reviewers: chandlerc, sanjoy, vsk, congh, junbuml, davidxl, dexonsmith Reviewed By: chandlerc, dexonsmith Subscribers: reames, llvm-commits Differential Revision: https://reviews.llvm.org/D30633 llvm-svn: 303327
2017-05-18 14:11:56 +08:00
/// All reachable probability will equally share the remaining part.
static const BranchProbability UR_TAKEN_PROB = BranchProbability::getRaw(1);
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 12:33:04 +08:00
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 20:26:52 +08:00
/// \brief Weight for a branch taken going into a cold block.
///
/// This is the weight for a branch taken toward a block marked
/// cold. A block is marked cold if it's postdominated by a
/// block containing a call to a cold function. Cold functions
/// are those marked with attribute 'cold'.
static const uint32_t CC_TAKEN_WEIGHT = 4;
/// \brief Weight for a branch not-taken into a cold block.
///
/// This is the weight for a branch not taken toward a block marked
/// cold.
static const uint32_t CC_NONTAKEN_WEIGHT = 64;
Simplify the design of BranchProbabilityInfo by collapsing it into a single class. Previously it was split between two classes, one internal and one external. The concern seemed to center around exposing the weights used, but those can remain confined to the implementation file. Having a single class to maintain the state and analyses in use will also simplify several of the enhancements I want to make to our static heuristics. llvm-svn: 142783
2011-10-24 09:40:45 +08:00
static const uint32_t PH_TAKEN_WEIGHT = 20;
static const uint32_t PH_NONTAKEN_WEIGHT = 12;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
Simplify the design of BranchProbabilityInfo by collapsing it into a single class. Previously it was split between two classes, one internal and one external. The concern seemed to center around exposing the weights used, but those can remain confined to the implementation file. Having a single class to maintain the state and analyses in use will also simplify several of the enhancements I want to make to our static heuristics. llvm-svn: 142783
2011-10-24 09:40:45 +08:00
static const uint32_t ZH_TAKEN_WEIGHT = 20;
static const uint32_t ZH_NONTAKEN_WEIGHT = 12;
static const uint32_t FPH_TAKEN_WEIGHT = 20;
static const uint32_t FPH_NONTAKEN_WEIGHT = 12;
Set the branch probability of branching to the 'normal' destination of an invoke instruction to something absurdly high, while setting the probability of branching to the 'unwind' destination to the bare minimum. This should set cause the normal destination's invoke blocks to be moved closer to the invoke. PR13612 llvm-svn: 161944
2012-08-15 20:22:35 +08:00
/// \brief Invoke-terminating normal branch taken weight
///
/// This is the weight for branching to the normal destination of an invoke
/// instruction. We expect this to happen most of the time. Set the weight to an
/// absurdly high value so that nested loops subsume it.
static const uint32_t IH_TAKEN_WEIGHT = 1024 * 1024 - 1;
/// \brief Invoke-terminating normal branch not-taken weight.
///
/// This is the weight for branching to the unwind destination of an invoke
/// instruction. This is essentially never taken.
static const uint32_t IH_NONTAKEN_WEIGHT = 1;
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 13:42:14 +08:00
/// \brief Add \p BB to PostDominatedByUnreachable set if applicable.
void
BranchProbabilityInfo::updatePostDominatedByUnreachable(const BasicBlock *BB) {
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-08 05:59:28 +08:00
const TerminatorInst *TI = BB->getTerminator();
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 20:01:08 +08:00
if (TI->getNumSuccessors() == 0) {
[BPI] Consider deoptimize calls as "unreachable" Summary: Calls to @llvm.experimental.deoptimize are expected to "never execute", so optimize them as such. Reviewers: chandlerc Subscribers: junbuml, mcrosier, llvm-commits Differential Revision: http://reviews.llvm.org/D19095 llvm-svn: 266654
2016-04-19 03:01:28 +08:00
if (isa<UnreachableInst>(TI) ||
// If this block is terminated by a call to
// @llvm.experimental.deoptimize then treat it like an unreachable since
// the @llvm.experimental.deoptimize call is expected to practically
// never execute.
BB->getTerminatingDeoptimizeCall())
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 20:01:08 +08:00
PostDominatedByUnreachable.insert(BB);
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 13:42:14 +08:00
return;
}
// If the terminator is an InvokeInst, check only the normal destination block
// as the unwind edge of InvokeInst is also very unlikely taken.
if (auto *II = dyn_cast<InvokeInst>(TI)) {
if (PostDominatedByUnreachable.count(II->getNormalDest()))
PostDominatedByUnreachable.insert(BB);
return;
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 20:01:08 +08:00
}
Simplify the design of BranchProbabilityInfo by collapsing it into a single class. Previously it was split between two classes, one internal and one external. The concern seemed to center around exposing the weights used, but those can remain confined to the implementation file. Having a single class to maintain the state and analyses in use will also simplify several of the enhancements I want to make to our static heuristics. llvm-svn: 142783
2011-10-24 09:40:45 +08:00
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 13:42:14 +08:00
for (auto *I : successors(BB))
// If any of successor is not post dominated then BB is also not.
if (!PostDominatedByUnreachable.count(I))
return;
PostDominatedByUnreachable.insert(BB);
}
/// \brief Add \p BB to PostDominatedByColdCall set if applicable.
void
BranchProbabilityInfo::updatePostDominatedByColdCall(const BasicBlock *BB) {
assert(!PostDominatedByColdCall.count(BB));
const TerminatorInst *TI = BB->getTerminator();
if (TI->getNumSuccessors() == 0)
return;
// If all of successor are post dominated then BB is also done.
if (llvm::all_of(successors(BB), [&](const BasicBlock *SuccBB) {
return PostDominatedByColdCall.count(SuccBB);
})) {
PostDominatedByColdCall.insert(BB);
return;
}
// If the terminator is an InvokeInst, check only the normal destination
// block as the unwind edge of InvokeInst is also very unlikely taken.
if (auto *II = dyn_cast<InvokeInst>(TI))
if (PostDominatedByColdCall.count(II->getNormalDest())) {
PostDominatedByColdCall.insert(BB);
return;
}
// Otherwise, if the block itself contains a cold function, add it to the
// set of blocks post-dominated by a cold call.
for (auto &I : *BB)
if (const CallInst *CI = dyn_cast<CallInst>(&I))
if (CI->hasFnAttr(Attribute::Cold)) {
PostDominatedByColdCall.insert(BB);
return;
}
}
/// \brief Calculate edge weights for successors lead to unreachable.
///
/// Predict that a successor which leads necessarily to an
/// unreachable-terminated block as extremely unlikely.
bool BranchProbabilityInfo::calcUnreachableHeuristics(const BasicBlock *BB) {
const TerminatorInst *TI = BB->getTerminator();
[BPI] NFC: reorder ifs to bail out earlier This is non-functional change to re-order if statements to bail out earlier from unreachable and ColdCall heuristics. Reviewers: sanjoy, reames, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31704 llvm-svn: 300442
2017-04-17 14:39:47 +08:00
assert(TI->getNumSuccessors() > 1 && "expected more than one successor!");
// Return false here so that edge weights for InvokeInst could be decided
// in calcInvokeHeuristics().
if (isa<InvokeInst>(TI))
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 13:42:14 +08:00
return false;
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
SmallVector<unsigned, 4> UnreachableEdges;
SmallVector<unsigned, 4> ReachableEdges;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 13:42:14 +08:00
for (succ_const_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I)
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 20:01:08 +08:00
if (PostDominatedByUnreachable.count(*I))
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
UnreachableEdges.push_back(I.getSuccessorIndex());
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 20:01:08 +08:00
else
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
ReachableEdges.push_back(I.getSuccessorIndex());
Simplify the design of BranchProbabilityInfo by collapsing it into a single class. Previously it was split between two classes, one internal and one external. The concern seemed to center around exposing the weights used, but those can remain confined to the implementation file. Having a single class to maintain the state and analyses in use will also simplify several of the enhancements I want to make to our static heuristics. llvm-svn: 142783
2011-10-24 09:40:45 +08:00
[BPI] NFC: reorder ifs to bail out earlier This is non-functional change to re-order if statements to bail out earlier from unreachable and ColdCall heuristics. Reviewers: sanjoy, reames, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31704 llvm-svn: 300442
2017-04-17 14:39:47 +08:00
// Skip probabilities if all were reachable.
if (UnreachableEdges.empty())
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 13:42:14 +08:00
return false;
Enhance BranchProbabilityInfo::calcUnreachableHeuristics for InvokeInst This is recommit of r256028 with minor fixes in unittests: CodeGen/Mips/eh.ll CodeGen/Mips/insn-zero-size-bb.ll Original commit message: When identifying blocks post-dominated by an unreachable-terminated block in BranchProbabilityInfo, consider only the edge to the normal destination block if the terminator is InvokeInst and let calcInvokeHeuristics() decide edge weights for the InvokeInst. llvm-svn: 256202
2015-12-22 06:00:51 +08:00
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
if (ReachableEdges.empty()) {
BranchProbability Prob(1, UnreachableEdges.size());
for (unsigned SuccIdx : UnreachableEdges)
setEdgeProbability(BB, SuccIdx, Prob);
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 20:01:08 +08:00
return true;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
}
[BPI] Reduce the probability of unreachable edge to minimal value greater than 0 The probability of edge coming to unreachable block should be as low as possible. The change reduces the probability to minimal value greater than zero. The bug https://bugs.llvm.org/show_bug.cgi?id=32214 show the example when the probability of edge coming to unreachable block is greater than for edge coming to out of the loop and it causes incorrect loop rotation. Please note that with this change the behavior of unreachable heuristic is a bit different than others. Specifically, before this change the sum of probabilities coming to unreachable blocks have the same weight for all branches (it was just split over all edges of this block coming to unreachable blocks). With this change it might be slightly different but not to much due to probability of taken branch to unreachable block is really small. Reviewers: chandlerc, sanjoy, vsk, congh, junbuml, davidxl, dexonsmith Reviewed By: chandlerc, dexonsmith Subscribers: reames, llvm-commits Differential Revision: https://reviews.llvm.org/D30633 llvm-svn: 303327
2017-05-18 14:11:56 +08:00
auto UnreachableProb = UR_TAKEN_PROB;
auto ReachableProb =
(BranchProbability::getOne() - UR_TAKEN_PROB * UnreachableEdges.size()) /
ReachableEdges.size();
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
for (unsigned SuccIdx : UnreachableEdges)
setEdgeProbability(BB, SuccIdx, UnreachableProb);
for (unsigned SuccIdx : ReachableEdges)
setEdgeProbability(BB, SuccIdx, ReachableProb);
Remove return heuristics from the static branch probabilities, and introduce no-return or unreachable heuristics. The return heuristics from the Ball and Larus paper don't work well in practice as they pessimize early return paths. The only good hitrate return heuristics are those for: - NULL return - Constant return - negative integer return Only the last of these three can possibly require significant code for the returning block, and even the last is fairly rare and usually also a constant. As a consequence, even for the cold return paths, there is little code on that return path, and so little code density to be gained by sinking it. The places where sinking these blocks is valuable (inner loops) will already be weighted appropriately as the edge is a loop-exit branch. All of this aside, early returns are nearly as common as all three of these return categories, and should actually be predicted as taken! Rather than muddy the waters of the static predictions, just remain silent on returns and let the CFG itself dictate any layout or other issues. However, the return heuristic was flagging one very important case: unreachable. Unfortunately it still gave a 1/4 chance of the branch-to-unreachable occuring. It also didn't do a rigorous job of finding those blocks which post-dominate an unreachable block. This patch builds a more powerful analysis that should flag all branches to blocks known to then reach unreachable. It also has better worst-case runtime complexity by not looping through successors for each block. The previous code would perform an N^2 walk in the event of a single entry block branching to N successors with a switch where each successor falls through to the next and they finally fall through to a return. Test case added for noreturn heuristics. Also doxygen comments improved along the way. llvm-svn: 142793
2011-10-24 20:01:08 +08:00
return true;
}
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
Teach the BranchProbabilityInfo analysis pass to read any metadata encoding of probabilities. In the absense of metadata, it continues to fall back on static heuristics. This allows __builtin_expect, after lowering through llvm.expect a branch instruction's metadata, to actually enter the branch probability model. This is one component of resolving PR2577. llvm-svn: 142492
2011-10-19 18:30:30 +08:00
// Propagate existing explicit probabilities from either profile data or
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 12:33:04 +08:00
// 'expect' intrinsic processing. Examine metadata against unreachable
// heuristic. The probability of the edge coming to unreachable block is
// set to min of metadata and unreachable heuristic.
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-08 05:59:28 +08:00
bool BranchProbabilityInfo::calcMetadataWeights(const BasicBlock *BB) {
const TerminatorInst *TI = BB->getTerminator();
[BPI] NFC: reorder ifs to bail out earlier This is non-functional change to re-order if statements to bail out earlier from unreachable and ColdCall heuristics. Reviewers: sanjoy, reames, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31704 llvm-svn: 300442
2017-04-17 14:39:47 +08:00
assert(TI->getNumSuccessors() > 1 && "expected more than one successor!");
[PGO] Set edge weights for indirectbr instruction with profile counts Current PGO only annotates the edge weight for branch and switch instructions with profile counts. We should also annotate the indirectbr instruction as all the information is there. This patch enables the annotating for indirectbr instructions. Also uses this annotation in branch probability analysis. Differential Revision: https://reviews.llvm.org/D37074 llvm-svn: 311604
2017-08-24 05:36:02 +08:00
if (!(isa<BranchInst>(TI) || isa<SwitchInst>(TI) || isa<IndirectBrInst>(TI)))
Teach the BranchProbabilityInfo analysis pass to read any metadata encoding of probabilities. In the absense of metadata, it continues to fall back on static heuristics. This allows __builtin_expect, after lowering through llvm.expect a branch instruction's metadata, to actually enter the branch probability model. This is one component of resolving PR2577. llvm-svn: 142492
2011-10-19 18:30:30 +08:00
return false;
Revert "IR: MDNode => Value" Instead, we're going to separate metadata from the Value hierarchy. See PR21532. This reverts commit r221375. This reverts commit r221373. This reverts commit r221359. This reverts commit r221167. This reverts commit r221027. This reverts commit r221024. This reverts commit r221023. This reverts commit r220995. This reverts commit r220994. llvm-svn: 221711
2014-11-12 05:30:22 +08:00
MDNode *WeightsNode = TI->getMetadata(LLVMContext::MD_prof);
Generalize the reading of probability metadata to work for both branches and switches, with arbitrary numbers of successors. Still optimized for the common case of 2 successors for a conditional branch. Add a test case for switch metadata showing up in the BlockFrequencyInfo pass. llvm-svn: 142493
2011-10-19 18:32:19 +08:00
if (!WeightsNode)
Teach the BranchProbabilityInfo analysis pass to read any metadata encoding of probabilities. In the absense of metadata, it continues to fall back on static heuristics. This allows __builtin_expect, after lowering through llvm.expect a branch instruction's metadata, to actually enter the branch probability model. This is one component of resolving PR2577. llvm-svn: 142492
2011-10-19 18:30:30 +08:00
return false;
Fix information loss in branch probability computation. Summary: This addresses PR 22718. When branch weights are too large, they were being clamped to the range [1, MaxWeightForBB]. But this clamping is only applied to edges that go outside the range, so it distorts the relative branch probabilities. This patch changes the weight calculation to scale every branch so the relative probabilities are preserved. The scaling is done differently now. First, all the branch weights are added up, and if the sum exceeds 32 bits, it computes an integer scale to bring all the weights within the range. The patch fixes an existing test that had slightly wrong branch probabilities due to the previous clamping. It now gets branch weights scaled accordingly. Reviewers: dexonsmith Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9442 llvm-svn: 236750
2015-05-08 01:22:06 +08:00
// Check that the number of successors is manageable.
assert(TI->getNumSuccessors() < UINT32_MAX && "Too many successors");
Generalize the reading of probability metadata to work for both branches and switches, with arbitrary numbers of successors. Still optimized for the common case of 2 successors for a conditional branch. Add a test case for switch metadata showing up in the BlockFrequencyInfo pass. llvm-svn: 142493
2011-10-19 18:32:19 +08:00
// Ensure there are weights for all of the successors. Note that the first
// operand to the metadata node is a name, not a weight.
if (WeightsNode->getNumOperands() != TI->getNumSuccessors() + 1)
Teach the BranchProbabilityInfo analysis pass to read any metadata encoding of probabilities. In the absense of metadata, it continues to fall back on static heuristics. This allows __builtin_expect, after lowering through llvm.expect a branch instruction's metadata, to actually enter the branch probability model. This is one component of resolving PR2577. llvm-svn: 142492
2011-10-19 18:30:30 +08:00
return false;
Fix information loss in branch probability computation. Summary: This addresses PR 22718. When branch weights are too large, they were being clamped to the range [1, MaxWeightForBB]. But this clamping is only applied to edges that go outside the range, so it distorts the relative branch probabilities. This patch changes the weight calculation to scale every branch so the relative probabilities are preserved. The scaling is done differently now. First, all the branch weights are added up, and if the sum exceeds 32 bits, it computes an integer scale to bring all the weights within the range. The patch fixes an existing test that had slightly wrong branch probabilities due to the previous clamping. It now gets branch weights scaled accordingly. Reviewers: dexonsmith Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9442 llvm-svn: 236750
2015-05-08 01:22:06 +08:00
// Build up the final weights that will be used in a temporary buffer.
// Compute the sum of all weights to later decide whether they need to
// be scaled to fit in 32 bits.
uint64_t WeightSum = 0;
Generalize the reading of probability metadata to work for both branches and switches, with arbitrary numbers of successors. Still optimized for the common case of 2 successors for a conditional branch. Add a test case for switch metadata showing up in the BlockFrequencyInfo pass. llvm-svn: 142493
2011-10-19 18:32:19 +08:00
SmallVector<uint32_t, 2> Weights;
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 12:33:04 +08:00
SmallVector<unsigned, 2> UnreachableIdxs;
SmallVector<unsigned, 2> ReachableIdxs;
Generalize the reading of probability metadata to work for both branches and switches, with arbitrary numbers of successors. Still optimized for the common case of 2 successors for a conditional branch. Add a test case for switch metadata showing up in the BlockFrequencyInfo pass. llvm-svn: 142493
2011-10-19 18:32:19 +08:00
Weights.reserve(TI->getNumSuccessors());
for (unsigned i = 1, e = WeightsNode->getNumOperands(); i != e; ++i) {
IR: Split Metadata from Value Split `Metadata` away from the `Value` class hierarchy, as part of PR21532. Assembly and bitcode changes are in the wings, but this is the bulk of the change for the IR C++ API. I have a follow-up patch prepared for `clang`. If this breaks other sub-projects, I apologize in advance :(. Help me compile it on Darwin I'll try to fix it. FWIW, the errors should be easy to fix, so it may be simpler to just fix it yourself. This breaks the build for all metadata-related code that's out-of-tree. Rest assured the transition is mechanical and the compiler should catch almost all of the problems. Here's a quick guide for updating your code: - `Metadata` is the root of a class hierarchy with three main classes: `MDNode`, `MDString`, and `ValueAsMetadata`. It is distinct from the `Value` class hierarchy. It is typeless -- i.e., instances do *not* have a `Type`. - `MDNode`'s operands are all `Metadata *` (instead of `Value *`). - `TrackingVH<MDNode>` and `WeakVH` referring to metadata can be replaced with `TrackingMDNodeRef` and `TrackingMDRef`, respectively. If you're referring solely to resolved `MDNode`s -- post graph construction -- just use `MDNode*`. - `MDNode` (and the rest of `Metadata`) have only limited support for `replaceAllUsesWith()`. As long as an `MDNode` is pointing at a forward declaration -- the result of `MDNode::getTemporary()` -- it maintains a side map of its uses and can RAUW itself. Once the forward declarations are fully resolved RAUW support is dropped on the ground. This means that uniquing collisions on changing operands cause nodes to become "distinct". (This already happened fairly commonly, whenever an operand went to null.) If you're constructing complex (non self-reference) `MDNode` cycles, you need to call `MDNode::resolveCycles()` on each node (or on a top-level node that somehow references all of the nodes). Also, don't do that. Metadata cycles (and the RAUW machinery needed to construct them) are expensive. - An `MDNode` can only refer to a `Constant` through a bridge called `ConstantAsMetadata` (one of the subclasses of `ValueAsMetadata`). As a side effect, accessing an operand of an `MDNode` that is known to be, e.g., `ConstantInt`, takes three steps: first, cast from `Metadata` to `ConstantAsMetadata`; second, extract the `Constant`; third, cast down to `ConstantInt`. The eventual goal is to introduce `MDInt`/`MDFloat`/etc. and have metadata schema owners transition away from using `Constant`s when the type isn't important (and they don't care about referring to `GlobalValue`s). In the meantime, I've added transitional API to the `mdconst` namespace that matches semantics with the old code, in order to avoid adding the error-prone three-step equivalent to every call site. If your old code was: MDNode *N = foo(); bar(isa <ConstantInt>(N->getOperand(0))); baz(cast <ConstantInt>(N->getOperand(1))); bak(cast_or_null <ConstantInt>(N->getOperand(2))); bat(dyn_cast <ConstantInt>(N->getOperand(3))); bay(dyn_cast_or_null<ConstantInt>(N->getOperand(4))); you can trivially match its semantics with: MDNode *N = foo(); bar(mdconst::hasa <ConstantInt>(N->getOperand(0))); baz(mdconst::extract <ConstantInt>(N->getOperand(1))); bak(mdconst::extract_or_null <ConstantInt>(N->getOperand(2))); bat(mdconst::dyn_extract <ConstantInt>(N->getOperand(3))); bay(mdconst::dyn_extract_or_null<ConstantInt>(N->getOperand(4))); and when you transition your metadata schema to `MDInt`: MDNode *N = foo(); bar(isa <MDInt>(N->getOperand(0))); baz(cast <MDInt>(N->getOperand(1))); bak(cast_or_null <MDInt>(N->getOperand(2))); bat(dyn_cast <MDInt>(N->getOperand(3))); bay(dyn_cast_or_null<MDInt>(N->getOperand(4))); - A `CallInst` -- specifically, intrinsic instructions -- can refer to metadata through a bridge called `MetadataAsValue`. This is a subclass of `Value` where `getType()->isMetadataTy()`. `MetadataAsValue` is the *only* class that can legally refer to a `LocalAsMetadata`, which is a bridged form of non-`Constant` values like `Argument` and `Instruction`. It can also refer to any other `Metadata` subclass. (I'll break all your testcases in a follow-up commit, when I propagate this change to assembly.) llvm-svn: 223802
2014-12-10 02:38:53 +08:00
ConstantInt *Weight =
mdconst::dyn_extract<ConstantInt>(WeightsNode->getOperand(i));
Generalize the reading of probability metadata to work for both branches and switches, with arbitrary numbers of successors. Still optimized for the common case of 2 successors for a conditional branch. Add a test case for switch metadata showing up in the BlockFrequencyInfo pass. llvm-svn: 142493
2011-10-19 18:32:19 +08:00
if (!Weight)
return false;
Fix information loss in branch probability computation. Summary: This addresses PR 22718. When branch weights are too large, they were being clamped to the range [1, MaxWeightForBB]. But this clamping is only applied to edges that go outside the range, so it distorts the relative branch probabilities. This patch changes the weight calculation to scale every branch so the relative probabilities are preserved. The scaling is done differently now. First, all the branch weights are added up, and if the sum exceeds 32 bits, it computes an integer scale to bring all the weights within the range. The patch fixes an existing test that had slightly wrong branch probabilities due to the previous clamping. It now gets branch weights scaled accordingly. Reviewers: dexonsmith Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9442 llvm-svn: 236750
2015-05-08 01:22:06 +08:00
assert(Weight->getValue().getActiveBits() <= 32 &&
"Too many bits for uint32_t");
Weights.push_back(Weight->getZExtValue());
WeightSum += Weights.back();
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 12:33:04 +08:00
if (PostDominatedByUnreachable.count(TI->getSuccessor(i - 1)))
UnreachableIdxs.push_back(i - 1);
else
ReachableIdxs.push_back(i - 1);
Generalize the reading of probability metadata to work for both branches and switches, with arbitrary numbers of successors. Still optimized for the common case of 2 successors for a conditional branch. Add a test case for switch metadata showing up in the BlockFrequencyInfo pass. llvm-svn: 142493
2011-10-19 18:32:19 +08:00
}
assert(Weights.size() == TI->getNumSuccessors() && "Checked above");
Fix information loss in branch probability computation. Summary: This addresses PR 22718. When branch weights are too large, they were being clamped to the range [1, MaxWeightForBB]. But this clamping is only applied to edges that go outside the range, so it distorts the relative branch probabilities. This patch changes the weight calculation to scale every branch so the relative probabilities are preserved. The scaling is done differently now. First, all the branch weights are added up, and if the sum exceeds 32 bits, it computes an integer scale to bring all the weights within the range. The patch fixes an existing test that had slightly wrong branch probabilities due to the previous clamping. It now gets branch weights scaled accordingly. Reviewers: dexonsmith Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9442 llvm-svn: 236750
2015-05-08 01:22:06 +08:00
// If the sum of weights does not fit in 32 bits, scale every weight down
// accordingly.
uint64_t ScalingFactor =
(WeightSum > UINT32_MAX) ? WeightSum / UINT32_MAX + 1 : 1;
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 12:33:04 +08:00
if (ScalingFactor > 1) {
WeightSum = 0;
for (unsigned i = 0, e = TI->getNumSuccessors(); i != e; ++i) {
Weights[i] /= ScalingFactor;
WeightSum += Weights[i];
}
Fix information loss in branch probability computation. Summary: This addresses PR 22718. When branch weights are too large, they were being clamped to the range [1, MaxWeightForBB]. But this clamping is only applied to edges that go outside the range, so it distorts the relative branch probabilities. This patch changes the weight calculation to scale every branch so the relative probabilities are preserved. The scaling is done differently now. First, all the branch weights are added up, and if the sum exceeds 32 bits, it computes an integer scale to bring all the weights within the range. The patch fixes an existing test that had slightly wrong branch probabilities due to the previous clamping. It now gets branch weights scaled accordingly. Reviewers: dexonsmith Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9442 llvm-svn: 236750
2015-05-08 01:22:06 +08:00
}
[BPI] Ignore remainder while distributing the remaining probability from unreachanble This is a follow up patch for https://reviews.llvm.org/rL300440 to address a comment. To make implementation to be consistent with other cases we just ignore the remainder after distribution of remaining probability between reachable edges. If we reduced the probability of some edges coming to unreachable blocks we should distribute the remaining part across other edges coming to reachable blocks to satisfy the condition that sum of all probabilities should be equal to one. If this remaining part is not divided by number of "reachable" edges then we get this remainder. This remainder probability should be pretty small. Other cases just ignore if the sum of probabilities is not equal to one so we do the same. Reviewers: chandlerc, sanjoy, vsk, junbuml, reames Reviewed By: reames Subscribers: reames, llvm-commits Differential Revision: https://reviews.llvm.org/D32124 llvm-svn: 302883
2017-05-12 15:50:06 +08:00
assert(WeightSum <= UINT32_MAX &&
"Expected weights to scale down to 32 bits");
[BPI] Fix two potential divide-by-zero operations that are introduced in r256263. llvm-svn: 256303
2015-12-23 07:45:55 +08:00
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 12:33:04 +08:00
if (WeightSum == 0 || ReachableIdxs.size() == 0) {
[BPI] Fix two potential divide-by-zero operations that are introduced in r256263. llvm-svn: 256303
2015-12-23 07:45:55 +08:00
for (unsigned i = 0, e = TI->getNumSuccessors(); i != e; ++i)
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 12:33:04 +08:00
Weights[i] = 1;
WeightSum = TI->getNumSuccessors();
[BPI] Fix two potential divide-by-zero operations that are introduced in r256263. llvm-svn: 256303
2015-12-23 07:45:55 +08:00
}
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 12:33:04 +08:00
// Set the probability.
SmallVector<BranchProbability, 2> BP;
for (unsigned i = 0, e = TI->getNumSuccessors(); i != e; ++i)
BP.push_back({ Weights[i], static_cast<uint32_t>(WeightSum) });
// Examine the metadata against unreachable heuristic.
// If the unreachable heuristic is more strong then we use it for this edge.
if (UnreachableIdxs.size() > 0 && ReachableIdxs.size() > 0) {
auto ToDistribute = BranchProbability::getZero();
[BPI] Reduce the probability of unreachable edge to minimal value greater than 0 The probability of edge coming to unreachable block should be as low as possible. The change reduces the probability to minimal value greater than zero. The bug https://bugs.llvm.org/show_bug.cgi?id=32214 show the example when the probability of edge coming to unreachable block is greater than for edge coming to out of the loop and it causes incorrect loop rotation. Please note that with this change the behavior of unreachable heuristic is a bit different than others. Specifically, before this change the sum of probabilities coming to unreachable blocks have the same weight for all branches (it was just split over all edges of this block coming to unreachable blocks). With this change it might be slightly different but not to much due to probability of taken branch to unreachable block is really small. Reviewers: chandlerc, sanjoy, vsk, congh, junbuml, davidxl, dexonsmith Reviewed By: chandlerc, dexonsmith Subscribers: reames, llvm-commits Differential Revision: https://reviews.llvm.org/D30633 llvm-svn: 303327
2017-05-18 14:11:56 +08:00
auto UnreachableProb = UR_TAKEN_PROB;
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 12:33:04 +08:00
for (auto i : UnreachableIdxs)
if (UnreachableProb < BP[i]) {
ToDistribute += BP[i] - UnreachableProb;
BP[i] = UnreachableProb;
}
// If we modified the probability of some edges then we must distribute
// the difference between reachable blocks.
if (ToDistribute > BranchProbability::getZero()) {
BranchProbability PerEdge = ToDistribute / ReachableIdxs.size();
[BPI] Ignore remainder while distributing the remaining probability from unreachanble This is a follow up patch for https://reviews.llvm.org/rL300440 to address a comment. To make implementation to be consistent with other cases we just ignore the remainder after distribution of remaining probability between reachable edges. If we reduced the probability of some edges coming to unreachable blocks we should distribute the remaining part across other edges coming to reachable blocks to satisfy the condition that sum of all probabilities should be equal to one. If this remaining part is not divided by number of "reachable" edges then we get this remainder. This remainder probability should be pretty small. Other cases just ignore if the sum of probabilities is not equal to one so we do the same. Reviewers: chandlerc, sanjoy, vsk, junbuml, reames Reviewed By: reames Subscribers: reames, llvm-commits Differential Revision: https://reviews.llvm.org/D32124 llvm-svn: 302883
2017-05-12 15:50:06 +08:00
for (auto i : ReachableIdxs)
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 12:33:04 +08:00
BP[i] += PerEdge;
}
}
for (unsigned i = 0, e = TI->getNumSuccessors(); i != e; ++i)
setEdgeProbability(BB, i, BP[i]);
Teach the BranchProbabilityInfo analysis pass to read any metadata encoding of probabilities. In the absense of metadata, it continues to fall back on static heuristics. This allows __builtin_expect, after lowering through llvm.expect a branch instruction's metadata, to actually enter the branch probability model. This is one component of resolving PR2577. llvm-svn: 142492
2011-10-19 18:30:30 +08:00
return true;
}
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 20:26:52 +08:00
/// \brief Calculate edge weights for edges leading to cold blocks.
///
/// A cold block is one post-dominated by a block with a call to a
/// cold function. Those edges are unlikely to be taken, so we give
/// them relatively low weight.
///
/// Return true if we could compute the weights for cold edges.
/// Return false, otherwise.
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-08 05:59:28 +08:00
bool BranchProbabilityInfo::calcColdCallHeuristics(const BasicBlock *BB) {
const TerminatorInst *TI = BB->getTerminator();
[BPI] NFC: reorder ifs to bail out earlier This is non-functional change to re-order if statements to bail out earlier from unreachable and ColdCall heuristics. Reviewers: sanjoy, reames, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31704 llvm-svn: 300442
2017-04-17 14:39:47 +08:00
assert(TI->getNumSuccessors() > 1 && "expected more than one successor!");
// Return false here so that edge weights for InvokeInst could be decided
// in calcInvokeHeuristics().
if (isa<InvokeInst>(TI))
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 20:26:52 +08:00
return false;
// Determine which successors are post-dominated by a cold block.
SmallVector<unsigned, 4> ColdEdges;
SmallVector<unsigned, 4> NormalEdges;
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-08 05:59:28 +08:00
for (succ_const_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I)
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 20:26:52 +08:00
if (PostDominatedByColdCall.count(*I))
ColdEdges.push_back(I.getSuccessorIndex());
else
NormalEdges.push_back(I.getSuccessorIndex());
[BPI] NFC: reorder ifs to bail out earlier This is non-functional change to re-order if statements to bail out earlier from unreachable and ColdCall heuristics. Reviewers: sanjoy, reames, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31704 llvm-svn: 300442
2017-04-17 14:39:47 +08:00
// Skip probabilities if no cold edges.
if (ColdEdges.empty())
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 20:26:52 +08:00
return false;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
if (NormalEdges.empty()) {
BranchProbability Prob(1, ColdEdges.size());
for (unsigned SuccIdx : ColdEdges)
setEdgeProbability(BB, SuccIdx, Prob);
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 20:26:52 +08:00
return true;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
}
Retry: [BPI] Use a safer constructor to calculate branch probabilities BPI may trigger signed overflow UB while computing branch probabilities for cold calls or to unreachables. For example, with our current choice of weights, we'll crash if there are >= 2^12 branches to an unreachable. Use a safer BranchProbability constructor which is better at handling fractions with large denominators. Changes since the initial commit: - Use explicit casts to ensure that multiplication operands are 64-bit ints. rdar://problem/29368161 Differential Revision: https://reviews.llvm.org/D27862 llvm-svn: 290022
2016-12-17 09:02:08 +08:00
auto ColdProb = BranchProbability::getBranchProbability(
CC_TAKEN_WEIGHT,
(CC_TAKEN_WEIGHT + CC_NONTAKEN_WEIGHT) * uint64_t(ColdEdges.size()));
auto NormalProb = BranchProbability::getBranchProbability(
CC_NONTAKEN_WEIGHT,
(CC_TAKEN_WEIGHT + CC_NONTAKEN_WEIGHT) * uint64_t(NormalEdges.size()));
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
for (unsigned SuccIdx : ColdEdges)
setEdgeProbability(BB, SuccIdx, ColdProb);
for (unsigned SuccIdx : NormalEdges)
setEdgeProbability(BB, SuccIdx, NormalProb);
Add a new function attribute 'cold' to functions. Other than recognizing the attribute, the patch does little else. It changes the branch probability analyzer so that edges into blocks postdominated by a cold function are given low weight. Added analysis and code generation tests. Added documentation for the new attribute. llvm-svn: 182638
2013-05-24 20:26:52 +08:00
return true;
}
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
// Calculate Edge Weights using "Pointer Heuristics". Predict a comparsion
// between two pointer or pointer and NULL will fail.
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-08 05:59:28 +08:00
bool BranchProbabilityInfo::calcPointerHeuristics(const BasicBlock *BB) {
const BranchInst *BI = dyn_cast<BranchInst>(BB->getTerminator());
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
if (!BI || !BI->isConditional())
Heuristics are in descending priority now. If we use one of them, skip the rest. llvm-svn: 136402
2011-07-29 05:45:07 +08:00
return false;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
Value *Cond = BI->getCondition();
ICmpInst *CI = dyn_cast<ICmpInst>(Cond);
Fix pointer heuristic. Check whether predicator is ICMP_NE instead of if it is not isEquality(). llvm-svn: 135296
2011-07-16 04:51:06 +08:00
if (!CI || !CI->isEquality())
Heuristics are in descending priority now. If we use one of them, skip the rest. llvm-svn: 136402
2011-07-29 05:45:07 +08:00
return false;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
Value *LHS = CI->getOperand(0);
if (!LHS->getType()->isPointerTy())
Heuristics are in descending priority now. If we use one of them, skip the rest. llvm-svn: 136402
2011-07-29 05:45:07 +08:00
return false;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
Fold assert-only-used variable into the assert. llvm-svn: 132620
2011-06-04 10:07:10 +08:00
assert(CI->getOperand(1)->getType()->isPointerTy());
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
// p != 0 -> isProb = true
// p == 0 -> isProb = false
// p != q -> isProb = true
// p == q -> isProb = false;
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
unsigned TakenIdx = 0, NonTakenIdx = 1;
Fix pointer heuristic. Check whether predicator is ICMP_NE instead of if it is not isEquality(). llvm-svn: 135296
2011-07-16 04:51:06 +08:00
bool isProb = CI->getPredicate() == ICmpInst::ICMP_NE;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
if (!isProb)
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
std::swap(TakenIdx, NonTakenIdx);
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
BranchProbability TakenProb(PH_TAKEN_WEIGHT,
PH_TAKEN_WEIGHT + PH_NONTAKEN_WEIGHT);
setEdgeProbability(BB, TakenIdx, TakenProb);
setEdgeProbability(BB, NonTakenIdx, TakenProb.getCompl());
Heuristics are in descending priority now. If we use one of them, skip the rest. llvm-svn: 136402
2011-07-29 05:45:07 +08:00
return true;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
}
// Calculate Edge Weights using "Loop Branch Heuristics". Predict backedges
// as taken, exiting edges as not-taken.
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-08 05:59:28 +08:00
bool BranchProbabilityInfo::calcLoopBranchHeuristics(const BasicBlock *BB,
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 06:48:29 +08:00
const LoopInfo &LI) {
Loop *L = LI.getLoopFor(BB);
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
if (!L)
Heuristics are in descending priority now. If we use one of them, skip the rest. llvm-svn: 136402
2011-07-29 05:45:07 +08:00
return false;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
SmallVector<unsigned, 8> BackEdges;
SmallVector<unsigned, 8> ExitingEdges;
SmallVector<unsigned, 8> InEdges; // Edges from header to the loop.
Add InEdges (edges from header to the loop) in Loop Branch Heuristics, so there is no frequency difference whether condition is in the header or in the latch. llvm-svn: 136398
2011-07-29 05:33:46 +08:00
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-08 05:59:28 +08:00
for (succ_const_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I) {
Fix the API usage in loop probability heuristics. It was incorrectly classifying many edges as exiting which were in fact not. These mainly formed edges into sub-loops. It was also not correctly classifying all returning edges out of loops as leaving the loop. With this match most of the loop heuristics are more rational. Several serious regressions on loop-intesive benchmarks like perlbench's loop tests when built with -enable-block-placement are fixed by these updated heuristics. Unfortunately they in turn uncover some other regressions. There are still several improvemenst that should be made to loop heuristics including trip-count, and early back-edge management. llvm-svn: 142917
2011-10-25 17:47:41 +08:00
if (!L->contains(*I))
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
ExitingEdges.push_back(I.getSuccessorIndex());
Fix the API usage in loop probability heuristics. It was incorrectly classifying many edges as exiting which were in fact not. These mainly formed edges into sub-loops. It was also not correctly classifying all returning edges out of loops as leaving the loop. With this match most of the loop heuristics are more rational. Several serious regressions on loop-intesive benchmarks like perlbench's loop tests when built with -enable-block-placement are fixed by these updated heuristics. Unfortunately they in turn uncover some other regressions. There are still several improvemenst that should be made to loop heuristics including trip-count, and early back-edge management. llvm-svn: 142917
2011-10-25 17:47:41 +08:00
else if (L->getHeader() == *I)
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
BackEdges.push_back(I.getSuccessorIndex());
Fix the API usage in loop probability heuristics. It was incorrectly classifying many edges as exiting which were in fact not. These mainly formed edges into sub-loops. It was also not correctly classifying all returning edges out of loops as leaving the loop. With this match most of the loop heuristics are more rational. Several serious regressions on loop-intesive benchmarks like perlbench's loop tests when built with -enable-block-placement are fixed by these updated heuristics. Unfortunately they in turn uncover some other regressions. There are still several improvemenst that should be made to loop heuristics including trip-count, and early back-edge management. llvm-svn: 142917
2011-10-25 17:47:41 +08:00
else
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
InEdges.push_back(I.getSuccessorIndex());
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
}
Fix a bug in which BranchProbabilityInfo wasn't setting branch weights of basic blocks inside loops correctly. Previously, BranchProbabilityInfo::calcLoopBranchHeuristics would determine the weights of basic blocks inside loops even when it didn't have enough information to estimate the branch probabilities correctly. This patch fixes the function to exit early if it doesn't see any exit edges or back edges and let the later heuristics determine the weights. This fixes PR18705 and <rdar://problem/15991090>. Differential Revision: http://reviews.llvm.org/D3363 llvm-svn: 206194
2014-04-15 00:56:19 +08:00
if (BackEdges.empty() && ExitingEdges.empty())
return false;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
// Collect the sum of probabilities of back-edges/in-edges/exiting-edges, and
// normalize them so that they sum up to one.
[PPC] Strength-reduce SmallVectors into arrays. No functionality change intended. llvm-svn: 272999
2016-06-17 21:15:10 +08:00
BranchProbability Probs[] = {BranchProbability::getZero(),
BranchProbability::getZero(),
BranchProbability::getZero()};
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
unsigned Denom = (BackEdges.empty() ? 0 : LBH_TAKEN_WEIGHT) +
(InEdges.empty() ? 0 : LBH_TAKEN_WEIGHT) +
(ExitingEdges.empty() ? 0 : LBH_NONTAKEN_WEIGHT);
if (!BackEdges.empty())
Probs[0] = BranchProbability(LBH_TAKEN_WEIGHT, Denom);
if (!InEdges.empty())
Probs[1] = BranchProbability(LBH_TAKEN_WEIGHT, Denom);
if (!ExitingEdges.empty())
Probs[2] = BranchProbability(LBH_NONTAKEN_WEIGHT, Denom);
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
if (uint32_t numBackEdges = BackEdges.size()) {
auto Prob = Probs[0] / numBackEdges;
for (unsigned SuccIdx : BackEdges)
setEdgeProbability(BB, SuccIdx, Prob);
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
}
Add InEdges (edges from header to the loop) in Loop Branch Heuristics, so there is no frequency difference whether condition is in the header or in the latch. llvm-svn: 136398
2011-07-29 05:33:46 +08:00
if (uint32_t numInEdges = InEdges.size()) {
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
auto Prob = Probs[1] / numInEdges;
for (unsigned SuccIdx : InEdges)
setEdgeProbability(BB, SuccIdx, Prob);
Add InEdges (edges from header to the loop) in Loop Branch Heuristics, so there is no frequency difference whether condition is in the header or in the latch. llvm-svn: 136398
2011-07-29 05:33:46 +08:00
}
Fix the API usage in loop probability heuristics. It was incorrectly classifying many edges as exiting which were in fact not. These mainly formed edges into sub-loops. It was also not correctly classifying all returning edges out of loops as leaving the loop. With this match most of the loop heuristics are more rational. Several serious regressions on loop-intesive benchmarks like perlbench's loop tests when built with -enable-block-placement are fixed by these updated heuristics. Unfortunately they in turn uncover some other regressions. There are still several improvemenst that should be made to loop heuristics including trip-count, and early back-edge management. llvm-svn: 142917
2011-10-25 17:47:41 +08:00
if (uint32_t numExitingEdges = ExitingEdges.size()) {
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
auto Prob = Probs[2] / numExitingEdges;
for (unsigned SuccIdx : ExitingEdges)
setEdgeProbability(BB, SuccIdx, Prob);
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
}
Heuristics are in descending priority now. If we use one of them, skip the rest. llvm-svn: 136402
2011-07-29 05:45:07 +08:00
return true;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
}
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 17:44:40 +08:00
bool BranchProbabilityInfo::calcZeroHeuristics(const BasicBlock *BB,
const TargetLibraryInfo *TLI) {
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-08 05:59:28 +08:00
const BranchInst *BI = dyn_cast<BranchInst>(BB->getTerminator());
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 11:27:24 +08:00
if (!BI || !BI->isConditional())
return false;
Value *Cond = BI->getCondition();
ICmpInst *CI = dyn_cast<ICmpInst>(Cond);
if (!CI)
return false;
Value *RHS = CI->getOperand(1);
Do not handle case where LHS is equal to zero, because InstCombiner always moves it to RHS anyway. llvm-svn: 136586
2011-07-31 12:47:20 +08:00
ConstantInt *CV = dyn_cast<ConstantInt>(RHS);
Use canonical forms for the branch probability zero heutistic. - Drop support for X >u 0, it's equivalent to X != 0 and should be canonicalized into the latter. - Add X < 1 -> unlikely, which is what instcombine canonicalizes X <= 0 into. - Add X > -1 -> likely, which is what instcombine canonicalizes X >= 0 into. llvm-svn: 139110
2011-09-05 07:53:04 +08:00
if (!CV)
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 11:27:24 +08:00
return false;
Re-apply r234898 and fix tests. This commit makes LLVM not estimate branch probabilities when doing a single bit bitmask tests. The code that originally made me discover this is: if ((a & 0x1) == 0x1) { .. } In this case we don't actually have any branch probability information and should not assume to have any. LLVM transforms this into: %and = and i32 %a, 1 %tobool = icmp eq i32 %and, 0 So, in this case, the result of a bitwise and is compared against 0, but nevertheless, we should not assume to have probability information. CodeGen/ARM/2013-10-11-select-stalls.ll started failing because the changed probabilities changed the results of ARMBaseInstrInfo::isProfitableToIfCvt() and led to an Ifcvt of the diamond in the test. AFAICT, the test was never meant to test this and thus changing the test input slightly to not change the probabilities seems like the best way to preserve the meaning of the test. llvm-svn: 234979
2015-04-15 14:24:07 +08:00
// If the LHS is the result of AND'ing a value with a single bit bitmask,
// we don't have information about probabilities.
if (Instruction *LHS = dyn_cast<Instruction>(CI->getOperand(0)))
if (LHS->getOpcode() == Instruction::And)
if (ConstantInt *AndRHS = dyn_cast<ConstantInt>(LHS->getOperand(1)))
[ConstantInt] Use ConstantInt::getValue instead of Constant::getUniqueInteger in a few places where we obviously have a ConstantInt. NFC getUniqueInteger will ultimately call ConstantInt::getValue, but calling ConstantInt::getValue should be inlined. llvm-svn: 310069
2017-08-05 00:59:29 +08:00
if (AndRHS->getValue().isPowerOf2())
Re-apply r234898 and fix tests. This commit makes LLVM not estimate branch probabilities when doing a single bit bitmask tests. The code that originally made me discover this is: if ((a & 0x1) == 0x1) { .. } In this case we don't actually have any branch probability information and should not assume to have any. LLVM transforms this into: %and = and i32 %a, 1 %tobool = icmp eq i32 %and, 0 So, in this case, the result of a bitwise and is compared against 0, but nevertheless, we should not assume to have probability information. CodeGen/ARM/2013-10-11-select-stalls.ll started failing because the changed probabilities changed the results of ARMBaseInstrInfo::isProfitableToIfCvt() and led to an Ifcvt of the diamond in the test. AFAICT, the test was never meant to test this and thus changing the test input slightly to not change the probabilities seems like the best way to preserve the meaning of the test. llvm-svn: 234979
2015-04-15 14:24:07 +08:00
return false;
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 17:44:40 +08:00
// Check if the LHS is the return value of a library function
LibFunc Func = NumLibFuncs;
if (TLI)
if (CallInst *Call = dyn_cast<CallInst>(CI->getOperand(0)))
if (Function *CalledFn = Call->getCalledFunction())
TLI->getLibFunc(*CalledFn, Func);
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 11:27:24 +08:00
bool isProb;
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 17:44:40 +08:00
if (Func == LibFunc_strcasecmp ||
Func == LibFunc_strcmp ||
Func == LibFunc_strncasecmp ||
Func == LibFunc_strncmp ||
Func == LibFunc_memcmp) {
// strcmp and similar functions return zero, negative, or positive, if the
// first string is equal, less, or greater than the second. We consider it
// likely that the strings are not equal, so a comparison with zero is
// probably false, but also a comparison with any other number is also
// probably false given that what exactly is returned for nonzero values is
// not specified. Any kind of comparison other than equality we know
// nothing about.
switch (CI->getPredicate()) {
case CmpInst::ICMP_EQ:
isProb = false;
break;
case CmpInst::ICMP_NE:
isProb = true;
break;
default:
return false;
}
} else if (CV->isZero()) {
Use canonical forms for the branch probability zero heutistic. - Drop support for X >u 0, it's equivalent to X != 0 and should be canonicalized into the latter. - Add X < 1 -> unlikely, which is what instcombine canonicalizes X <= 0 into. - Add X > -1 -> likely, which is what instcombine canonicalizes X >= 0 into. llvm-svn: 139110
2011-09-05 07:53:04 +08:00
switch (CI->getPredicate()) {
case CmpInst::ICMP_EQ:
// X == 0 -> Unlikely
isProb = false;
break;
case CmpInst::ICMP_NE:
// X != 0 -> Likely
isProb = true;
break;
case CmpInst::ICMP_SLT:
// X < 0 -> Unlikely
isProb = false;
break;
case CmpInst::ICMP_SGT:
// X > 0 -> Likely
isProb = true;
break;
default:
return false;
}
} else if (CV->isOne() && CI->getPredicate() == CmpInst::ICMP_SLT) {
// InstCombine canonicalizes X <= 0 into X < 1.
// X <= 0 -> Unlikely
Do not handle case where LHS is equal to zero, because InstCombiner always moves it to RHS anyway. llvm-svn: 136586
2011-07-31 12:47:20 +08:00
isProb = false;
[Constants] If we already have a ConstantInt*, prefer to use isZero/isOne/isMinusOne instead of isNullValue/isOneValue/isAllOnesValue inherited from Constant. NFCI Going through the Constant methods requires redetermining that the Constant is a ConstantInt and then calling isZero/isOne/isMinusOne. llvm-svn: 307292
2017-07-07 02:39:47 +08:00
} else if (CV->isMinusOne()) {
Consider (x == -1) unlikely in BranchProbabilityInfo This adds another heuristic to BPI, similar to the existing heuristic that considers (x == 0) unlikely to be true. As suggested in the PACT'98 paper by Deitrich, Cheng, and Hwu, -1 is often used to indicate an invalid index, and equality comparisons with -1 are also unlikely to succeed. Local experimentation supports this hypothesis: This yields a 1-2% speedup in the test-suite sqlite benchmark on the PPC A2 core, with no significant regressions. llvm-svn: 193855
2013-11-01 18:58:22 +08:00
switch (CI->getPredicate()) {
case CmpInst::ICMP_EQ:
// X == -1 -> Unlikely
isProb = false;
break;
case CmpInst::ICMP_NE:
// X != -1 -> Likely
isProb = true;
break;
case CmpInst::ICMP_SGT:
// InstCombine canonicalizes X >= 0 into X > -1.
// X >= 0 -> Likely
isProb = true;
break;
default:
return false;
}
Use canonical forms for the branch probability zero heutistic. - Drop support for X >u 0, it's equivalent to X != 0 and should be canonicalized into the latter. - Add X < 1 -> unlikely, which is what instcombine canonicalizes X <= 0 into. - Add X > -1 -> likely, which is what instcombine canonicalizes X >= 0 into. llvm-svn: 139110
2011-09-05 07:53:04 +08:00
} else {
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 11:27:24 +08:00
return false;
Use canonical forms for the branch probability zero heutistic. - Drop support for X >u 0, it's equivalent to X != 0 and should be canonicalized into the latter. - Add X < 1 -> unlikely, which is what instcombine canonicalizes X <= 0 into. - Add X > -1 -> likely, which is what instcombine canonicalizes X >= 0 into. llvm-svn: 139110
2011-09-05 07:53:04 +08:00
}
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 11:27:24 +08:00
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
unsigned TakenIdx = 0, NonTakenIdx = 1;
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 11:27:24 +08:00
if (!isProb)
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
std::swap(TakenIdx, NonTakenIdx);
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 11:27:24 +08:00
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
BranchProbability TakenProb(ZH_TAKEN_WEIGHT,
ZH_TAKEN_WEIGHT + ZH_NONTAKEN_WEIGHT);
setEdgeProbability(BB, TakenIdx, TakenProb);
setEdgeProbability(BB, NonTakenIdx, TakenProb.getCompl());
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 11:27:24 +08:00
return true;
}
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-08 05:59:28 +08:00
bool BranchProbabilityInfo::calcFloatingPointHeuristics(const BasicBlock *BB) {
const BranchInst *BI = dyn_cast<BranchInst>(BB->getTerminator());
BranchProbabilityInfo: floating point equality is unlikely. This is from the same paper from Ball and Larus as the rest of the currently implemented heuristics. llvm-svn: 142677
2011-10-22 04:12:47 +08:00
if (!BI || !BI->isConditional())
return false;
Value *Cond = BI->getCondition();
FCmpInst *FCmp = dyn_cast<FCmpInst>(Cond);
Extend the floating point heuristic to consider NaN checks unlikely. llvm-svn: 142687
2011-10-22 05:13:47 +08:00
if (!FCmp)
BranchProbabilityInfo: floating point equality is unlikely. This is from the same paper from Ball and Larus as the rest of the currently implemented heuristics. llvm-svn: 142677
2011-10-22 04:12:47 +08:00
return false;
Extend the floating point heuristic to consider NaN checks unlikely. llvm-svn: 142687
2011-10-22 05:13:47 +08:00
bool isProb;
if (FCmp->isEquality()) {
// f1 == f2 -> Unlikely
// f1 != f2 -> Likely
isProb = !FCmp->isTrueWhenEqual();
} else if (FCmp->getPredicate() == FCmpInst::FCMP_ORD) {
// !isnan -> Likely
isProb = true;
} else if (FCmp->getPredicate() == FCmpInst::FCMP_UNO) {
// isnan -> Unlikely
isProb = false;
} else {
return false;
}
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
unsigned TakenIdx = 0, NonTakenIdx = 1;
BranchProbabilityInfo: floating point equality is unlikely. This is from the same paper from Ball and Larus as the rest of the currently implemented heuristics. llvm-svn: 142677
2011-10-22 04:12:47 +08:00
Extend the floating point heuristic to consider NaN checks unlikely. llvm-svn: 142687
2011-10-22 05:13:47 +08:00
if (!isProb)
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
std::swap(TakenIdx, NonTakenIdx);
BranchProbabilityInfo: floating point equality is unlikely. This is from the same paper from Ball and Larus as the rest of the currently implemented heuristics. llvm-svn: 142677
2011-10-22 04:12:47 +08:00
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
BranchProbability TakenProb(FPH_TAKEN_WEIGHT,
FPH_TAKEN_WEIGHT + FPH_NONTAKEN_WEIGHT);
setEdgeProbability(BB, TakenIdx, TakenProb);
setEdgeProbability(BB, NonTakenIdx, TakenProb.getCompl());
BranchProbabilityInfo: floating point equality is unlikely. This is from the same paper from Ball and Larus as the rest of the currently implemented heuristics. llvm-svn: 142677
2011-10-22 04:12:47 +08:00
return true;
}
Add Zero Heurestics to BranchProbabilityInfo. If we compare value to zero we decide whether condition is likely to be true this way: x == 0 -> false x < 0 -> false x <= 0 -> false x != 0 -> true x > 0 -> true x >= 0 -> true llvm-svn: 136583
2011-07-31 11:27:24 +08:00
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-08 05:59:28 +08:00
bool BranchProbabilityInfo::calcInvokeHeuristics(const BasicBlock *BB) {
const InvokeInst *II = dyn_cast<InvokeInst>(BB->getTerminator());
Set the branch probability of branching to the 'normal' destination of an invoke instruction to something absurdly high, while setting the probability of branching to the 'unwind' destination to the bare minimum. This should set cause the normal destination's invoke blocks to be moved closer to the invoke. PR13612 llvm-svn: 161944
2012-08-15 20:22:35 +08:00
if (!II)
return false;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
BranchProbability TakenProb(IH_TAKEN_WEIGHT,
IH_TAKEN_WEIGHT + IH_NONTAKEN_WEIGHT);
setEdgeProbability(BB, 0 /*Index for Normal*/, TakenProb);
setEdgeProbability(BB, 1 /*Index for Unwind*/, TakenProb.getCompl());
Set the branch probability of branching to the 'normal' destination of an invoke instruction to something absurdly high, while setting the probability of branching to the 'unwind' destination to the bare minimum. This should set cause the normal destination's invoke blocks to be moved closer to the invoke. PR13612 llvm-svn: 161944
2012-08-15 20:22:35 +08:00
return true;
}
Add BranchProbabilityInfo::releaseMemory to clear the Weights field. BranchProbabilityInfo was leaking 3MB of memory when running 'opt -O2 verify-uselistorder.lto.bc'. This was due to the Weights member not being cleared once the pass is no longer needed. This adds the releaseMemory override to clear that field. The other fields are cleared at the end of runOnFunction so can stay there. llvm-svn: 238462
2015-05-29 03:43:06 +08:00
void BranchProbabilityInfo::releaseMemory() {
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
Probs.clear();
Add BranchProbabilityInfo::releaseMemory to clear the Weights field. BranchProbabilityInfo was leaking 3MB of memory when running 'opt -O2 verify-uselistorder.lto.bc'. This was due to the Weights member not being cleared once the pass is no longer needed. This adds the releaseMemory override to clear that field. The other fields are cleared at the end of runOnFunction so can stay there. llvm-svn: 238462
2015-05-29 03:43:06 +08:00
}
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 06:48:29 +08:00
void BranchProbabilityInfo::print(raw_ostream &OS) const {
Teach the BranchProbabilityInfo pass to print its results, and use that to bring it under direct test instead of merely indirectly testing it in the BlockFrequencyInfo pass. The next step is to start adding tests for the various heuristics employed, and to start fixing those heuristics once they're under test. llvm-svn: 142778
2011-10-24 05:21:50 +08:00
OS << "---- Branch Probabilities ----\n";
// We print the probabilities from the last function the analysis ran over,
// or the function it is currently running over.
assert(LastF && "Cannot print prior to running over a function");
Analysis: Remove implicit ilist iterator conversions Remove implicit ilist iterator conversions from LLVMAnalysis. I came across something really scary in `llvm::isKnownNotFullPoison()` which relied on `Instruction::getNextNode()` being completely broken (not surprising, but scary nevertheless). This function is documented (and coded to) return `nullptr` when it gets to the sentinel, but with an `ilist_half_node` as a sentinel, the sentinel check looks into some other memory and we don't recognize we've hit the end. Rooting out these scary cases is the reason I'm removing the implicit conversions before doing anything else with `ilist`; I'm not at all surprised that clients rely on badness. I found another scary case -- this time, not relying on badness, just bad (but I guess getting lucky so far) -- in `ObjectSizeOffsetEvaluator::compute_()`. Here, we save out the insertion point, do some things, and then restore it. Previously, we let the iterator auto-convert to `Instruction*`, and then set it back using the `Instruction*` version: Instruction *PrevInsertPoint = Builder.GetInsertPoint(); /* Logic that may change insert point */ if (PrevInsertPoint) Builder.SetInsertPoint(PrevInsertPoint); The check for `PrevInsertPoint` doesn't protect correctly against bad accesses. If the insertion point has been set to the end of a basic block (i.e., `SetInsertPoint(SomeBB)`), then `GetInsertPoint()` returns an iterator pointing at the list sentinel. The version of `SetInsertPoint()` that's getting called will then call `PrevInsertPoint->getParent()`, which explodes horribly. The only reason this hasn't blown up is that it's fairly unlikely the builder is adding to the end of the block; usually, we're adding instructions somewhere before the terminator. llvm-svn: 249925
2015-10-10 08:53:03 +08:00
for (const auto &BI : *LastF) {
for (succ_const_iterator SI = succ_begin(&BI), SE = succ_end(&BI); SI != SE;
++SI) {
printEdgeProbability(OS << " ", &BI, *SI);
Revert "[C++11] Add predecessors(BasicBlock *) / successors(BasicBlock *) iterator ranges." This reverts commit r213474 (and r213475), which causes a miscompile on a stage2 LTO build. I'll reply on the list in a moment. llvm-svn: 213562
2014-07-22 01:06:51 +08:00
}
}
Teach the BranchProbabilityInfo pass to print its results, and use that to bring it under direct test instead of merely indirectly testing it in the BlockFrequencyInfo pass. The next step is to start adding tests for the various heuristics employed, and to start fixing those heuristics once they're under test. llvm-svn: 142778
2011-10-24 05:21:50 +08:00
}
Add more constantness in BranchProbabilityInfo. llvm-svn: 136502
2011-07-30 03:30:00 +08:00
bool BranchProbabilityInfo::
isEdgeHot(const BasicBlock *Src, const BasicBlock *Dst) const {
Branch profiling: floating-point avoidance. Patch by: Jakub Staszak! Introduces BranchProbability. Changes unsigned to uint32_t all over and uint64_t only when overflow is expected. llvm-svn: 132867
2011-06-11 09:05:22 +08:00
// Hot probability is at least 4/5 = 80%
Add compare operators to BranchProbability and use it to determine if an edge is hot. llvm-svn: 142751
2011-10-23 19:19:14 +08:00
// FIXME: Compare against a static "hot" BranchProbability.
return getEdgeProbability(Src, Dst) > BranchProbability(4, 5);
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
}
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-08 05:59:28 +08:00
const BasicBlock *
BranchProbabilityInfo::getHotSucc(const BasicBlock *BB) const {
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
auto MaxProb = BranchProbability::getZero();
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-08 05:59:28 +08:00
const BasicBlock *MaxSucc = nullptr;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
Const correctness for BranchProbabilityInfo (NFC) From: Mehdi Amini <mehdi.amini@apple.com> llvm-svn: 265731
2016-04-08 05:59:28 +08:00
for (succ_const_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I) {
const BasicBlock *Succ = *I;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
auto Prob = getEdgeProbability(BB, Succ);
if (Prob > MaxProb) {
MaxProb = Prob;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
MaxSucc = Succ;
}
}
Add compare operators to BranchProbability and use it to determine if an edge is hot. llvm-svn: 142751
2011-10-23 19:19:14 +08:00
// Hot probability is at least 4/5 = 80%
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
if (MaxProb > BranchProbability(4, 5))
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
return MaxSucc;
[C++11] More 'nullptr' conversion. In some cases just using a boolean check instead of comparing to nullptr. llvm-svn: 206243
2014-04-15 12:59:12 +08:00
return nullptr;
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
}
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
/// Get the raw edge probability for the edge. If can't find it, return a
/// default probability 1/N where N is the number of successors. Here an edge is
/// specified using PredBlock and an
/// index to the successors.
BranchProbability
BranchProbabilityInfo::getEdgeProbability(const BasicBlock *Src,
unsigned IndexInSuccessors) const {
auto I = Probs.find(std::make_pair(Src, IndexInSuccessors));
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
if (I != Probs.end())
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
return I->second;
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
return {1,
static_cast<uint32_t>(std::distance(succ_begin(Src), succ_end(Src)))};
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
}
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
BranchProbability
BranchProbabilityInfo::getEdgeProbability(const BasicBlock *Src,
succ_const_iterator Dst) const {
return getEdgeProbability(Src, Dst.getSuccessorIndex());
[block-freq] Teach branch probability how to return the edge weight in between a BasicBlock and one of its successors. IMHO At some point BasicBlock should be refactored along the lines of MachineBasicBlock so that successors/weights are actually embedded within the block. Now is not that time though. llvm-svn: 197303
2013-12-14 10:24:25 +08:00
}
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
/// Get the raw edge probability calculated for the block pair. This returns the
/// sum of all raw edge probabilities from Src to Dst.
BranchProbability
BranchProbabilityInfo::getEdgeProbability(const BasicBlock *Src,
const BasicBlock *Dst) const {
auto Prob = BranchProbability::getZero();
bool FoundProb = false;
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
for (succ_const_iterator I = succ_begin(Src), E = succ_end(Src); I != E; ++I)
if (*I == Dst) {
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
auto MapI = Probs.find(std::make_pair(Src, I.getSuccessorIndex()));
if (MapI != Probs.end()) {
FoundProb = true;
Prob += MapI->second;
Allow 0-weight branches in BranchProbabilityInfo. Summary: When computing branch weights in BPI, we used to disallow branches with weight 0. This is a minor nuisance, because a branch with weight 0 is different to "don't have information". In the context of instrumentation, it may mean "never executed", in the context of sampling, it means "never or seldom executed". In allowing 0 weight branches, I ran into issues with the switch expansion code in selection DAG. It is currently hardwired to not handle branches with weight 0. To maintain the current behaviour, I changed it to use 1 when it finds 0, but perhaps the algorithm needs changes to tolerate branches with weight zero. Reviewers: hansw Subscribers: llvm-commits Differential Revision: http://reviews.llvm.org/D9533 llvm-svn: 236617
2015-05-07 01:55:11 +08:00
}
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
}
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
uint32_t succ_num = std::distance(succ_begin(Src), succ_end(Src));
return FoundProb ? Prob : BranchProbability(1, succ_num);
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. llvm-svn: 162572
2012-08-25 02:14:27 +08:00
}
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
/// Set the edge probability for a given edge specified by PredBlock and an
/// index to the successors.
void BranchProbabilityInfo::setEdgeProbability(const BasicBlock *Src,
unsigned IndexInSuccessors,
BranchProbability Prob) {
Probs[std::make_pair(Src, IndexInSuccessors)] = Prob;
Re-submit r272891 "Prevent dangling pointer problems in BranchProbabilityInfo" Most possibly problem was caused by the same reason as PR28400. This change bypasses it by using CallbackVH instead of AssertingVH. Differential Revision: https://reviews.llvm.org/D20957 llvm-svn: 275563
2016-07-15 22:31:16 +08:00
Handles.insert(BasicBlockCallbackVH(Src, this));
[BPI] Replace weights by probabilities in BPI. This patch removes all weight-related interfaces from BPI and replace them by probability versions. With this patch, we won't use edge weight anymore in either IR or MC passes. Edge probabilitiy is a better representation in terms of CFG update and validation. Differential revision: http://reviews.llvm.org/D15519 llvm-svn: 256263
2015-12-23 02:56:14 +08:00
DEBUG(dbgs() << "set edge " << Src->getName() << " -> " << IndexInSuccessors
<< " successor probability to " << Prob << "\n");
Replace all weight-based interfaces in MBB with probability-based interfaces, and update all uses of old interfaces. (This is the second attempt to submit this patch. The first caused two assertion failures and was reverted. See https://llvm.org/bugs/show_bug.cgi?id=25687) The patch in http://reviews.llvm.org/D13745 is broken into four parts: 1. New interfaces without functional changes (http://reviews.llvm.org/D13908). 2. Use new interfaces in SelectionDAG, while in other passes treat probabilities as weights (http://reviews.llvm.org/D14361). 3. Use new interfaces in all other passes. 4. Remove old interfaces. This patch is 3+4 above. In this patch, MBB won't provide weight-based interfaces any more, which are totally replaced by probability-based ones. The interface addSuccessor() is redesigned so that the default probability is unknown. We allow unknown probabilities but don't allow using it together with known probabilities in successor list. That is to say, we either have a list of successors with all known probabilities, or all unknown probabilities. In the latter case, we assume each successor has 1/N probability where N is the number of successors. An assertion checks if the user is attempting to add a successor with the disallowed mixed use as stated above. This can help us catch many misuses. All uses of weight-based interfaces are now updated to use probability-based ones. Differential revision: http://reviews.llvm.org/D14973 llvm-svn: 254377
2015-12-01 13:29:22 +08:00
}
Branch profiling: floating-point avoidance. Patch by: Jakub Staszak! Introduces BranchProbability. Changes unsigned to uint32_t all over and uint64_t only when overflow is expected. llvm-svn: 132867
2011-06-11 09:05:22 +08:00
raw_ostream &
Teach the BranchProbabilityInfo pass to print its results, and use that to bring it under direct test instead of merely indirectly testing it in the BlockFrequencyInfo pass. The next step is to start adding tests for the various heuristics employed, and to start fixing those heuristics once they're under test. llvm-svn: 142778
2011-10-24 05:21:50 +08:00
BranchProbabilityInfo::printEdgeProbability(raw_ostream &OS,
const BasicBlock *Src,
const BasicBlock *Dst) const {
Introduce MachineBranchProbabilityInfo class, which has similar API to BranchProbabilityInfo (expect setEdgeWeight which is not available here). Branch Weights are kept in MachineBasicBlocks. To turn off this analysis set -use-mbpi=false. llvm-svn: 133184
2011-06-17 04:22:37 +08:00
const BranchProbability Prob = getEdgeProbability(Src, Dst);
Remove all remaining uses of Value::getNameStr(). llvm-svn: 144648
2011-11-16 00:27:03 +08:00
OS << "edge " << Src->getName() << " -> " << Dst->getName()
Branch profiling: floating-point avoidance. Patch by: Jakub Staszak! Introduces BranchProbability. Changes unsigned to uint32_t all over and uint64_t only when overflow is expected. llvm-svn: 132867
2011-06-11 09:05:22 +08:00
<< " probability is " << Prob
<< (isEdgeHot(Src, Dst) ? " [HOT edge]\n" : "\n");
New BranchProbabilityInfo analysis. Patch by Jakub Staszak! BranchProbabilityInfo provides an interface for IR passes to query the likelihood that control follows a CFG edge. This patch provides an initial implementation of static branch predication that will populate BranchProbabilityInfo for branches with no external profile information using very simple heuristics. It currently isn't hooked up to any external profile data, so static prediction does all the work. llvm-svn: 132613
2011-06-04 09:16:30 +08:00
return OS;
}
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 06:48:29 +08:00
Re-submit r272891 "Prevent dangling pointer problems in BranchProbabilityInfo" Most possibly problem was caused by the same reason as PR28400. This change bypasses it by using CallbackVH instead of AssertingVH. Differential Revision: https://reviews.llvm.org/D20957 llvm-svn: 275563
2016-07-15 22:31:16 +08:00
void BranchProbabilityInfo::eraseBlock(const BasicBlock *BB) {
for (auto I = Probs.begin(), E = Probs.end(); I != E; ++I) {
auto Key = I->first;
if (Key.first == BB)
Probs.erase(Key);
}
}
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 17:44:40 +08:00
void BranchProbabilityInfo::calculate(const Function &F, const LoopInfo &LI,
const TargetLibraryInfo *TLI) {
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 06:48:29 +08:00
DEBUG(dbgs() << "---- Branch Probability Info : " << F.getName()
<< " ----\n\n");
LastF = &F; // Store the last function we ran on for printing.
assert(PostDominatedByUnreachable.empty());
assert(PostDominatedByColdCall.empty());
// Walk the basic blocks in post-order so that we can build up state about
// the successors of a block iteratively.
for (auto BB : post_order(&F.getEntryBlock())) {
DEBUG(dbgs() << "Computing probabilities for " << BB->getName() << "\n");
[BPI] Refactor post domination calculation and simple fix for ColdCall Collection of PostDominatedByUnreachable and PostDominatedByColdCall have been split out of heuristics itself. Update of the data happens now for each basic block (before update for PostDominatedByColdCall might be skipped if unreachable or matadata heuristic handled this basic block). This separation allows re-ordering of heuristics without loosing the post-domination information. Reviewers: sanjoy, junbuml, vsk, chandlerc, reames Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31701 llvm-svn: 300029
2017-04-12 13:42:14 +08:00
updatePostDominatedByUnreachable(BB);
updatePostDominatedByColdCall(BB);
[BPI] NFC: reorder ifs to bail out earlier This is non-functional change to re-order if statements to bail out earlier from unreachable and ColdCall heuristics. Reviewers: sanjoy, reames, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D31704 llvm-svn: 300442
2017-04-17 14:39:47 +08:00
// If there is no at least two successors, no sense to set probability.
if (BB->getTerminator()->getNumSuccessors() < 2)
continue;
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 06:48:29 +08:00
if (calcMetadataWeights(BB))
continue;
[BPI] Use metadata info before any other heuristics Metadata potentially is more precise than any heuristics we use, so it makes sense to use first metadata info if it is available. However it makes sense to examine it against other strong heuristics like unreachable one. If edge coming to unreachable block has higher probability then it is expected by unreachable heuristic then we use heuristic and remaining probability is distributed among other reachable blocks equally. An example where metadata might be more strong then unreachable heuristic is as follows: it is possible that there are two branches and for the branch A metadata says that its probability is (0, 2^25). For the branch B the probability is (1, 2^25). So the expectation is that first edge of B is hotter than first edge of A because first edge of A did not executed at least once. If first edge of A points to the unreachable block then using the unreachable heuristics we'll set the probability for A to (1, 2^20) and now edge of A becomes hotter than edge of B. This is unexpected behavior. This fixed the biggest part of https://bugs.llvm.org/show_bug.cgi?id=32214 Reviewers: sanjoy, junbuml, vsk, chandlerc Reviewed By: chandlerc Subscribers: llvm-commits, reames, davidxl Differential Revision: https://reviews.llvm.org/D30631 llvm-svn: 300440
2017-04-17 12:33:04 +08:00
if (calcUnreachableHeuristics(BB))
continue;
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 06:48:29 +08:00
if (calcColdCallHeuristics(BB))
continue;
if (calcLoopBranchHeuristics(BB, LI))
continue;
if (calcPointerHeuristics(BB))
continue;
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 17:44:40 +08:00
if (calcZeroHeuristics(BB, TLI))
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 06:48:29 +08:00
continue;
if (calcFloatingPointHeuristics(BB))
continue;
calcInvokeHeuristics(BB);
}
PostDominatedByUnreachable.clear();
PostDominatedByColdCall.clear();
Add options to dump block frequency/branch probability info in text. Summary: Add options -print-bfi/-print-bpi that dump block frequency and branch probability info like -view-block-freq-propagation-dags and -view-machine-block-freq-propagation-dags do but in text. This is useful when the graph is very large and complex (the dot command crashes, lines/edges too close to tell apart, hard to navigate without textual search) or simply when text is preferred. Reviewers: davidxl Reviewed By: davidxl Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D37165 llvm-svn: 311822
2017-08-26 08:31:00 +08:00
if (PrintBranchProb &&
(PrintBranchProbFuncName.empty() ||
F.getName().equals(PrintBranchProbFuncName))) {
print(dbgs());
}
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 06:48:29 +08:00
}
void BranchProbabilityInfoWrapperPass::getAnalysisUsage(
AnalysisUsage &AU) const {
AU.addRequired<LoopInfoWrapperPass>();
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 17:44:40 +08:00
AU.addRequired<TargetLibraryInfoWrapperPass>();
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 06:48:29 +08:00
AU.setPreservesAll();
}
bool BranchProbabilityInfoWrapperPass::runOnFunction(Function &F) {
const LoopInfo &LI = getAnalysis<LoopInfoWrapperPass>().getLoopInfo();
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 17:44:40 +08:00
const TargetLibraryInfo &TLI = getAnalysis<TargetLibraryInfoWrapperPass>().getTLI();
BPI.calculate(F, LI, &TLI);
Create a wrapper pass for BranchProbabilityInfo. This new wrapper pass is useful when we want to do branch probability analysis conditionally (e.g. only in PGO mode) but don't want to add one more pass dependence. http://reviews.llvm.org/D11241 llvm-svn: 242349
2015-07-16 06:48:29 +08:00
return false;
}
void BranchProbabilityInfoWrapperPass::releaseMemory() { BPI.releaseMemory(); }
void BranchProbabilityInfoWrapperPass::print(raw_ostream &OS,
const Module *) const {
BPI.print(OS);
}
[PM] Port Branch Probability Analysis pass to the new pass manager. Differential Revision: http://reviews.llvm.org/D19839 llvm-svn: 268601
2016-05-05 10:59:57 +08:00
[PM] Change the static object whose address is used to uniquely identify analyses to have a common type which is enforced rather than using a char object and a `void *` type when used as an identifier. This has a number of advantages. First, it at least helps some of the confusion raised in Justin Lebar's code review of why `void *` was being used everywhere by having a stronger type that connects to documentation about this. However, perhaps more importantly, it addresses a serious issue where the alignment of these pointer-like identifiers was unknown. This made it hard to use them in pointer-like data structures. We were already dodging this in dangerous ways to create the "all analyses" entry. In a subsequent patch I attempted to use these with TinyPtrVector and things fell apart in a very bad way. And it isn't just a compile time or type system issue. Worse than that, the actual alignment of these pointer-like opaque identifiers wasn't guaranteed to be a useful alignment as they were just characters. This change introduces a type to use as the "key" object whose address forms the opaque identifier. This both forces the objects to have proper alignment, and provides type checking that we get it right everywhere. It also makes the types somewhat less mysterious than `void *`. We could go one step further and introduce a truly opaque pointer-like type to return from the `ID()` static function rather than returning `AnalysisKey *`, but that didn't seem to be a clear win so this is just the initial change to get to a reliably typed and aligned object serving is a key for all the analyses. Thanks to Richard Smith and Justin Lebar for helping pick plausible names and avoid making this refactoring many times. =] And thanks to Sean for the super fast review! While here, I've tried to move away from the "PassID" nomenclature entirely as it wasn't really helping and is overloaded with old pass manager constructs. Now we have IDs for analyses, and key objects whose address can be used as IDs. Where possible and clear I've shortened this to just "ID". In a few places I kept "AnalysisID" to make it clear what was being identified. Differential Revision: https://reviews.llvm.org/D27031 llvm-svn: 287783
2016-11-24 01:53:26 +08:00
AnalysisKey BranchProbabilityAnalysis::Key;
[PM] Port Branch Probability Analysis pass to the new pass manager. Differential Revision: http://reviews.llvm.org/D19839 llvm-svn: 268601
2016-05-05 10:59:57 +08:00
BranchProbabilityInfo
Consistently use FunctionAnalysisManager Besides a general consistently benefit, the extra layer of indirection allows the mechanical part of https://reviews.llvm.org/D23256 that requires touching every transformation and analysis to be factored out cleanly. Thanks to David for the suggestion. llvm-svn: 278077
2016-08-09 08:28:15 +08:00
BranchProbabilityAnalysis::run(Function &F, FunctionAnalysisManager &AM) {
[PM] Port Branch Probability Analysis pass to the new pass manager. Differential Revision: http://reviews.llvm.org/D19839 llvm-svn: 268601
2016-05-05 10:59:57 +08:00
BranchProbabilityInfo BPI;
[BPI] Don't assume that strcmp returning >0 is more likely than <0 The zero heuristic assumes that integers are more likely positive than negative, but this also has the effect of assuming that strcmp return values are more likely positive than negative. Given that for nonzero strcmp return values it's the ordering of arguments that determines the sign of the result there's no reason to assume that's true. Fix this by inspecting the LHS of the compare and using TargetLibraryInfo to decide if it's strcmp-like, and if so only assume that nonzero is more likely than zero i.e. strings are more often different than the same. This causes a slight code generation change in the spec2006 benchmark 403.gcc, but with no noticeable performance impact. The intent of this patch is to allow better optimisation of dhrystone on Cortex-M cpus, but currently it won't as there are also some changes that need to be made to if-conversion. Differential Revision: https://reviews.llvm.org/D33934 llvm-svn: 304970
2017-06-08 17:44:40 +08:00
BPI.calculate(F, AM.getResult<LoopAnalysis>(F), &AM.getResult<TargetLibraryAnalysis>(F));
[PM] Port Branch Probability Analysis pass to the new pass manager. Differential Revision: http://reviews.llvm.org/D19839 llvm-svn: 268601
2016-05-05 10:59:57 +08:00
return BPI;
}
PreservedAnalyses
Consistently use FunctionAnalysisManager Besides a general consistently benefit, the extra layer of indirection allows the mechanical part of https://reviews.llvm.org/D23256 that requires touching every transformation and analysis to be factored out cleanly. Thanks to David for the suggestion. llvm-svn: 278077
2016-08-09 08:28:15 +08:00
BranchProbabilityPrinterPass::run(Function &F, FunctionAnalysisManager &AM) {
[PM] Port Branch Probability Analysis pass to the new pass manager. Differential Revision: http://reviews.llvm.org/D19839 llvm-svn: 268601
2016-05-05 10:59:57 +08:00
OS << "Printing analysis results of BPI for function "
<< "'" << F.getName() << "':"
<< "\n";
AM.getResult<BranchProbabilityAnalysis>(F).print(OS);
return PreservedAnalyses::all();
}