[CSSPGO] Call site prioritized inlining for sample PGO
This change implemented call site prioritized BFS profile guided inlining for sample profile loader. The new inlining strategy maximize the benefit of context-sensitive profile as mentioned in the follow up discussion of CSSPGO RFC. The change will not affect today's AutoFDO as it's opt-in. CSSPGO now defaults to the new FDO inliner, but can fall back to today's replay inliner using a switch (`-sample-profile-prioritized-inline=0`).
Motivation
With baseline AutoFDO, the inliner in sample profile loader only replays previous inlining, and the use of profile is only for pruning previous inlining that turned out to be cold. Due to the nature of replay, the FDO inliner is simple with hotness being the only decision factor. It has the following limitations that we're improving now for CSSPGO.
- It doesn't take inline candidate size into account. Since it's doing replay, the size growth is bounded by previous CGSCC inlining. With context-sensitive profile, FDO inliner is no longer limited by previous inlining, so we need to take size into account to avoid significant size bloat.
- The way it looks at hotness is not accurate. It uses total samples in an inlinee as proxy for hotness, while what really matters for an inline decision is the call site count. This is an unfortunate fall back because call site count and callee entry count are not reliable due to dwarf based correlation, especially for inlinees. Now paired with pseudo-probe, we have accurate call site count and callee's entry count, so we can use that to gauge hotness more accurately.
- It treats all call sites from a block as hot as long as there's one call site considered hot. This is normally true, but since total samples is used as hotness proxy, this transitiveness within block magnifies the inacurate hotness heuristic. With pseduo-probe and the change above, this is no longer an issue for CSSPGO.
New FDO Inliner
Putting all the requirement for CSSPGO together, we need a top-down call site prioritized BFS inliner. Here're reasons why each component is needed.
- Top-down: We need a top-down inliner to better leverage context-sensitive profile, so inlining is driven by accurate context profile, and post-inline is also accurate. This is already implemented in https://reviews.llvm.org/D70655.
- Size Cap: For top-down inliner, taking function size into account for inline decision alone isn't sufficient to control size growth. We also need to explicitly cap size growth because with top-down inlining, we can grow inliner size significantly with large number of smaller inlinees even if each individually passes the cost/size check.
- Prioritize call sites: With size cap, inlining order also becomes important, because if we stop inlining due to size budget limit, we'd want to use budget towards the most beneficial call sites.
- BFS inline: Same as call site prioritization, if we stop inlining due to size budget limit, we want a balanced inline tree, rather than going deep on one call path.
Note that the new inliner avoids repeatedly evaluating same set of call site, so it should help with compile time too. For this reason, we could transition today's FDO inliner to use a queue with equal priority to avoid wasted reevaluation of same call site (TODO).
Speculative indirect call promotion and inlining is also supported now with CSSPGO just like baseline AutoFDO.
Tunings and knobs
I created tuning knobs for size growth/cap control, and for hot threshold separate from CGSCC inliner. The default values are selected based on initial tuning with CSSPGO.
Results
Evaluated with an internal LLVM fork couple months ago, plus another change to adjust hot-threshold cutoff for context profile (will send up after this one), the new inliner show ~1% geomean perf win on spec2006 with CSSPGO, while reducing code size too. The measurement was done using train-train setup, MonoLTO w/ new pass manager and pseudo-probe. Note that this is just a starting point - we hope that the new inliner will open up more opportunity with CSSPGO, but it will certainly take more time and effort to make it fully calibrated and ready for bigger workloads (we're working on it).
Differential Revision: https://reviews.llvm.org/D94001
2021-01-04 08:43:06 +08:00
; Test for CSSPGO's new early inliner using priority queue
; Note that we need new pass manager to enable top-down processing for sample profile loader
; Test we inlined the following in top-down order with old inliner
; main:3 @ _Z5funcAi
; main:3 @ _Z5funcAi:1 @ _Z8funcLeafi
; _Z5funcBi:1 @ _Z8funcLeafi
; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/profile-context-tracker.prof -sample-profile-inline-size -sample-profile-prioritized-inline=0 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-BASE
2021-08-31 01:31:47 +08:00
; RUN: llvm-profdata merge --sample --extbinary --use-md5 %S/Inputs/profile-context-tracker.prof -o %t.md5
; RUN: opt < %s -passes=sample-profile -sample-profile-file=%t.md5 -sample-profile-inline-size -sample-profile-prioritized-inline=0 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-BASE
[CSSPGO] Call site prioritized inlining for sample PGO
This change implemented call site prioritized BFS profile guided inlining for sample profile loader. The new inlining strategy maximize the benefit of context-sensitive profile as mentioned in the follow up discussion of CSSPGO RFC. The change will not affect today's AutoFDO as it's opt-in. CSSPGO now defaults to the new FDO inliner, but can fall back to today's replay inliner using a switch (`-sample-profile-prioritized-inline=0`).
Motivation
With baseline AutoFDO, the inliner in sample profile loader only replays previous inlining, and the use of profile is only for pruning previous inlining that turned out to be cold. Due to the nature of replay, the FDO inliner is simple with hotness being the only decision factor. It has the following limitations that we're improving now for CSSPGO.
- It doesn't take inline candidate size into account. Since it's doing replay, the size growth is bounded by previous CGSCC inlining. With context-sensitive profile, FDO inliner is no longer limited by previous inlining, so we need to take size into account to avoid significant size bloat.
- The way it looks at hotness is not accurate. It uses total samples in an inlinee as proxy for hotness, while what really matters for an inline decision is the call site count. This is an unfortunate fall back because call site count and callee entry count are not reliable due to dwarf based correlation, especially for inlinees. Now paired with pseudo-probe, we have accurate call site count and callee's entry count, so we can use that to gauge hotness more accurately.
- It treats all call sites from a block as hot as long as there's one call site considered hot. This is normally true, but since total samples is used as hotness proxy, this transitiveness within block magnifies the inacurate hotness heuristic. With pseduo-probe and the change above, this is no longer an issue for CSSPGO.
New FDO Inliner
Putting all the requirement for CSSPGO together, we need a top-down call site prioritized BFS inliner. Here're reasons why each component is needed.
- Top-down: We need a top-down inliner to better leverage context-sensitive profile, so inlining is driven by accurate context profile, and post-inline is also accurate. This is already implemented in https://reviews.llvm.org/D70655.
- Size Cap: For top-down inliner, taking function size into account for inline decision alone isn't sufficient to control size growth. We also need to explicitly cap size growth because with top-down inlining, we can grow inliner size significantly with large number of smaller inlinees even if each individually passes the cost/size check.
- Prioritize call sites: With size cap, inlining order also becomes important, because if we stop inlining due to size budget limit, we'd want to use budget towards the most beneficial call sites.
- BFS inline: Same as call site prioritization, if we stop inlining due to size budget limit, we want a balanced inline tree, rather than going deep on one call path.
Note that the new inliner avoids repeatedly evaluating same set of call site, so it should help with compile time too. For this reason, we could transition today's FDO inliner to use a queue with equal priority to avoid wasted reevaluation of same call site (TODO).
Speculative indirect call promotion and inlining is also supported now with CSSPGO just like baseline AutoFDO.
Tunings and knobs
I created tuning knobs for size growth/cap control, and for hot threshold separate from CGSCC inliner. The default values are selected based on initial tuning with CSSPGO.
Results
Evaluated with an internal LLVM fork couple months ago, plus another change to adjust hot-threshold cutoff for context profile (will send up after this one), the new inliner show ~1% geomean perf win on spec2006 with CSSPGO, while reducing code size too. The measurement was done using train-train setup, MonoLTO w/ new pass manager and pseudo-probe. Note that this is just a starting point - we hope that the new inliner will open up more opportunity with CSSPGO, but it will certainly take more time and effort to make it fully calibrated and ready for bigger workloads (we're working on it).
Differential Revision: https://reviews.llvm.org/D94001
2021-01-04 08:43:06 +08:00
; With new FDO early inliner, callee entry count is used to drive inlining instead of callee total samples, so we get less inlining for given profile
; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/profile-context-tracker.prof -sample-profile-inline-size -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-NEW
;
; With new FDO early inliner, callee entry count is used to drive inlining instead of callee total samples, tuning hot cutoff can get us the same inlining
; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/profile-context-tracker.prof -sample-profile-inline-size -profile-summary-cutoff-hot=999900 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-BASE
;
; With new FDO early inliner, callee entry count is used to drive inlining instead of callee total samples, tuning cold sample profile inline threshold can get us the same inlining
; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/profile-context-tracker.prof -sample-profile-inline-size -sample-profile-cold-inline-threshold=200 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-BASE
;
; With new FDO early inliner and tuned cutoff, we can control inlining through size growth tuning knob.
; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/profile-context-tracker.prof -sample-profile-inline-size -profile-summary-cutoff-hot=999900 -sample-profile-inline-limit-min=0 -sample-profile-inline-growth-limit=1 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --allow-empty --check-prefix=INLINE-NEW-LIMIT1
; RUN: opt < %s -passes=sample-profile -sample-profile-file=%S/Inputs/profile-context-tracker.prof -sample-profile-inline-size -profile-summary-cutoff-hot=999900 -sample-profile-inline-limit-min=10 -sample-profile-inline-growth-limit=1 -profile-sample-accurate -S -pass-remarks=inline -o /dev/null 2>&1 | FileCheck %s --check-prefix=INLINE-NEW-LIMIT2
2021-08-11 02:51:31 +08:00
; INLINE-BASE: remark: merged.cpp:14:10: '_Z5funcAi' inlined into 'main' to match profiling context with (cost={{[0-9]+}}, threshold={{[0-9]+}}) at callsite main:3:10
; INLINE-BASE: remark: merged.cpp:27:11: '_Z8funcLeafi' inlined into 'main' to match profiling context with (cost={{[0-9]+}}, threshold={{[0-9]+}}) at callsite _Z5funcAi:1:11 @ main:3:10
; INLINE-BASE: remark: merged.cpp:33:11: '_Z8funcLeafi' inlined into '_Z5funcBi' to match profiling context with (cost={{[0-9]+}}, threshold={{[0-9]+}}) at callsite _Z5funcBi:1:11
[CSSPGO] Call site prioritized inlining for sample PGO
This change implemented call site prioritized BFS profile guided inlining for sample profile loader. The new inlining strategy maximize the benefit of context-sensitive profile as mentioned in the follow up discussion of CSSPGO RFC. The change will not affect today's AutoFDO as it's opt-in. CSSPGO now defaults to the new FDO inliner, but can fall back to today's replay inliner using a switch (`-sample-profile-prioritized-inline=0`).
Motivation
With baseline AutoFDO, the inliner in sample profile loader only replays previous inlining, and the use of profile is only for pruning previous inlining that turned out to be cold. Due to the nature of replay, the FDO inliner is simple with hotness being the only decision factor. It has the following limitations that we're improving now for CSSPGO.
- It doesn't take inline candidate size into account. Since it's doing replay, the size growth is bounded by previous CGSCC inlining. With context-sensitive profile, FDO inliner is no longer limited by previous inlining, so we need to take size into account to avoid significant size bloat.
- The way it looks at hotness is not accurate. It uses total samples in an inlinee as proxy for hotness, while what really matters for an inline decision is the call site count. This is an unfortunate fall back because call site count and callee entry count are not reliable due to dwarf based correlation, especially for inlinees. Now paired with pseudo-probe, we have accurate call site count and callee's entry count, so we can use that to gauge hotness more accurately.
- It treats all call sites from a block as hot as long as there's one call site considered hot. This is normally true, but since total samples is used as hotness proxy, this transitiveness within block magnifies the inacurate hotness heuristic. With pseduo-probe and the change above, this is no longer an issue for CSSPGO.
New FDO Inliner
Putting all the requirement for CSSPGO together, we need a top-down call site prioritized BFS inliner. Here're reasons why each component is needed.
- Top-down: We need a top-down inliner to better leverage context-sensitive profile, so inlining is driven by accurate context profile, and post-inline is also accurate. This is already implemented in https://reviews.llvm.org/D70655.
- Size Cap: For top-down inliner, taking function size into account for inline decision alone isn't sufficient to control size growth. We also need to explicitly cap size growth because with top-down inlining, we can grow inliner size significantly with large number of smaller inlinees even if each individually passes the cost/size check.
- Prioritize call sites: With size cap, inlining order also becomes important, because if we stop inlining due to size budget limit, we'd want to use budget towards the most beneficial call sites.
- BFS inline: Same as call site prioritization, if we stop inlining due to size budget limit, we want a balanced inline tree, rather than going deep on one call path.
Note that the new inliner avoids repeatedly evaluating same set of call site, so it should help with compile time too. For this reason, we could transition today's FDO inliner to use a queue with equal priority to avoid wasted reevaluation of same call site (TODO).
Speculative indirect call promotion and inlining is also supported now with CSSPGO just like baseline AutoFDO.
Tunings and knobs
I created tuning knobs for size growth/cap control, and for hot threshold separate from CGSCC inliner. The default values are selected based on initial tuning with CSSPGO.
Results
Evaluated with an internal LLVM fork couple months ago, plus another change to adjust hot-threshold cutoff for context profile (will send up after this one), the new inliner show ~1% geomean perf win on spec2006 with CSSPGO, while reducing code size too. The measurement was done using train-train setup, MonoLTO w/ new pass manager and pseudo-probe. Note that this is just a starting point - we hope that the new inliner will open up more opportunity with CSSPGO, but it will certainly take more time and effort to make it fully calibrated and ready for bigger workloads (we're working on it).
Differential Revision: https://reviews.llvm.org/D94001
2021-01-04 08:43:06 +08:00
2021-08-11 02:51:31 +08:00
; INLINE-NEW: remark: merged.cpp:14:10: '_Z5funcAi' inlined into 'main' to match profiling context with (cost={{[0-9]+}}, threshold={{[0-9]+}}) at callsite main:3:10
[CSSPGO] Call site prioritized inlining for sample PGO
This change implemented call site prioritized BFS profile guided inlining for sample profile loader. The new inlining strategy maximize the benefit of context-sensitive profile as mentioned in the follow up discussion of CSSPGO RFC. The change will not affect today's AutoFDO as it's opt-in. CSSPGO now defaults to the new FDO inliner, but can fall back to today's replay inliner using a switch (`-sample-profile-prioritized-inline=0`).
Motivation
With baseline AutoFDO, the inliner in sample profile loader only replays previous inlining, and the use of profile is only for pruning previous inlining that turned out to be cold. Due to the nature of replay, the FDO inliner is simple with hotness being the only decision factor. It has the following limitations that we're improving now for CSSPGO.
- It doesn't take inline candidate size into account. Since it's doing replay, the size growth is bounded by previous CGSCC inlining. With context-sensitive profile, FDO inliner is no longer limited by previous inlining, so we need to take size into account to avoid significant size bloat.
- The way it looks at hotness is not accurate. It uses total samples in an inlinee as proxy for hotness, while what really matters for an inline decision is the call site count. This is an unfortunate fall back because call site count and callee entry count are not reliable due to dwarf based correlation, especially for inlinees. Now paired with pseudo-probe, we have accurate call site count and callee's entry count, so we can use that to gauge hotness more accurately.
- It treats all call sites from a block as hot as long as there's one call site considered hot. This is normally true, but since total samples is used as hotness proxy, this transitiveness within block magnifies the inacurate hotness heuristic. With pseduo-probe and the change above, this is no longer an issue for CSSPGO.
New FDO Inliner
Putting all the requirement for CSSPGO together, we need a top-down call site prioritized BFS inliner. Here're reasons why each component is needed.
- Top-down: We need a top-down inliner to better leverage context-sensitive profile, so inlining is driven by accurate context profile, and post-inline is also accurate. This is already implemented in https://reviews.llvm.org/D70655.
- Size Cap: For top-down inliner, taking function size into account for inline decision alone isn't sufficient to control size growth. We also need to explicitly cap size growth because with top-down inlining, we can grow inliner size significantly with large number of smaller inlinees even if each individually passes the cost/size check.
- Prioritize call sites: With size cap, inlining order also becomes important, because if we stop inlining due to size budget limit, we'd want to use budget towards the most beneficial call sites.
- BFS inline: Same as call site prioritization, if we stop inlining due to size budget limit, we want a balanced inline tree, rather than going deep on one call path.
Note that the new inliner avoids repeatedly evaluating same set of call site, so it should help with compile time too. For this reason, we could transition today's FDO inliner to use a queue with equal priority to avoid wasted reevaluation of same call site (TODO).
Speculative indirect call promotion and inlining is also supported now with CSSPGO just like baseline AutoFDO.
Tunings and knobs
I created tuning knobs for size growth/cap control, and for hot threshold separate from CGSCC inliner. The default values are selected based on initial tuning with CSSPGO.
Results
Evaluated with an internal LLVM fork couple months ago, plus another change to adjust hot-threshold cutoff for context profile (will send up after this one), the new inliner show ~1% geomean perf win on spec2006 with CSSPGO, while reducing code size too. The measurement was done using train-train setup, MonoLTO w/ new pass manager and pseudo-probe. Note that this is just a starting point - we hope that the new inliner will open up more opportunity with CSSPGO, but it will certainly take more time and effort to make it fully calibrated and ready for bigger workloads (we're working on it).
Differential Revision: https://reviews.llvm.org/D94001
2021-01-04 08:43:06 +08:00
; INLINE-NEW-NOT: remark
; INLINE-NEW-LIMIT1-NOT: remark
2021-08-11 02:51:31 +08:00
; INLINE-NEW-LIMIT2: remark: merged.cpp:33:11: '_Z8funcLeafi' inlined into '_Z5funcBi' to match profiling context with (cost={{[0-9]+}}, threshold={{[0-9]+}}) at callsite _Z5funcBi:1:11
[CSSPGO] Call site prioritized inlining for sample PGO
This change implemented call site prioritized BFS profile guided inlining for sample profile loader. The new inlining strategy maximize the benefit of context-sensitive profile as mentioned in the follow up discussion of CSSPGO RFC. The change will not affect today's AutoFDO as it's opt-in. CSSPGO now defaults to the new FDO inliner, but can fall back to today's replay inliner using a switch (`-sample-profile-prioritized-inline=0`).
Motivation
With baseline AutoFDO, the inliner in sample profile loader only replays previous inlining, and the use of profile is only for pruning previous inlining that turned out to be cold. Due to the nature of replay, the FDO inliner is simple with hotness being the only decision factor. It has the following limitations that we're improving now for CSSPGO.
- It doesn't take inline candidate size into account. Since it's doing replay, the size growth is bounded by previous CGSCC inlining. With context-sensitive profile, FDO inliner is no longer limited by previous inlining, so we need to take size into account to avoid significant size bloat.
- The way it looks at hotness is not accurate. It uses total samples in an inlinee as proxy for hotness, while what really matters for an inline decision is the call site count. This is an unfortunate fall back because call site count and callee entry count are not reliable due to dwarf based correlation, especially for inlinees. Now paired with pseudo-probe, we have accurate call site count and callee's entry count, so we can use that to gauge hotness more accurately.
- It treats all call sites from a block as hot as long as there's one call site considered hot. This is normally true, but since total samples is used as hotness proxy, this transitiveness within block magnifies the inacurate hotness heuristic. With pseduo-probe and the change above, this is no longer an issue for CSSPGO.
New FDO Inliner
Putting all the requirement for CSSPGO together, we need a top-down call site prioritized BFS inliner. Here're reasons why each component is needed.
- Top-down: We need a top-down inliner to better leverage context-sensitive profile, so inlining is driven by accurate context profile, and post-inline is also accurate. This is already implemented in https://reviews.llvm.org/D70655.
- Size Cap: For top-down inliner, taking function size into account for inline decision alone isn't sufficient to control size growth. We also need to explicitly cap size growth because with top-down inlining, we can grow inliner size significantly with large number of smaller inlinees even if each individually passes the cost/size check.
- Prioritize call sites: With size cap, inlining order also becomes important, because if we stop inlining due to size budget limit, we'd want to use budget towards the most beneficial call sites.
- BFS inline: Same as call site prioritization, if we stop inlining due to size budget limit, we want a balanced inline tree, rather than going deep on one call path.
Note that the new inliner avoids repeatedly evaluating same set of call site, so it should help with compile time too. For this reason, we could transition today's FDO inliner to use a queue with equal priority to avoid wasted reevaluation of same call site (TODO).
Speculative indirect call promotion and inlining is also supported now with CSSPGO just like baseline AutoFDO.
Tunings and knobs
I created tuning knobs for size growth/cap control, and for hot threshold separate from CGSCC inliner. The default values are selected based on initial tuning with CSSPGO.
Results
Evaluated with an internal LLVM fork couple months ago, plus another change to adjust hot-threshold cutoff for context profile (will send up after this one), the new inliner show ~1% geomean perf win on spec2006 with CSSPGO, while reducing code size too. The measurement was done using train-train setup, MonoLTO w/ new pass manager and pseudo-probe. Note that this is just a starting point - we hope that the new inliner will open up more opportunity with CSSPGO, but it will certainly take more time and effort to make it fully calibrated and ready for bigger workloads (we're working on it).
Differential Revision: https://reviews.llvm.org/D94001
2021-01-04 08:43:06 +08:00
; INLINE-NEW-LIMIT2-NOT: remark
@factor = d s o _ l o c a l global i32 3 , align 4 , !dbg !0
define d s o _ l o c a l i32 @main ( ) local_unnamed_addr #0 !dbg !18 {
entry:
br label %for.body , !dbg !25
for.cond.cleanup: ; preds = %for.body
ret i32 %add3 , !dbg !27
for.body: ; preds = %for.body, %entry
%x.011 = phi i32 [ 300000 , %entry ] , [ %dec , %for.body ]
%r.010 = phi i32 [ 0 , %entry ] , [ %add3 , %for.body ]
%call = tail call i32 @_Z5funcBi ( i32 %x.011 ) , !dbg !32
%add = add nuw nsw i32 %x.011 , 1 , !dbg !31
%call1 = tail call i32 @_Z5funcAi ( i32 %add ) , !dbg !28
%add2 = add i32 %call , %r.010 , !dbg !34
%add3 = add i32 %add2 , %call1 , !dbg !35
%dec = add nsw i32 %x.011 , -1 , !dbg !36
%cmp = icmp eq i32 %x.011 , 0 , !dbg !38
br i1 %cmp , label %for.cond.cleanup , label %for.body , !dbg !25
}
define d s o _ l o c a l i32 @_Z5funcAi ( i32 %x ) local_unnamed_addr #1 !dbg !40 {
entry:
%add = add nsw i32 %x , 100000 , !dbg !44
%call = tail call i32 @_Z8funcLeafi ( i32 %add ) , !dbg !45
ret i32 %call , !dbg !46
}
define d s o _ l o c a l i32 @_Z8funcLeafi ( i32 %x ) local_unnamed_addr #1 !dbg !54 {
entry:
%cmp = icmp sgt i32 %x , 0 , !dbg !57
br i1 %cmp , label %while.body , label %while.cond2.preheader , !dbg !59
while.cond2.preheader: ; preds = %entry
%cmp313 = icmp slt i32 %x , 0 , !dbg !60
br i1 %cmp313 , label %while.body4 , label %if.end , !dbg !63
while.body: ; preds = %while.body, %entry
%x.addr.016 = phi i32 [ %sub , %while.body ] , [ %x , %entry ]
%tmp = load volatile i32 , i32 * @factor , align 4 , !dbg !64
%call = tail call i32 @_Z3fibi ( i32 %tmp ) , !dbg !67
%sub = sub nsw i32 %x.addr.016 , %call , !dbg !68
%cmp1 = icmp sgt i32 %sub , 0 , !dbg !69
br i1 %cmp1 , label %while.body , label %if.end , !dbg !71
while.body4: ; preds = %while.body4, %while.cond2.preheader
%x.addr.114 = phi i32 [ %add , %while.body4 ] , [ %x , %while.cond2.preheader ]
%tmp1 = load volatile i32 , i32 * @factor , align 4 , !dbg !72
%call5 = tail call i32 @_Z3fibi ( i32 %tmp1 ) , !dbg !74
%add = add nsw i32 %call5 , %x.addr.114 , !dbg !75
%cmp3 = icmp slt i32 %add , 0 , !dbg !60
br i1 %cmp3 , label %while.body4 , label %if.end , !dbg !63
if.end: ; preds = %while.body4, %while.body, %while.cond2.preheader
%x.addr.2 = phi i32 [ 0 , %while.cond2.preheader ] , [ %sub , %while.body ] , [ %add , %while.body4 ]
ret i32 %x.addr.2 , !dbg !76
}
define d s o _ l o c a l i32 @_Z5funcBi ( i32 %x ) local_unnamed_addr #0 !dbg !47 {
entry:
%sub = add nsw i32 %x , -100000 , !dbg !51
%call = tail call i32 @_Z8funcLeafi ( i32 %sub ) , !dbg !52
ret i32 %call , !dbg !53
}
declare i32 @_Z3fibi ( i32 )
2021-05-25 01:43:40 +08:00
attributes #0 = { n o free noinline norecurse nounwind uwtable "correctly-rounded-divide-sqrt-fp-math" = "false" "disable-tail-calls" = "false" "frame-pointer" = "none" "less-precise-fpmad" = "false" "min-legal-vector-width" = "0" "no-infs-fp-math" = "false" "no-jump-tables" = "false" "no-nans-fp-math" = "false" "no-signed-zeros-fp-math" = "false" "no-trapping-math" = "false" "stack-protector-buffer-size" = "8" "target-cpu" = "x86-64" "target-features" = "+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math" = "false" "use-soft-float" = "false" "use-sample-profile" }
attributes #1 = { n o free norecurse nounwind uwtable "correctly-rounded-divide-sqrt-fp-math" = "false" "disable-tail-calls" = "false" "frame-pointer" = "none" "less-precise-fpmad" = "false" "min-legal-vector-width" = "0" "no-infs-fp-math" = "false" "no-jump-tables" = "false" "no-nans-fp-math" = "false" "no-signed-zeros-fp-math" = "false" "no-trapping-math" = "false" "stack-protector-buffer-size" = "8" "target-cpu" = "x86-64" "target-features" = "+cx8,+fxsr,+mmx,+sse,+sse2,+x87" "unsafe-fp-math" = "false" "use-soft-float" = "false" "use-sample-profile" }
[CSSPGO] Call site prioritized inlining for sample PGO
This change implemented call site prioritized BFS profile guided inlining for sample profile loader. The new inlining strategy maximize the benefit of context-sensitive profile as mentioned in the follow up discussion of CSSPGO RFC. The change will not affect today's AutoFDO as it's opt-in. CSSPGO now defaults to the new FDO inliner, but can fall back to today's replay inliner using a switch (`-sample-profile-prioritized-inline=0`).
Motivation
With baseline AutoFDO, the inliner in sample profile loader only replays previous inlining, and the use of profile is only for pruning previous inlining that turned out to be cold. Due to the nature of replay, the FDO inliner is simple with hotness being the only decision factor. It has the following limitations that we're improving now for CSSPGO.
- It doesn't take inline candidate size into account. Since it's doing replay, the size growth is bounded by previous CGSCC inlining. With context-sensitive profile, FDO inliner is no longer limited by previous inlining, so we need to take size into account to avoid significant size bloat.
- The way it looks at hotness is not accurate. It uses total samples in an inlinee as proxy for hotness, while what really matters for an inline decision is the call site count. This is an unfortunate fall back because call site count and callee entry count are not reliable due to dwarf based correlation, especially for inlinees. Now paired with pseudo-probe, we have accurate call site count and callee's entry count, so we can use that to gauge hotness more accurately.
- It treats all call sites from a block as hot as long as there's one call site considered hot. This is normally true, but since total samples is used as hotness proxy, this transitiveness within block magnifies the inacurate hotness heuristic. With pseduo-probe and the change above, this is no longer an issue for CSSPGO.
New FDO Inliner
Putting all the requirement for CSSPGO together, we need a top-down call site prioritized BFS inliner. Here're reasons why each component is needed.
- Top-down: We need a top-down inliner to better leverage context-sensitive profile, so inlining is driven by accurate context profile, and post-inline is also accurate. This is already implemented in https://reviews.llvm.org/D70655.
- Size Cap: For top-down inliner, taking function size into account for inline decision alone isn't sufficient to control size growth. We also need to explicitly cap size growth because with top-down inlining, we can grow inliner size significantly with large number of smaller inlinees even if each individually passes the cost/size check.
- Prioritize call sites: With size cap, inlining order also becomes important, because if we stop inlining due to size budget limit, we'd want to use budget towards the most beneficial call sites.
- BFS inline: Same as call site prioritization, if we stop inlining due to size budget limit, we want a balanced inline tree, rather than going deep on one call path.
Note that the new inliner avoids repeatedly evaluating same set of call site, so it should help with compile time too. For this reason, we could transition today's FDO inliner to use a queue with equal priority to avoid wasted reevaluation of same call site (TODO).
Speculative indirect call promotion and inlining is also supported now with CSSPGO just like baseline AutoFDO.
Tunings and knobs
I created tuning knobs for size growth/cap control, and for hot threshold separate from CGSCC inliner. The default values are selected based on initial tuning with CSSPGO.
Results
Evaluated with an internal LLVM fork couple months ago, plus another change to adjust hot-threshold cutoff for context profile (will send up after this one), the new inliner show ~1% geomean perf win on spec2006 with CSSPGO, while reducing code size too. The measurement was done using train-train setup, MonoLTO w/ new pass manager and pseudo-probe. Note that this is just a starting point - we hope that the new inliner will open up more opportunity with CSSPGO, but it will certainly take more time and effort to make it fully calibrated and ready for bigger workloads (we're working on it).
Differential Revision: https://reviews.llvm.org/D94001
2021-01-04 08:43:06 +08:00
!llvm.dbg.cu = ! { !2 }
!llvm.module.flags = ! { !14 , !15 , !16 }
!llvm.ident = ! { !17 }
!0 = !DIGlobalVariableExpression ( var: !1 , expr: !DIExpression ( ) )
!1 = distinct !DIGlobalVariable ( name: "factor" , scope: !2 , file: !3 , line: 21 , type: !13 , isLocal: false , isDefinition: true )
!2 = distinct !DICompileUnit ( language: D W _ L A N G _ C _ p l u s _ p l u s _ 14 , file: !3 , producer: "clang version 11.0.0" , isOptimized: true , runtimeVersion: 0 , emissionKind: F u l l D e b u g , enums: !4 , retainedTypes: !5 , globals: !12 , splitDebugInlining: false , debugInfoForProfiling: true , nameTableKind: N one )
!3 = !DIFile ( filename: "merged.cpp" , directory: "/local/autofdo" )
!4 = ! { }
!5 = ! { !6 , !10 , !11 }
!6 = !DISubprogram ( name: "funcA" , linkageName: "_Z5funcAi" , scope: !3 , file: !3 , line: 6 , type: !7 , flags: D I F l a g P r o t o t y p e d , spFlags: D I S P F l a g O p t i m i z e d , retainedNodes: !4 )
!7 = !DISubroutineType ( types: !8 )
!8 = ! { !9 , !9 }
!9 = !DIBasicType ( name: "int" , size: 32 , encoding: D W _ A T E _ s i g n e d )
!10 = !DISubprogram ( name: "funcB" , linkageName: "_Z5funcBi" , scope: !3 , file: !3 , line: 7 , type: !7 , flags: D I F l a g P r o t o t y p e d , spFlags: D I S P F l a g O p t i m i z e d , retainedNodes: !4 )
!11 = !DISubprogram ( name: "funcLeaf" , linkageName: "_Z8funcLeafi" , scope: !3 , file: !3 , line: 22 , type: !7 , flags: D I F l a g P r o t o t y p e d , spFlags: D I S P F l a g O p t i m i z e d , retainedNodes: !4 )
!12 = ! { !0 }
!13 = !DIDerivedType ( tag: D W _ T A G _ v o l a t i l e _ type , baseType: !9 )
!14 = ! { i32 7 , !"Dwarf Version" , i32 4 }
!15 = ! { i32 2 , !"Debug Info Version" , i32 3 }
!16 = ! { i32 1 , !"wchar_size" , i32 4 }
!17 = ! { !"clang version 11.0.0" }
!18 = distinct !DISubprogram ( name: "main" , scope: !3 , file: !3 , line: 11 , type: !19 , scopeLine: 11 , flags: D I F l a g P r o t o t y p e d | D I F l a g A l l C a l l s D e s c r i b e d , spFlags: D I S P F l a g D e f i n i t i o n | D I S P F l a g O p t i m i z e d , unit: !2 , retainedNodes: !21 )
!19 = !DISubroutineType ( types: !20 )
!20 = ! { !9 }
!21 = ! { !22 , !23 }
!22 = !DILocalVariable ( name: "r" , scope: !18 , file: !3 , line: 12 , type: !9 )
!23 = !DILocalVariable ( name: "x" , scope: !24 , file: !3 , line: 13 , type: !9 )
!24 = distinct !DILexicalBlock ( scope: !18 , file: !3 , line: 13 , column: 3 )
!25 = !DILocation ( line: 13 , column: 3 , scope: !26 )
!26 = !DILexicalBlockFile ( scope: !24 , file: !3 , discriminator: 2 )
!27 = !DILocation ( line: 17 , column: 3 , scope: !18 )
!28 = !DILocation ( line: 14 , column: 10 , scope: !29 )
!29 = distinct !DILexicalBlock ( scope: !30 , file: !3 , line: 13 , column: 37 )
!30 = distinct !DILexicalBlock ( scope: !24 , file: !3 , line: 13 , column: 3 )
!31 = !DILocation ( line: 14 , column: 29 , scope: !29 )
!32 = !DILocation ( line: 14 , column: 21 , scope: !33 )
!33 = !DILexicalBlockFile ( scope: !29 , file: !3 , discriminator: 2 )
!34 = !DILocation ( line: 14 , column: 19 , scope: !29 )
!35 = !DILocation ( line: 14 , column: 7 , scope: !29 )
!36 = !DILocation ( line: 13 , column: 33 , scope: !37 )
!37 = !DILexicalBlockFile ( scope: !30 , file: !3 , discriminator: 6 )
!38 = !DILocation ( line: 13 , column: 26 , scope: !39 )
!39 = !DILexicalBlockFile ( scope: !30 , file: !3 , discriminator: 2 )
!40 = distinct !DISubprogram ( name: "funcA" , linkageName: "_Z5funcAi" , scope: !3 , file: !3 , line: 26 , type: !7 , scopeLine: 26 , flags: D I F l a g P r o t o t y p e d | D I F l a g A l l C a l l s D e s c r i b e d , spFlags: D I S P F l a g D e f i n i t i o n | D I S P F l a g O p t i m i z e d , unit: !2 )
!44 = !DILocation ( line: 27 , column: 22 , scope: !40 )
!45 = !DILocation ( line: 27 , column: 11 , scope: !40 )
!46 = !DILocation ( line: 29 , column: 3 , scope: !40 )
!47 = distinct !DISubprogram ( name: "funcB" , linkageName: "_Z5funcBi" , scope: !3 , file: !3 , line: 32 , type: !7 , scopeLine: 32 , flags: D I F l a g P r o t o t y p e d | D I F l a g A l l C a l l s D e s c r i b e d , spFlags: D I S P F l a g D e f i n i t i o n | D I S P F l a g O p t i m i z e d , unit: !2 )
!51 = !DILocation ( line: 33 , column: 22 , scope: !47 )
!52 = !DILocation ( line: 33 , column: 11 , scope: !47 )
!53 = !DILocation ( line: 35 , column: 3 , scope: !47 )
!54 = distinct !DISubprogram ( name: "funcLeaf" , linkageName: "_Z8funcLeafi" , scope: !3 , file: !3 , line: 48 , type: !7 , scopeLine: 48 , flags: D I F l a g P r o t o t y p e d | D I F l a g A l l C a l l s D e s c r i b e d , spFlags: D I S P F l a g D e f i n i t i o n | D I S P F l a g O p t i m i z e d , unit: !2 )
!57 = !DILocation ( line: 49 , column: 9 , scope: !58 )
!58 = distinct !DILexicalBlock ( scope: !54 , file: !3 , line: 49 , column: 7 )
!59 = !DILocation ( line: 49 , column: 7 , scope: !54 )
!60 = !DILocation ( line: 58 , column: 14 , scope: !61 )
!61 = !DILexicalBlockFile ( scope: !62 , file: !3 , discriminator: 2 )
!62 = distinct !DILexicalBlock ( scope: !58 , file: !3 , line: 56 , column: 8 )
!63 = !DILocation ( line: 58 , column: 5 , scope: !61 )
!64 = !DILocation ( line: 52 , column: 16 , scope: !65 )
!65 = distinct !DILexicalBlock ( scope: !66 , file: !3 , line: 51 , column: 19 )
!66 = distinct !DILexicalBlock ( scope: !58 , file: !3 , line: 49 , column: 14 )
!67 = !DILocation ( line: 52 , column: 12 , scope: !65 )
!68 = !DILocation ( line: 52 , column: 9 , scope: !65 )
!69 = !DILocation ( line: 51 , column: 14 , scope: !70 )
!70 = !DILexicalBlockFile ( scope: !66 , file: !3 , discriminator: 2 )
!71 = !DILocation ( line: 51 , column: 5 , scope: !70 )
!72 = !DILocation ( line: 59 , column: 16 , scope: !73 )
!73 = distinct !DILexicalBlock ( scope: !62 , file: !3 , line: 58 , column: 19 )
!74 = !DILocation ( line: 59 , column: 12 , scope: !73 )
!75 = !DILocation ( line: 59 , column: 9 , scope: !73 )
!76 = !DILocation ( line: 63 , column: 3 , scope: !54 )
