maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/llvm/test/Transforms/HardwareLoops/loop-guards.ll

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

351 lines
15 KiB
LLVM
Raw Normal View History

[HardwareLoops] Loop counter guard intrinsic Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero. Differential Revision: https://reviews.llvm.org/D63809 llvm-svn: 364628
2019-06-28 15:38:16 +08:00
; RUN: opt -hardware-loops -force-hardware-loops=true -hardware-loop-decrement=1 -hardware-loop-counter-bitwidth=32 -force-hardware-loop-guard=true -S %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-EXIT
; RUN: opt -hardware-loops -force-hardware-loops=true -hardware-loop-decrement=1 -hardware-loop-counter-bitwidth=32 -force-hardware-loop-guard=true -force-hardware-loop-phi=true -S %s -o - | FileCheck %s --check-prefix=CHECK --check-prefix=CHECK-LATCH
; RUN: opt -hardware-loops -force-hardware-loops=true -hardware-loop-decrement=1 -hardware-loop-counter-bitwidth=32 -force-hardware-loop-guard=false -S %s -o - | FileCheck %s --check-prefix=NO-GUARD
; NO-GUARD-NOT: @llvm.test.set.loop.iterations
; CHECK-LABEL: test1
; CHECK: entry:
[NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions These intrinsics, not the icmp+select are the canonical form nowadays, so we might as well directly emit them. This should not cause any regressions, but if it does, then then they would needed to be fixed regardless. Note that this doesn't deal with `SCEVExpander::isHighCostExpansion()`, but that is a pessimization, not a correctness issue. Additionally, the non-intrinsic form has issues with undef, see https://reviews.llvm.org/D88287#2587863
2021-02-26 21:48:58 +08:00
; CHECK: [[MAX:%[^ ]+]] = call i32 @llvm.umax.i32(i32 %N, i32 2)
[HardwareLoops] Loop counter guard intrinsic Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero. Differential Revision: https://reviews.llvm.org/D63809 llvm-svn: 364628
2019-06-28 15:38:16 +08:00
; CHECK: [[COUNT:%[^ ]+]] = add i32 [[MAX]], -1
; CHECK: br i1 %t1, label %do.body.preheader
; CHECK: do.body.preheader:
[ARM] Alter t2DoLoopStart to define lr This changes the definition of t2DoLoopStart from t2DoLoopStart rGPR to GPRlr = t2DoLoopStart rGPR This will hopefully mean that low overhead loops are more tied together, and we can more reliably generate loops without reverting or being at the whims of the register allocator. This is a fairly simple change in itself, but leads to a number of other required alterations. - The hardware loop pass, if UsePhi is set, now generates loops of the form: %start = llvm.start.loop.iterations(%N) loop: %p = phi [%start], [%dec] %dec = llvm.loop.decrement.reg(%p, 1) %c = icmp ne %dec, 0 br %c, loop, exit - For this a new llvm.start.loop.iterations intrinsic was added, identical to llvm.set.loop.iterations but produces a value as seen above, gluing the loop together more through def-use chains. - This new instrinsic conceptually produces the same output as input, which is taught to SCEV so that the checks in MVETailPredication are not affected. - Some minor changes are needed to the ARMLowOverheadLoop pass, but it has been left mostly as before. We should now more reliably be able to tell that the t2DoLoopStart is correct without having to prove it, but t2WhileLoopStart and tail-predicated loops will remain the same. - And all the tests have been updated. There are a lot of them! This patch on it's own might cause more trouble that it helps, with more tail-predicated loops being reverted, but some additional patches can hopefully improve upon that to get to something that is better overall. Differential Revision: https://reviews.llvm.org/D89881
2020-11-10 23:57:58 +08:00
; CHECK-EXIT: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
; CHECK-LATCH: call i32 @llvm.start.loop.iterations.i32(i32 [[COUNT]])
[HardwareLoops] Loop counter guard intrinsic Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero. Differential Revision: https://reviews.llvm.org/D63809 llvm-svn: 364628
2019-06-28 15:38:16 +08:00
; CHECK: br label %do.body
define void @test1(i1 zeroext %t1, i32* nocapture %a, i32* nocapture readonly %b, i32 %N) {
entry:
br i1 %t1, label %do.body, label %if.end
do.body: ; preds = %do.body, %entry
%b.addr.0 = phi i32* [ %incdec.ptr, %do.body ], [ %b, %entry ]
%a.addr.0 = phi i32* [ %incdec.ptr1, %do.body ], [ %a, %entry ]
%i.0 = phi i32 [ %inc, %do.body ], [ 1, %entry ]
%incdec.ptr = getelementptr inbounds i32, i32* %b.addr.0, i32 1
%tmp = load i32, i32* %b.addr.0, align 4
%incdec.ptr1 = getelementptr inbounds i32, i32* %a.addr.0, i32 1
store i32 %tmp, i32* %a.addr.0, align 4
%inc = add nuw i32 %i.0, 1
%cmp = icmp ult i32 %inc, %N
br i1 %cmp, label %do.body, label %if.end
if.end: ; preds = %do.body, %entry
ret void
}
; CHECK-LABEL: test2
; CHECK-NOT: call i1 @llvm.test.set.loop.iterations
; CHECK-NOT: call void @llvm.set.loop.iterations
[ARM] Alter t2DoLoopStart to define lr This changes the definition of t2DoLoopStart from t2DoLoopStart rGPR to GPRlr = t2DoLoopStart rGPR This will hopefully mean that low overhead loops are more tied together, and we can more reliably generate loops without reverting or being at the whims of the register allocator. This is a fairly simple change in itself, but leads to a number of other required alterations. - The hardware loop pass, if UsePhi is set, now generates loops of the form: %start = llvm.start.loop.iterations(%N) loop: %p = phi [%start], [%dec] %dec = llvm.loop.decrement.reg(%p, 1) %c = icmp ne %dec, 0 br %c, loop, exit - For this a new llvm.start.loop.iterations intrinsic was added, identical to llvm.set.loop.iterations but produces a value as seen above, gluing the loop together more through def-use chains. - This new instrinsic conceptually produces the same output as input, which is taught to SCEV so that the checks in MVETailPredication are not affected. - Some minor changes are needed to the ARMLowOverheadLoop pass, but it has been left mostly as before. We should now more reliably be able to tell that the t2DoLoopStart is correct without having to prove it, but t2WhileLoopStart and tail-predicated loops will remain the same. - And all the tests have been updated. There are a lot of them! This patch on it's own might cause more trouble that it helps, with more tail-predicated loops being reverted, but some additional patches can hopefully improve upon that to get to something that is better overall. Differential Revision: https://reviews.llvm.org/D89881
2020-11-10 23:57:58 +08:00
; CHECK-NOT: call i32 @llvm.start.loop.iterations
[HardwareLoops] Loop counter guard intrinsic Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero. Differential Revision: https://reviews.llvm.org/D63809 llvm-svn: 364628
2019-06-28 15:38:16 +08:00
define void @test2(i1 zeroext %t1, i32* nocapture %a, i32* nocapture readonly %b, i32 %N) {
entry:
br i1 %t1, label %do.body, label %if.end
do.body: ; preds = %do.body, %entry
%b.addr.0 = phi i32* [ %incdec.ptr, %do.body ], [ %b, %entry ]
%a.addr.0 = phi i32* [ %incdec.ptr1, %do.body ], [ %a, %entry ]
%i.0 = phi i32 [ %add, %do.body ], [ 1, %entry ]
%incdec.ptr = getelementptr inbounds i32, i32* %b.addr.0, i32 1
%tmp = load i32, i32* %b.addr.0, align 4
%incdec.ptr1 = getelementptr inbounds i32, i32* %a.addr.0, i32 1
store i32 %tmp, i32* %a.addr.0, align 4
%add = add i32 %i.0, 2
%cmp = icmp ult i32 %add, %N
br i1 %cmp, label %do.body, label %if.end
if.end: ; preds = %do.body, %entry
ret void
}
; CHECK-LABEL: test3
; CHECK: entry:
[NFCI] SCEVExpander: emit intrinsics for integral {u,s}{min,max} SCEV expressions These intrinsics, not the icmp+select are the canonical form nowadays, so we might as well directly emit them. This should not cause any regressions, but if it does, then then they would needed to be fixed regardless. Note that this doesn't deal with `SCEVExpander::isHighCostExpansion()`, but that is a pessimization, not a correctness issue. Additionally, the non-intrinsic form has issues with undef, see https://reviews.llvm.org/D88287#2587863
2021-02-26 21:48:58 +08:00
; CHECK: [[COUNT:%[^ ]+]] = call i32 @llvm.umax.i32(i32 %N, i32 1)
[HardwareLoops] Loop counter guard intrinsic Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero. Differential Revision: https://reviews.llvm.org/D63809 llvm-svn: 364628
2019-06-28 15:38:16 +08:00
; CHECK: br i1 %brmerge.demorgan, label %do.body.preheader
; CHECK: do.body.preheader:
[ARM] Alter t2DoLoopStart to define lr This changes the definition of t2DoLoopStart from t2DoLoopStart rGPR to GPRlr = t2DoLoopStart rGPR This will hopefully mean that low overhead loops are more tied together, and we can more reliably generate loops without reverting or being at the whims of the register allocator. This is a fairly simple change in itself, but leads to a number of other required alterations. - The hardware loop pass, if UsePhi is set, now generates loops of the form: %start = llvm.start.loop.iterations(%N) loop: %p = phi [%start], [%dec] %dec = llvm.loop.decrement.reg(%p, 1) %c = icmp ne %dec, 0 br %c, loop, exit - For this a new llvm.start.loop.iterations intrinsic was added, identical to llvm.set.loop.iterations but produces a value as seen above, gluing the loop together more through def-use chains. - This new instrinsic conceptually produces the same output as input, which is taught to SCEV so that the checks in MVETailPredication are not affected. - Some minor changes are needed to the ARMLowOverheadLoop pass, but it has been left mostly as before. We should now more reliably be able to tell that the t2DoLoopStart is correct without having to prove it, but t2WhileLoopStart and tail-predicated loops will remain the same. - And all the tests have been updated. There are a lot of them! This patch on it's own might cause more trouble that it helps, with more tail-predicated loops being reverted, but some additional patches can hopefully improve upon that to get to something that is better overall. Differential Revision: https://reviews.llvm.org/D89881
2020-11-10 23:57:58 +08:00
; CHECK-EXIT: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
; CHECK-LATCH: call i32 @llvm.start.loop.iterations.i32(i32 [[COUNT]])
[HardwareLoops] Loop counter guard intrinsic Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero. Differential Revision: https://reviews.llvm.org/D63809 llvm-svn: 364628
2019-06-28 15:38:16 +08:00
; CHECK: br label %do.body
define void @test3(i1 zeroext %t1, i1 zeroext %t2, i32* nocapture %a, i32* nocapture readonly %b, i32 %N) {
entry:
%brmerge.demorgan = and i1 %t1, %t2
br i1 %brmerge.demorgan, label %do.body, label %if.end
do.body: ; preds = %do.body, %entry
%b.addr.0 = phi i32* [ %incdec.ptr, %do.body ], [ %b, %entry ]
%a.addr.0 = phi i32* [ %incdec.ptr3, %do.body ], [ %a, %entry ]
%i.0 = phi i32 [ %inc, %do.body ], [ 0, %entry ]
%incdec.ptr = getelementptr inbounds i32, i32* %b.addr.0, i32 1
%tmp = load i32, i32* %b.addr.0, align 4
%incdec.ptr3 = getelementptr inbounds i32, i32* %a.addr.0, i32 1
store i32 %tmp, i32* %a.addr.0, align 4
%inc = add nuw i32 %i.0, 1
%cmp = icmp ult i32 %inc, %N
br i1 %cmp, label %do.body, label %if.end
if.end: ; preds = %do.body, %entry
ret void
}
; CHECK-LABEL: test4
; CHECK: entry:
; CHECK-LATCH: br i1 %brmerge.demorgan, label %while.cond
[ARM] Alter t2DoLoopStart to define lr This changes the definition of t2DoLoopStart from t2DoLoopStart rGPR to GPRlr = t2DoLoopStart rGPR This will hopefully mean that low overhead loops are more tied together, and we can more reliably generate loops without reverting or being at the whims of the register allocator. This is a fairly simple change in itself, but leads to a number of other required alterations. - The hardware loop pass, if UsePhi is set, now generates loops of the form: %start = llvm.start.loop.iterations(%N) loop: %p = phi [%start], [%dec] %dec = llvm.loop.decrement.reg(%p, 1) %c = icmp ne %dec, 0 br %c, loop, exit - For this a new llvm.start.loop.iterations intrinsic was added, identical to llvm.set.loop.iterations but produces a value as seen above, gluing the loop together more through def-use chains. - This new instrinsic conceptually produces the same output as input, which is taught to SCEV so that the checks in MVETailPredication are not affected. - Some minor changes are needed to the ARMLowOverheadLoop pass, but it has been left mostly as before. We should now more reliably be able to tell that the t2DoLoopStart is correct without having to prove it, but t2WhileLoopStart and tail-predicated loops will remain the same. - And all the tests have been updated. There are a lot of them! This patch on it's own might cause more trouble that it helps, with more tail-predicated loops being reverted, but some additional patches can hopefully improve upon that to get to something that is better overall. Differential Revision: https://reviews.llvm.org/D89881
2020-11-10 23:57:58 +08:00
; CHECK-LATCH-NOT: @llvm{{.*}}loop.iterations
[HardwareLoops] Loop counter guard intrinsic Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero. Differential Revision: https://reviews.llvm.org/D63809 llvm-svn: 364628
2019-06-28 15:38:16 +08:00
; CHECK-EXIT: br i1 %brmerge.demorgan, label %while.cond.preheader
; CHECK-EXIT: while.cond.preheader:
; CHECK-EXIT: [[COUNT:%[^ ]+]] = add i32 %N, 1
; CHECK-EXIT: call void @llvm.set.loop.iterations.i32(i32 [[COUNT]])
; CHECK-EXIT: br label %while.cond
define void @test4(i1 zeroext %t1, i1 zeroext %t2, i32* nocapture %a, i32* nocapture readonly %b, i32 %N) {
entry:
%brmerge.demorgan = and i1 %t1, %t2
br i1 %brmerge.demorgan, label %while.cond, label %if.end
while.cond: ; preds = %while.body, %entry
%b.addr.0 = phi i32* [ %incdec.ptr, %while.body ], [ %b, %entry ]
%a.addr.0 = phi i32* [ %incdec.ptr3, %while.body ], [ %a, %entry ]
%i.0 = phi i32 [ %inc, %while.body ], [ 0, %entry ]
%exitcond = icmp eq i32 %i.0, %N
br i1 %exitcond, label %if.end, label %while.body
while.body: ; preds = %while.cond
%incdec.ptr = getelementptr inbounds i32, i32* %b.addr.0, i32 1
%tmp = load i32, i32* %b.addr.0, align 4
%incdec.ptr3 = getelementptr inbounds i32, i32* %a.addr.0, i32 1
store i32 %tmp, i32* %a.addr.0, align 4
%inc = add i32 %i.0, 1
br label %while.cond
if.end: ; preds = %while.cond, %entry
ret void
}
; CHECK-LABEL: test5
; CHECK: entry:
; CHECK: br i1 %or.cond, label %while.body.preheader
; CHECK: while.body.preheader:
[ARM] Alter t2DoLoopStart to define lr This changes the definition of t2DoLoopStart from t2DoLoopStart rGPR to GPRlr = t2DoLoopStart rGPR This will hopefully mean that low overhead loops are more tied together, and we can more reliably generate loops without reverting or being at the whims of the register allocator. This is a fairly simple change in itself, but leads to a number of other required alterations. - The hardware loop pass, if UsePhi is set, now generates loops of the form: %start = llvm.start.loop.iterations(%N) loop: %p = phi [%start], [%dec] %dec = llvm.loop.decrement.reg(%p, 1) %c = icmp ne %dec, 0 br %c, loop, exit - For this a new llvm.start.loop.iterations intrinsic was added, identical to llvm.set.loop.iterations but produces a value as seen above, gluing the loop together more through def-use chains. - This new instrinsic conceptually produces the same output as input, which is taught to SCEV so that the checks in MVETailPredication are not affected. - Some minor changes are needed to the ARMLowOverheadLoop pass, but it has been left mostly as before. We should now more reliably be able to tell that the t2DoLoopStart is correct without having to prove it, but t2WhileLoopStart and tail-predicated loops will remain the same. - And all the tests have been updated. There are a lot of them! This patch on it's own might cause more trouble that it helps, with more tail-predicated loops being reverted, but some additional patches can hopefully improve upon that to get to something that is better overall. Differential Revision: https://reviews.llvm.org/D89881
2020-11-10 23:57:58 +08:00
; CHECK-EXIT: call void @llvm.set.loop.iterations.i32(i32 %N)
; CHECK-LATCH: call i32 @llvm.start.loop.iterations.i32(i32 %N)
[HardwareLoops] Loop counter guard intrinsic Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero. Differential Revision: https://reviews.llvm.org/D63809 llvm-svn: 364628
2019-06-28 15:38:16 +08:00
; CHECK: br label %while.body
define void @test5(i1 zeroext %t1, i1 zeroext %t2, i32* nocapture %a, i32* nocapture readonly %b, i32 %N) {
entry:
%brmerge.demorgan = and i1 %t1, %t2
%cmp6 = icmp ne i32 %N, 0
%or.cond = and i1 %brmerge.demorgan, %cmp6
br i1 %or.cond, label %while.body, label %if.end
while.body: ; preds = %while.body, %entry
%i.09 = phi i32 [ %inc, %while.body ], [ 0, %entry ]
%a.addr.08 = phi i32* [ %incdec.ptr3, %while.body ], [ %a, %entry ]
%b.addr.07 = phi i32* [ %incdec.ptr, %while.body ], [ %b, %entry ]
%incdec.ptr = getelementptr inbounds i32, i32* %b.addr.07, i32 1
%tmp = load i32, i32* %b.addr.07, align 4
%incdec.ptr3 = getelementptr inbounds i32, i32* %a.addr.08, i32 1
store i32 %tmp, i32* %a.addr.08, align 4
%inc = add nuw i32 %i.09, 1
%exitcond = icmp eq i32 %inc, %N
br i1 %exitcond, label %if.end, label %while.body
if.end: ; preds = %while.body, %entry
ret void
}
; CHECK-LABEL: test6
; CHECK: entry:
; CHECK: br i1 %brmerge.demorgan, label %while.preheader
; CHECK: while.preheader:
[ARM] Improve WLS lowering Recently we improved the lowering of low overhead loops and tail predicated loops, but concentrated first on the DLS do style loops. This extends those improvements over to the WLS while loops, improving the chance of lowering them successfully. To do this the lowering has to change a little as the instructions are terminators that produce a value - something that needs to be treated carefully. Lowering starts at the Hardware Loop pass, inserting a new llvm.test.start.loop.iterations that produces both an i1 to control the loop entry and an i32 similar to the llvm.start.loop.iterations intrinsic added for do loops. This feeds into the loop phi, properly gluing the values together: %wls = call { i32, i1 } @llvm.test.start.loop.iterations.i32(i32 %div) %wls0 = extractvalue { i32, i1 } %wls, 0 %wls1 = extractvalue { i32, i1 } %wls, 1 br i1 %wls1, label %loop.ph, label %loop.exit ... loop: %lsr.iv = phi i32 [ %wls0, %loop.ph ], [ %iv.next, %loop ] .. %iv.next = call i32 @llvm.loop.decrement.reg.i32(i32 %lsr.iv, i32 1) %cmp = icmp ne i32 %iv.next, 0 br i1 %cmp, label %loop, label %loop.exit The llvm.test.start.loop.iterations need to be lowered through ISel lowering as a pair of WLS and WLSSETUP nodes, which each get converted to t2WhileLoopSetup and t2WhileLoopStart Pseudos. This helps prevent t2WhileLoopStart from being a terminator that produces a value, something difficult to control at that stage in the pipeline. Instead the t2WhileLoopSetup produces the value of LR (essentially acting as a lr = subs rn, 0), t2WhileLoopStart consumes that lr value (the Bcc). These are then converted into a single t2WhileLoopStartLR at the same point as t2DoLoopStartTP and t2LoopEndDec. Otherwise we revert the loop to prevent them from progressing further in the pipeline. The t2WhileLoopStartLR is a single instruction that takes a GPR and produces LR, similar to the WLS instruction. %1:gprlr = t2WhileLoopStartLR %0:rgpr, %bb.3 t2B %bb.1 ... bb.2.loop: %2:gprlr = PHI %1:gprlr, %bb.1, %3:gprlr, %bb.2 ... %3:gprlr = t2LoopEndDec %2:gprlr, %bb.2 t2B %bb.3 The t2WhileLoopStartLR can then be treated similar to the other low overhead loop pseudos, eventually being lowered to a WLS providing the branches are within range. Differential Revision: https://reviews.llvm.org/D97729
2021-03-11 22:06:04 +08:00
; CHECK-EXIT: [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 %N)
; CHECK-LATCH: [[TEST1:%[^ ]+]] = call { i32, i1 } @llvm.test.start.loop.iterations.i32(i32 %N)
; CHECK-LATCH: [[TEST:%[^ ]+]] = extractvalue { i32, i1 } [[TEST1]], 1
[HardwareLoops] Loop counter guard intrinsic Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero. Differential Revision: https://reviews.llvm.org/D63809 llvm-svn: 364628
2019-06-28 15:38:16 +08:00
; CHECK: br i1 [[TEST]], label %while.body.preheader, label %if.end
; CHECK: while.body.preheader:
; CHECK: br label %while.body
define void @test6(i1 zeroext %t1, i1 zeroext %t2, i32* nocapture %a, i32* nocapture readonly %b, i32 %N) {
entry:
%brmerge.demorgan = and i1 %t1, %t2
br i1 %brmerge.demorgan, label %while.preheader, label %if.end
while.preheader: ; preds = %entry
%cmp = icmp ne i32 %N, 0
br i1 %cmp, label %while.body, label %if.end
while.body: ; preds = %while.body, %while.preheader
%i.09 = phi i32 [ %inc, %while.body ], [ 0, %while.preheader ]
%a.addr.08 = phi i32* [ %incdec.ptr3, %while.body ], [ %a, %while.preheader ]
%b.addr.07 = phi i32* [ %incdec.ptr, %while.body ], [ %b, %while.preheader ]
%incdec.ptr = getelementptr inbounds i32, i32* %b.addr.07, i32 1
%tmp = load i32, i32* %b.addr.07, align 4
%incdec.ptr3 = getelementptr inbounds i32, i32* %a.addr.08, i32 1
store i32 %tmp, i32* %a.addr.08, align 4
%inc = add nuw i32 %i.09, 1
%exitcond = icmp eq i32 %inc, %N
br i1 %exitcond, label %if.end, label %while.body
if.end: ; preds = %while.body, %while.preheader, %entry
ret void
}
; CHECK-LABEL: test7
; CHECK: entry:
; CHECK: br i1 %brmerge.demorgan, label %while.preheader
; CHECK: while.preheader:
[ARM] Improve WLS lowering Recently we improved the lowering of low overhead loops and tail predicated loops, but concentrated first on the DLS do style loops. This extends those improvements over to the WLS while loops, improving the chance of lowering them successfully. To do this the lowering has to change a little as the instructions are terminators that produce a value - something that needs to be treated carefully. Lowering starts at the Hardware Loop pass, inserting a new llvm.test.start.loop.iterations that produces both an i1 to control the loop entry and an i32 similar to the llvm.start.loop.iterations intrinsic added for do loops. This feeds into the loop phi, properly gluing the values together: %wls = call { i32, i1 } @llvm.test.start.loop.iterations.i32(i32 %div) %wls0 = extractvalue { i32, i1 } %wls, 0 %wls1 = extractvalue { i32, i1 } %wls, 1 br i1 %wls1, label %loop.ph, label %loop.exit ... loop: %lsr.iv = phi i32 [ %wls0, %loop.ph ], [ %iv.next, %loop ] .. %iv.next = call i32 @llvm.loop.decrement.reg.i32(i32 %lsr.iv, i32 1) %cmp = icmp ne i32 %iv.next, 0 br i1 %cmp, label %loop, label %loop.exit The llvm.test.start.loop.iterations need to be lowered through ISel lowering as a pair of WLS and WLSSETUP nodes, which each get converted to t2WhileLoopSetup and t2WhileLoopStart Pseudos. This helps prevent t2WhileLoopStart from being a terminator that produces a value, something difficult to control at that stage in the pipeline. Instead the t2WhileLoopSetup produces the value of LR (essentially acting as a lr = subs rn, 0), t2WhileLoopStart consumes that lr value (the Bcc). These are then converted into a single t2WhileLoopStartLR at the same point as t2DoLoopStartTP and t2LoopEndDec. Otherwise we revert the loop to prevent them from progressing further in the pipeline. The t2WhileLoopStartLR is a single instruction that takes a GPR and produces LR, similar to the WLS instruction. %1:gprlr = t2WhileLoopStartLR %0:rgpr, %bb.3 t2B %bb.1 ... bb.2.loop: %2:gprlr = PHI %1:gprlr, %bb.1, %3:gprlr, %bb.2 ... %3:gprlr = t2LoopEndDec %2:gprlr, %bb.2 t2B %bb.3 The t2WhileLoopStartLR can then be treated similar to the other low overhead loop pseudos, eventually being lowered to a WLS providing the branches are within range. Differential Revision: https://reviews.llvm.org/D97729
2021-03-11 22:06:04 +08:00
; CHECK-EXIT: [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 %N)
; CHECK-LATCH: [[TEST1:%[^ ]+]] = call { i32, i1 } @llvm.test.start.loop.iterations.i32(i32 %N)
; CHECK-LATCH: [[TEST:%[^ ]+]] = extractvalue { i32, i1 } [[TEST1]], 1
[HardwareLoops] Loop counter guard intrinsic Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero. Differential Revision: https://reviews.llvm.org/D63809 llvm-svn: 364628
2019-06-28 15:38:16 +08:00
; CHECK: br i1 [[TEST]], label %while.body.preheader, label %if.end
; CHECK: while.body.preheader:
; CHECK: br label %while.body
define void @test7(i1 zeroext %t1, i1 zeroext %t2, i32* nocapture %a, i32* nocapture readonly %b, i32 %N) {
entry:
%brmerge.demorgan = and i1 %t1, %t2
br i1 %brmerge.demorgan, label %while.preheader, label %if.end
while.preheader: ; preds = %entry
%cmp = icmp eq i32 %N, 0
br i1 %cmp, label %if.end, label %while.body
while.body: ; preds = %while.body, %while.preheader
%i.09 = phi i32 [ %inc, %while.body ], [ 0, %while.preheader ]
%a.addr.08 = phi i32* [ %incdec.ptr3, %while.body ], [ %a, %while.preheader ]
%b.addr.07 = phi i32* [ %incdec.ptr, %while.body ], [ %b, %while.preheader ]
%incdec.ptr = getelementptr inbounds i32, i32* %b.addr.07, i32 1
%tmp = load i32, i32* %b.addr.07, align 4
%incdec.ptr3 = getelementptr inbounds i32, i32* %a.addr.08, i32 1
store i32 %tmp, i32* %a.addr.08, align 4
%inc = add nuw i32 %i.09, 1
%exitcond = icmp eq i32 %inc, %N
br i1 %exitcond, label %if.end, label %while.body
if.end: ; preds = %while.body, %while.preheader, %entry
ret void
}
; TODO: Can we rearrange the conditional blocks so that we can use the test form?
; CHECK-LABEL: test8
; CHECK: entry:
; CHECK: [[CMP:%[^ ]+]] = icmp ne i32 %N, 0
; CHECK: br i1 [[CMP]], label %while.preheader
; CHECK: while.preheader:
; CHECK: br i1 %brmerge.demorgan, label %while.body.preheader
; CHECK: while.body.preheader:
[ARM] Alter t2DoLoopStart to define lr This changes the definition of t2DoLoopStart from t2DoLoopStart rGPR to GPRlr = t2DoLoopStart rGPR This will hopefully mean that low overhead loops are more tied together, and we can more reliably generate loops without reverting or being at the whims of the register allocator. This is a fairly simple change in itself, but leads to a number of other required alterations. - The hardware loop pass, if UsePhi is set, now generates loops of the form: %start = llvm.start.loop.iterations(%N) loop: %p = phi [%start], [%dec] %dec = llvm.loop.decrement.reg(%p, 1) %c = icmp ne %dec, 0 br %c, loop, exit - For this a new llvm.start.loop.iterations intrinsic was added, identical to llvm.set.loop.iterations but produces a value as seen above, gluing the loop together more through def-use chains. - This new instrinsic conceptually produces the same output as input, which is taught to SCEV so that the checks in MVETailPredication are not affected. - Some minor changes are needed to the ARMLowOverheadLoop pass, but it has been left mostly as before. We should now more reliably be able to tell that the t2DoLoopStart is correct without having to prove it, but t2WhileLoopStart and tail-predicated loops will remain the same. - And all the tests have been updated. There are a lot of them! This patch on it's own might cause more trouble that it helps, with more tail-predicated loops being reverted, but some additional patches can hopefully improve upon that to get to something that is better overall. Differential Revision: https://reviews.llvm.org/D89881
2020-11-10 23:57:58 +08:00
; CHECK-EXIT: call void @llvm.set.loop.iterations.i32(i32 %N)
; CHECK-LATCH: call i32 @llvm.start.loop.iterations.i32(i32 %N)
[HardwareLoops] Loop counter guard intrinsic Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero. Differential Revision: https://reviews.llvm.org/D63809 llvm-svn: 364628
2019-06-28 15:38:16 +08:00
; CHECK: br label %while.body
define void @test8(i1 zeroext %t1, i1 zeroext %t2, i32* nocapture %a, i32* nocapture readonly %b, i32 %N) {
entry:
%cmp = icmp ne i32 %N, 0
br i1 %cmp, label %while.preheader, label %if.end
while.preheader: ; preds = %entry
%brmerge.demorgan = and i1 %t1, %t2
br i1 %brmerge.demorgan, label %while.body, label %if.end
while.body: ; preds = %while.body, %while.preheader
%i.09 = phi i32 [ %inc, %while.body ], [ 0, %while.preheader ]
%a.addr.08 = phi i32* [ %incdec.ptr3, %while.body ], [ %a, %while.preheader ]
%b.addr.07 = phi i32* [ %incdec.ptr, %while.body ], [ %b, %while.preheader ]
%incdec.ptr = getelementptr inbounds i32, i32* %b.addr.07, i32 1
%tmp = load i32, i32* %b.addr.07, align 4
%incdec.ptr3 = getelementptr inbounds i32, i32* %a.addr.08, i32 1
store i32 %tmp, i32* %a.addr.08, align 4
%inc = add nuw i32 %i.09, 1
%exitcond = icmp eq i32 %inc, %N
br i1 %exitcond, label %if.end, label %while.body
if.end: ; preds = %while.body, %while.preheader, %entry
ret void
}
; CHECK-LABEL: test9
; CHECK: entry:
; CHECK: br i1 %brmerge.demorgan, label %do.body.preheader
; CHECK: do.body.preheader:
[ARM] Alter t2DoLoopStart to define lr This changes the definition of t2DoLoopStart from t2DoLoopStart rGPR to GPRlr = t2DoLoopStart rGPR This will hopefully mean that low overhead loops are more tied together, and we can more reliably generate loops without reverting or being at the whims of the register allocator. This is a fairly simple change in itself, but leads to a number of other required alterations. - The hardware loop pass, if UsePhi is set, now generates loops of the form: %start = llvm.start.loop.iterations(%N) loop: %p = phi [%start], [%dec] %dec = llvm.loop.decrement.reg(%p, 1) %c = icmp ne %dec, 0 br %c, loop, exit - For this a new llvm.start.loop.iterations intrinsic was added, identical to llvm.set.loop.iterations but produces a value as seen above, gluing the loop together more through def-use chains. - This new instrinsic conceptually produces the same output as input, which is taught to SCEV so that the checks in MVETailPredication are not affected. - Some minor changes are needed to the ARMLowOverheadLoop pass, but it has been left mostly as before. We should now more reliably be able to tell that the t2DoLoopStart is correct without having to prove it, but t2WhileLoopStart and tail-predicated loops will remain the same. - And all the tests have been updated. There are a lot of them! This patch on it's own might cause more trouble that it helps, with more tail-predicated loops being reverted, but some additional patches can hopefully improve upon that to get to something that is better overall. Differential Revision: https://reviews.llvm.org/D89881
2020-11-10 23:57:58 +08:00
; CHECK-EXIT: call void @llvm.set.loop.iterations.i32(i32 %N)
; CHECK-LATCH: call i32 @llvm.start.loop.iterations.i32(i32 %N)
[HardwareLoops] Loop counter guard intrinsic Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero. Differential Revision: https://reviews.llvm.org/D63809 llvm-svn: 364628
2019-06-28 15:38:16 +08:00
; CHECK: br label %do.body
define void @test9(i1 zeroext %t1, i32* nocapture %a, i32* nocapture readonly %b, i32 %N) {
entry:
%cmp = icmp ne i32 %N, 0
%brmerge.demorgan = and i1 %t1, %cmp
br i1 %brmerge.demorgan, label %do.body, label %if.end
do.body: ; preds = %do.body, %entry
%b.addr.0 = phi i32* [ %incdec.ptr, %do.body ], [ %b, %entry ]
%a.addr.0 = phi i32* [ %incdec.ptr3, %do.body ], [ %a, %entry ]
%i.0 = phi i32 [ %inc, %do.body ], [ 0, %entry ]
%incdec.ptr = getelementptr inbounds i32, i32* %b.addr.0, i32 1
%tmp = load i32, i32* %b.addr.0, align 4
%incdec.ptr3 = getelementptr inbounds i32, i32* %a.addr.0, i32 1
store i32 %tmp, i32* %a.addr.0, align 4
%inc = add nuw i32 %i.0, 1
%cmp.1 = icmp ult i32 %inc, %N
br i1 %cmp.1, label %do.body, label %if.end
if.end: ; preds = %do.body, %entry
ret void
}
; CHECK-LABEL: test10
; CHECK: entry:
; CHECK: br i1 %cmp.1, label %do.body.preheader
; CHECK: do.body.preheader:
[ARM] Alter t2DoLoopStart to define lr This changes the definition of t2DoLoopStart from t2DoLoopStart rGPR to GPRlr = t2DoLoopStart rGPR This will hopefully mean that low overhead loops are more tied together, and we can more reliably generate loops without reverting or being at the whims of the register allocator. This is a fairly simple change in itself, but leads to a number of other required alterations. - The hardware loop pass, if UsePhi is set, now generates loops of the form: %start = llvm.start.loop.iterations(%N) loop: %p = phi [%start], [%dec] %dec = llvm.loop.decrement.reg(%p, 1) %c = icmp ne %dec, 0 br %c, loop, exit - For this a new llvm.start.loop.iterations intrinsic was added, identical to llvm.set.loop.iterations but produces a value as seen above, gluing the loop together more through def-use chains. - This new instrinsic conceptually produces the same output as input, which is taught to SCEV so that the checks in MVETailPredication are not affected. - Some minor changes are needed to the ARMLowOverheadLoop pass, but it has been left mostly as before. We should now more reliably be able to tell that the t2DoLoopStart is correct without having to prove it, but t2WhileLoopStart and tail-predicated loops will remain the same. - And all the tests have been updated. There are a lot of them! This patch on it's own might cause more trouble that it helps, with more tail-predicated loops being reverted, but some additional patches can hopefully improve upon that to get to something that is better overall. Differential Revision: https://reviews.llvm.org/D89881
2020-11-10 23:57:58 +08:00
; CHECK-EXIT: call void @llvm.set.loop.iterations.i32(i32
; CHECK-LATCH: call i32 @llvm.start.loop.iterations.i32(i32
[HardwareLoops] Loop counter guard intrinsic Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero. Differential Revision: https://reviews.llvm.org/D63809 llvm-svn: 364628
2019-06-28 15:38:16 +08:00
; CHECK: br label %do.body
define void @test10(i32* nocapture %a, i32* nocapture readonly %b, i32 %N) {
entry:
%cmp = icmp ne i32 %N, 0
%sub = sub i32 %N, 1
%be = select i1 %cmp, i32 0, i32 %sub
%cmp.1 = icmp ne i32 %be, 0
br i1 %cmp.1, label %do.body, label %if.end
do.body: ; preds = %do.body, %entry
%b.addr.0 = phi i32* [ %incdec.ptr, %do.body ], [ %b, %entry ]
%a.addr.0 = phi i32* [ %incdec.ptr3, %do.body ], [ %a, %entry ]
%i.0 = phi i32 [ %inc, %do.body ], [ 0, %entry ]
%incdec.ptr = getelementptr inbounds i32, i32* %b.addr.0, i32 1
%tmp = load i32, i32* %b.addr.0, align 4
%incdec.ptr3 = getelementptr inbounds i32, i32* %a.addr.0, i32 1
store i32 %tmp, i32* %a.addr.0, align 4
%inc = add nuw i32 %i.0, 1
%cmp.2 = icmp ult i32 %inc, %N
br i1 %cmp.2, label %do.body, label %if.end
if.end: ; preds = %do.body, %entry
ret void
}
; CHECK-LABEL: test11
; CHECK: entry:
; CHECK: br label %do.body.preheader
; CHECK: do.body.preheader:
[ARM] Improve WLS lowering Recently we improved the lowering of low overhead loops and tail predicated loops, but concentrated first on the DLS do style loops. This extends those improvements over to the WLS while loops, improving the chance of lowering them successfully. To do this the lowering has to change a little as the instructions are terminators that produce a value - something that needs to be treated carefully. Lowering starts at the Hardware Loop pass, inserting a new llvm.test.start.loop.iterations that produces both an i1 to control the loop entry and an i32 similar to the llvm.start.loop.iterations intrinsic added for do loops. This feeds into the loop phi, properly gluing the values together: %wls = call { i32, i1 } @llvm.test.start.loop.iterations.i32(i32 %div) %wls0 = extractvalue { i32, i1 } %wls, 0 %wls1 = extractvalue { i32, i1 } %wls, 1 br i1 %wls1, label %loop.ph, label %loop.exit ... loop: %lsr.iv = phi i32 [ %wls0, %loop.ph ], [ %iv.next, %loop ] .. %iv.next = call i32 @llvm.loop.decrement.reg.i32(i32 %lsr.iv, i32 1) %cmp = icmp ne i32 %iv.next, 0 br i1 %cmp, label %loop, label %loop.exit The llvm.test.start.loop.iterations need to be lowered through ISel lowering as a pair of WLS and WLSSETUP nodes, which each get converted to t2WhileLoopSetup and t2WhileLoopStart Pseudos. This helps prevent t2WhileLoopStart from being a terminator that produces a value, something difficult to control at that stage in the pipeline. Instead the t2WhileLoopSetup produces the value of LR (essentially acting as a lr = subs rn, 0), t2WhileLoopStart consumes that lr value (the Bcc). These are then converted into a single t2WhileLoopStartLR at the same point as t2DoLoopStartTP and t2LoopEndDec. Otherwise we revert the loop to prevent them from progressing further in the pipeline. The t2WhileLoopStartLR is a single instruction that takes a GPR and produces LR, similar to the WLS instruction. %1:gprlr = t2WhileLoopStartLR %0:rgpr, %bb.3 t2B %bb.1 ... bb.2.loop: %2:gprlr = PHI %1:gprlr, %bb.1, %3:gprlr, %bb.2 ... %3:gprlr = t2LoopEndDec %2:gprlr, %bb.2 t2B %bb.3 The t2WhileLoopStartLR can then be treated similar to the other low overhead loop pseudos, eventually being lowered to a WLS providing the branches are within range. Differential Revision: https://reviews.llvm.org/D97729
2021-03-11 22:06:04 +08:00
; CHECK-EXIT: [[TEST:%[^ ]+]] = call i1 @llvm.test.set.loop.iterations.i32(i32 %N)
; CHECK-LATCH: [[TEST1:%[^ ]+]] = call { i32, i1 } @llvm.test.start.loop.iterations.i32(i32 %N)
; CHECK-LATCH: [[TEST:%[^ ]+]] = extractvalue { i32, i1 } [[TEST1]], 1
[HardwareLoops] Loop counter guard intrinsic Introduce llvm.test.set.loop.iterations which sets the loop counter and also produces an i1 after testing that the count is not zero. Differential Revision: https://reviews.llvm.org/D63809 llvm-svn: 364628
2019-06-28 15:38:16 +08:00
; CHECK: br i1 [[TEST]], label %do.body.preheader1, label %if.end
; CHECK: do.body.preheader1:
; CHECK: br label %do.body
define void @test11(i1 zeroext %t1, i32* nocapture %a, i32* nocapture readonly %b, i32 %N) {
entry:
br label %do.body.preheader
do.body.preheader:
%cmp = icmp ne i32 %N, 0
br i1 %cmp, label %do.body, label %if.end
do.body:
%b.addr.0 = phi i32* [ %incdec.ptr, %do.body ], [ %b, %do.body.preheader ]
%a.addr.0 = phi i32* [ %incdec.ptr3, %do.body ], [ %a, %do.body.preheader ]
%i.0 = phi i32 [ %inc, %do.body ], [ 0, %do.body.preheader ]
%incdec.ptr = getelementptr inbounds i32, i32* %b.addr.0, i32 1
%tmp = load i32, i32* %b.addr.0, align 4
%incdec.ptr3 = getelementptr inbounds i32, i32* %a.addr.0, i32 1
store i32 %tmp, i32* %a.addr.0, align 4
%inc = add nuw i32 %i.0, 1
%cmp.1 = icmp ult i32 %inc, %N
br i1 %cmp.1, label %do.body, label %if.end
if.end: ; preds = %do.body, %entry
ret void
}