maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/mlir/test/Transforms/loop-fusion.mlir

2292 lines
79 KiB
MLIR
Raw Normal View History

Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// RUN: mlir-opt %s -loop-fusion -split-input-file -verify | FileCheck %s
Add missing run command to fusion test cases - follow up to cl/236882988 PiperOrigin-RevId: 236947383
2019-03-06 08:54:34 +08:00
// RUN: mlir-opt %s -loop-fusion -fusion-maximal -split-input-file -verify | FileCheck %s --check-prefix=MAXIMAL
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// TODO(andydavis) Add more tests:
// *) Add nested fusion test cases when non-constant loop bound support is
// added to iteration domain in dependence check.
// *) Add a test w/ floordiv/ceildiv/mod when supported in dependence check.
// *) Add tests which check fused computation slice indexing and loop bounds.
Eliminate extfunc/cfgfunc/mlfunc as a concept, and just use 'func' instead. The entire compiler now looks at structural properties of the function (e.g. does it have one block, does it contain an if/for stmt, etc) so the only thing holding up this difference is round tripping through the parser/printer syntax. Removing this shrinks the compile by ~140LOC. This is step 31/n towards merging instructions and statements. The last step is updating the docs, which I will do as a separate patch in order to split it from this mostly mechanical patch. PiperOrigin-RevId: 227540453
2019-01-03 02:20:00 +08:00
// TODO(andydavis) Test clean up: move memref allocs to func args.
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// -----
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
Eliminate extfunc/cfgfunc/mlfunc as a concept, and just use 'func' instead. The entire compiler now looks at structural properties of the function (e.g. does it have one block, does it contain an if/for stmt, etc) so the only thing holding up this difference is round tripping through the parser/printer syntax. Removing this shrinks the compile by ~140LOC. This is step 31/n towards merging instructions and statements. The last step is updating the docs, which I will do as a separate patch in order to split it from this mostly mechanical patch. PiperOrigin-RevId: 227540453
2019-01-03 02:20:00 +08:00
// CHECK-LABEL: func @should_fuse_raw_dep_for_locality() {
func @should_fuse_raw_dep_for_locality() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%m = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
store %cf7, %m[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v0 = load %m[%i1] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %1 = affine.apply [[MAP0]](%i0, %i0)
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-NEXT: store %cst, %0[%1] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %2 = affine.apply [[MAP0]](%i0, %i0)
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-NEXT: %3 = load %0[%2] : memref<1xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-DAG: [[MAP0:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
Eliminate extfunc/cfgfunc/mlfunc as a concept, and just use 'func' instead. The entire compiler now looks at structural properties of the function (e.g. does it have one block, does it contain an if/for stmt, etc) so the only thing holding up this difference is round tripping through the parser/printer syntax. Removing this shrinks the compile by ~140LOC. This is step 31/n towards merging instructions and statements. The last step is updating the docs, which I will do as a separate patch in order to split it from this mostly mechanical patch. PiperOrigin-RevId: 227540453
2019-01-03 02:20:00 +08:00
// CHECK-LABEL: func @should_fuse_reduction_to_pointwise() {
func @should_fuse_reduction_to_pointwise() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%a = alloc() : memref<10x10xf32>
%b = alloc() : memref<10xf32>
%c = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
for %i1 = 0 to 10 {
Improve loop fusion algorithm by using a memref dependence graph. Fixed TODO for reduction fusion unit test. PiperOrigin-RevId: 226277226
2018-12-20 12:42:55 +08:00
%v0 = load %b[%i0] : memref<10xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v1 = load %a[%i0, %i1] : memref<10x10xf32>
%v3 = addf %v0, %v1 : f32
store %v3, %b[%i0] : memref<10xf32>
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v4 = load %b[%i2] : memref<10xf32>
store %v4, %c[%i2] : memref<10xf32>
}
// Should fuse in entire inner loop on %i1 from source loop nest, as %i1
// is not used in the access function of the store/load on %b.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
// CHECK-NEXT: for %i1 = 0 to 10 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %3 = affine.apply [[MAP0]](%i0, %i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %4 = load %0[%3] : memref<1xf32>
// CHECK-NEXT: %5 = load %1[%i0, %i1] : memref<10x10xf32>
Improve loop fusion algorithm by using a memref dependence graph. Fixed TODO for reduction fusion unit test. PiperOrigin-RevId: 226277226
2018-12-20 12:42:55 +08:00
// CHECK-NEXT: %6 = addf %4, %5 : f32
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %7 = affine.apply [[MAP0]](%i0, %i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %6, %0[%7] : memref<1xf32>
Improve loop fusion algorithm by using a memref dependence graph. Fixed TODO for reduction fusion unit test. PiperOrigin-RevId: 226277226
2018-12-20 12:42:55 +08:00
// CHECK-NEXT: }
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %8 = affine.apply [[MAP0]](%i0, %i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %9 = load %0[%8] : memref<1xf32>
// CHECK-NEXT: store %9, %2[%i0] : memref<10xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-DAG: [[MAP_SHIFT_MINUS_ONE_R1:#map[0-9]+]] = (d0) -> (d0 - 1)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-DAG: [[MAP_SHIFT_BY_ONE:#map[0-9]+]] = (d0) -> (d0 + 1)
// CHECK-DAG: [[MAP_SHIFT_MINUS_IV_R2_EVEN:#map[0-9]+]] = (d0, d1, d2, d3) -> (-d0 + d2)
// CHECK-DAG: [[MAP_SHIFT_MINUS_IV_R2_ODD:#map[0-9]+]] = (d0, d1, d2, d3) -> (-d1 + d3)
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
Eliminate extfunc/cfgfunc/mlfunc as a concept, and just use 'func' instead. The entire compiler now looks at structural properties of the function (e.g. does it have one block, does it contain an if/for stmt, etc) so the only thing holding up this difference is round tripping through the parser/printer syntax. Removing this shrinks the compile by ~140LOC. This is step 31/n towards merging instructions and statements. The last step is updating the docs, which I will do as a separate patch in order to split it from this mostly mechanical patch. PiperOrigin-RevId: 227540453
2019-01-03 02:20:00 +08:00
// CHECK-LABEL: func @should_fuse_loop_nests_with_shifts() {
func @should_fuse_loop_nests_with_shifts() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%a = alloc() : memref<10x10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 9 {
for %i1 = 0 to 9 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
%idx = affine.apply (d0) -> (d0 + 1) (%i0)
%idy = affine.apply (d0) -> (d0 + 1) (%i1)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
store %cf7, %a[%idx, %idy] : memref<10x10xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 1 to 10 {
for %i3 = 1 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v0 = load %a[%i2, %i3] : memref<10x10xf32>
}
}
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// Source slice affine apply sequence:
// *) First two affine apply's map from the dst to src iteration space.
// *) Third affine apply is access function around src store.
// *) Fourth affine apply shifts the stores access function by '-1', because
// of the offset induced by reducing the memref shape from 10x10 to 9x9.
// *) Fifth affine apply shifts the loads access function by '-1', because
// of the offset induced by reducing the memref shape from 10x10 to 9x9.
// NOTE: Should create a private memref with reduced shape 9x9xf32.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 1 to 10 {
// CHECK-NEXT: for %i1 = 1 to 10 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %1 = affine.apply [[MAP_SHIFT_MINUS_ONE_R1]](%i0)
// CHECK-NEXT: %2 = affine.apply [[MAP_SHIFT_MINUS_ONE_R1]](%i1)
// CHECK-NEXT: %3 = affine.apply [[MAP_SHIFT_BY_ONE]](%1)
// CHECK-NEXT: %4 = affine.apply [[MAP_SHIFT_BY_ONE]](%2)
// CHECK-NEXT: %5 = affine.apply [[MAP_SHIFT_MINUS_IV_R2_EVEN]](%i0, %i1, %3, %4)
// CHECK-NEXT: %6 = affine.apply [[MAP_SHIFT_MINUS_IV_R2_ODD]](%i0, %i1, %3, %4)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-NEXT: store %cst, %0[%5, %6] : memref<1x1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %7 = affine.apply [[MAP_SHIFT_MINUS_IV_R2_EVEN]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: %8 = affine.apply [[MAP_SHIFT_MINUS_IV_R2_ODD]](%i0, %i1, %i0, %i1)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-NEXT: %9 = load %0[%7, %8] : memref<1x1xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
Adds a dependence check to test whether two accesses to the same memref access the same element. - Builds access functions and iterations domains for each access. - Builds dependence polyhedron constraint system which has equality constraints for equated access functions and inequality constraints for iteration domain loop bounds. - Runs elimination on the dependence polyhedron to test if no dependence exists between the accesses. - Adds a trivial LoopFusion transformation pass with a simple test policy to test dependence between accesses to the same memref in adjacent loops. - The LoopFusion pass will be extended in subsequent CLs. PiperOrigin-RevId: 219630898
2018-11-01 22:26:00 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: return
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
return
}
// -----
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-DAG: [[MAP_D2_D0_DIFF:#map[0-9]+]] = (d0, d1, d2, d3) -> (-d0 + d2)
// CHECK-DAG: [[MAP_D3_D1_DIFF:#map[0-9]+]] = (d0, d1, d2, d3) -> (-d1 + d3)
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
Eliminate extfunc/cfgfunc/mlfunc as a concept, and just use 'func' instead. The entire compiler now looks at structural properties of the function (e.g. does it have one block, does it contain an if/for stmt, etc) so the only thing holding up this difference is round tripping through the parser/printer syntax. Removing this shrinks the compile by ~140LOC. This is step 31/n towards merging instructions and statements. The last step is updating the docs, which I will do as a separate patch in order to split it from this mostly mechanical patch. PiperOrigin-RevId: 227540453
2019-01-03 02:20:00 +08:00
// CHECK-LABEL: func @should_fuse_loop_nest() {
func @should_fuse_loop_nest() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%a = alloc() : memref<10x10xf32>
%b = alloc() : memref<10x10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
for %i1 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
store %cf7, %a[%i0, %i1] : memref<10x10xf32>
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 10 {
for %i3 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v0 = load %a[%i3, %i2] : memref<10x10xf32>
store %v0, %b[%i2, %i3] : memref<10x10xf32>
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i4 = 0 to 10 {
for %i5 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v1 = load %b[%i4, %i5] : memref<10x10xf32>
}
}
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// Expecting private memref for '%a' first, then private memref for '%b'.
// CHECK-DAG: [[NEWA:%[0-9]+]] = alloc() : memref<1x1xf32>
// CHECK-DAG: [[NEWB:%[0-9]+]] = alloc() : memref<1x1xf32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
// CHECK-NEXT: for %i1 = 0 to 10 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %2 = affine.apply [[MAP_D2_D0_DIFF]](%i1, %i0, %i1, %i0)
// CHECK-NEXT: %3 = affine.apply [[MAP_D3_D1_DIFF]](%i1, %i0, %i1, %i0)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-NEXT: store %cst, [[NEWA]][%2, %3] : memref<1x1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %4 = affine.apply [[MAP_D2_D0_DIFF]](%i1, %i0, %i1, %i0)
// CHECK-NEXT: %5 = affine.apply [[MAP_D3_D1_DIFF]](%i1, %i0, %i1, %i0)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-NEXT: %6 = load [[NEWA]][%4, %5] : memref<1x1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %7 = affine.apply [[MAP_D2_D0_DIFF]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: %8 = affine.apply [[MAP_D3_D1_DIFF]](%i0, %i1, %i0, %i1)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-NEXT: store %6, [[NEWB]][%7, %8] : memref<1x1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %9 = affine.apply [[MAP_D2_D0_DIFF]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: %10 = affine.apply [[MAP_D3_D1_DIFF]](%i0, %i1, %i0, %i1)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-NEXT: %11 = load [[NEWB]][%9, %10] : memref<1x1xf32>
Adds a dependence check to test whether two accesses to the same memref access the same element. - Builds access functions and iterations domains for each access. - Builds dependence polyhedron constraint system which has equality constraints for equated access functions and inequality constraints for iteration domain loop bounds. - Runs elimination on the dependence polyhedron to test if no dependence exists between the accesses. - Adds a trivial LoopFusion transformation pass with a simple test policy to test dependence between accesses to the same memref in adjacent loops. - The LoopFusion pass will be extended in subsequent CLs. PiperOrigin-RevId: 219630898
2018-11-01 22:26:00 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
return
}
// -----
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-DAG: [[MAP0:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
Eliminate extfunc/cfgfunc/mlfunc as a concept, and just use 'func' instead. The entire compiler now looks at structural properties of the function (e.g. does it have one block, does it contain an if/for stmt, etc) so the only thing holding up this difference is round tripping through the parser/printer syntax. Removing this shrinks the compile by ~140LOC. This is step 31/n towards merging instructions and statements. The last step is updating the docs, which I will do as a separate patch in order to split it from this mostly mechanical patch. PiperOrigin-RevId: 227540453
2019-01-03 02:20:00 +08:00
// CHECK-LABEL: func @should_fuse_across_intermediate_loop_with_no_deps() {
func @should_fuse_across_intermediate_loop_with_no_deps() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%a = alloc() : memref<10xf32>
%b = alloc() : memref<10xf32>
%c = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v0 = load %a[%i0] : memref<10xf32>
store %v0, %b[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
store %cf7, %c[%i1] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v1 = load %b[%i2] : memref<10xf32>
}
// Should fuse first loop (past second loop with no dependences) into third.
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// Note that fusion creates a private memref '%2' for the fused loop nest.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %cst, %2[%i0] : memref<10xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i1 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %3 = load %1[%i1] : memref<10xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %4 = affine.apply [[MAP0]](%i1, %i1)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %3, %0[%4] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %5 = affine.apply [[MAP0]](%i1, %i1)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %6 = load %0[%5] : memref<1xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
Eliminate extfunc/cfgfunc/mlfunc as a concept, and just use 'func' instead. The entire compiler now looks at structural properties of the function (e.g. does it have one block, does it contain an if/for stmt, etc) so the only thing holding up this difference is round tripping through the parser/printer syntax. Removing this shrinks the compile by ~140LOC. This is step 31/n towards merging instructions and statements. The last step is updating the docs, which I will do as a separate patch in order to split it from this mostly mechanical patch. PiperOrigin-RevId: 227540453
2019-01-03 02:20:00 +08:00
// CHECK-LABEL: func @should_fuse_all_loops() {
func @should_fuse_all_loops() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%a = alloc() : memref<10xf32>
%b = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
// Set up flow dependences from first and second loops to third.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
store %cf7, %a[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
store %cf7, %b[%i1] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v0 = load %a[%i2] : memref<10xf32>
%v1 = load %b[%i2] : memref<10xf32>
}
// Should fuse first and second loops into third.
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// Expecting private memref for '%a' first, then private memref for '%b'.
// CHECK-DAG: [[NEWA:%[0-9]+]] = alloc() : memref<1xf32>
// CHECK-DAG: [[NEWB:%[0-9]+]] = alloc() : memref<1xf32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %2 = affine.apply [[MAP0]](%i0, %i0)
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-NEXT: store %cst, [[NEWA]][%2] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %3 = affine.apply [[MAP0]](%i0, %i0)
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-NEXT: store %cst, [[NEWB]][%3] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %4 = affine.apply [[MAP0]](%i0, %i0)
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-NEXT: %5 = load [[NEWA]][%4] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %6 = affine.apply [[MAP0]](%i0, %i0)
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-NEXT: %7 = load [[NEWB]][%6] : memref<1xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
Eliminate extfunc/cfgfunc/mlfunc as a concept, and just use 'func' instead. The entire compiler now looks at structural properties of the function (e.g. does it have one block, does it contain an if/for stmt, etc) so the only thing holding up this difference is round tripping through the parser/printer syntax. Removing this shrinks the compile by ~140LOC. This is step 31/n towards merging instructions and statements. The last step is updating the docs, which I will do as a separate patch in order to split it from this mostly mechanical patch. PiperOrigin-RevId: 227540453
2019-01-03 02:20:00 +08:00
// CHECK-LABEL: func @should_fuse_first_and_second_loops() {
func @should_fuse_first_and_second_loops() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%a = alloc() : memref<10xf32>
%b = alloc() : memref<10xf32>
%c = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
store %cf7, %a[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v0 = load %a[%i1] : memref<10xf32>
store %cf7, %b[%i1] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v1 = load %c[%i2] : memref<10xf32>
}
// Should fuse first loop into the second (last loop should not be fused).
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// Should create private memref '%2' for fused loop.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %3 = affine.apply [[MAP0]](%i0, %i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %cst, %0[%3] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %4 = affine.apply [[MAP0]](%i0, %i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %5 = load %0[%4] : memref<1xf32>
// CHECK-NEXT: store %cst, %1[%i0] : memref<10xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i1 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %6 = load %2[%i1] : memref<10xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
Eliminate extfunc/cfgfunc/mlfunc as a concept, and just use 'func' instead. The entire compiler now looks at structural properties of the function (e.g. does it have one block, does it contain an if/for stmt, etc) so the only thing holding up this difference is round tripping through the parser/printer syntax. Removing this shrinks the compile by ~140LOC. This is step 31/n towards merging instructions and statements. The last step is updating the docs, which I will do as a separate patch in order to split it from this mostly mechanical patch. PiperOrigin-RevId: 227540453
2019-01-03 02:20:00 +08:00
// CHECK-LABEL: func @should_not_fuse_would_create_cycle() {
func @should_not_fuse_would_create_cycle() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%a = alloc() : memref<10xf32>
%b = alloc() : memref<10xf32>
%c = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
// Set up the following dependences:
// 1) loop0 -> loop1 on memref '%a'
// 2) loop0 -> loop2 on memref '%b'
// 3) loop1 -> loop2 on memref '%c'
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v0 = load %a[%i0] : memref<10xf32>
store %cf7, %b[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
store %cf7, %a[%i1] : memref<10xf32>
%v1 = load %c[%i1] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v2 = load %b[%i2] : memref<10xf32>
store %cf7, %c[%i2] : memref<10xf32>
}
// Should not fuse: fusing loop first loop into last would create a cycle.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: %3 = load %0[%i0] : memref<10xf32>
// CHECK-NEXT: store %cst, %1[%i0] : memref<10xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i1 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: store %cst, %0[%i1] : memref<10xf32>
// CHECK-NEXT: %4 = load %2[%i1] : memref<10xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i2 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: %5 = load %1[%i2] : memref<10xf32>
// CHECK-NEXT: store %cst, %2[%i2] : memref<10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
// CHECK-LABEL: func @should_not_fuse_across_waw_dep() {
func @should_not_fuse_across_waw_dep() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%m = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
store %cf7, %m[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
store %cf7, %m[%i1] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v1 = load %m[%i2] : memref<10xf32>
}
// Fusing loop %i0 to %i2 would violate the WAW dependence between %i0 and %i1
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
// CHECK-NEXT: store %cst, %0[%i0] : memref<10xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i1 = 0 to 10 {
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
// CHECK-NEXT: store %cst, %0[%i1] : memref<10xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i2 = 0 to 10 {
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
// CHECK-NEXT: %1 = load %0[%i2] : memref<10xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-LABEL: func @should_fuse_and_move_to_preserve_war_dep() {
func @should_fuse_and_move_to_preserve_war_dep() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%a = alloc() : memref<10xf32>
%b = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v0 = load %a[%i0] : memref<10xf32>
store %v0, %b[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
store %cf7, %a[%i1] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v1 = load %b[%i2] : memref<10xf32>
}
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// Loops '%i1' and '%i2' have no dependences. We can fuse a slice of '%i0'
// into '%i2' if we move the fused loop nest before '%i1', which preserves
// the WAR dependence from load '%a' in '%i0' to the store '%a' in loop '%i1'.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %2 = load %1[%i0] : memref<10xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %3 = affine.apply [[MAP0]](%i0, %i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %2, %0[%3] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %4 = affine.apply [[MAP0]](%i0, %i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %5 = load %0[%4] : memref<1xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i1 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %cst, %1[%i1] : memref<10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: return
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
return
}
// -----
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// CHECK-LABEL: func @should_fuse_with_private_memref_if_top_level_access() {
func @should_fuse_with_private_memref_if_top_level_access() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%m = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
store %cf7, %m[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v0 = load %m[%i1] : memref<10xf32>
}
%c0 = constant 4 : index
%v1 = load %m[%c0] : memref<10xf32>
// Top-level load to '%m' should prevent fusion.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %cst, %1[%i0] : memref<10xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i1 = 0 to 10 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %2 = affine.apply [[MAP0]](%i1, %i1)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %cst, %0[%2] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %3 = affine.apply [[MAP0]](%i1, %i1)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %4 = load %0[%3] : memref<1xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
return
}
// -----
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
Eliminate extfunc/cfgfunc/mlfunc as a concept, and just use 'func' instead. The entire compiler now looks at structural properties of the function (e.g. does it have one block, does it contain an if/for stmt, etc) so the only thing holding up this difference is round tripping through the parser/printer syntax. Removing this shrinks the compile by ~140LOC. This is step 31/n towards merging instructions and statements. The last step is updating the docs, which I will do as a separate patch in order to split it from this mostly mechanical patch. PiperOrigin-RevId: 227540453
2019-01-03 02:20:00 +08:00
// CHECK-LABEL: func @should_fuse_no_top_level_access() {
func @should_fuse_no_top_level_access() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%m = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
store %cf7, %m[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v0 = load %m[%i1] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %1 = affine.apply [[MAP0]](%i0, %i0)
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-NEXT: store %cst, %0[%1] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %2 = affine.apply [[MAP0]](%i0, %i0)
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-NEXT: %3 = load %0[%2] : memref<1xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
#set0 = (d0) : (1 == 0)
Eliminate extfunc/cfgfunc/mlfunc as a concept, and just use 'func' instead. The entire compiler now looks at structural properties of the function (e.g. does it have one block, does it contain an if/for stmt, etc) so the only thing holding up this difference is round tripping through the parser/printer syntax. Removing this shrinks the compile by ~140LOC. This is step 31/n towards merging instructions and statements. The last step is updating the docs, which I will do as a separate patch in order to split it from this mostly mechanical patch. PiperOrigin-RevId: 227540453
2019-01-03 02:20:00 +08:00
// CHECK-LABEL: func @should_not_fuse_if_inst_at_top_level() {
func @should_not_fuse_if_inst_at_top_level() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%m = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
store %cf7, %m[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v0 = load %m[%i1] : memref<10xf32>
}
%c0 = constant 4 : index
Automated rollback of changelist 232728977. PiperOrigin-RevId: 232944889
2019-02-08 06:24:18 +08:00
if #set0(%c0) {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
}
Recommit: Define a AffineOps dialect as well as an AffineIfOp operation. Replace all instances of IfInst with AffineIfOp and delete IfInst. PiperOrigin-RevId: 231342063
2019-01-29 13:23:53 +08:00
// Top-level IfOp should prevent fusion.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: store %cst, %0[%i0] : memref<10xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i1 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: %1 = load %0[%i1] : memref<10xf32>
// CHECK-NEXT: }
return
}
// -----
#set0 = (d0) : (1 == 0)
Eliminate extfunc/cfgfunc/mlfunc as a concept, and just use 'func' instead. The entire compiler now looks at structural properties of the function (e.g. does it have one block, does it contain an if/for stmt, etc) so the only thing holding up this difference is round tripping through the parser/printer syntax. Removing this shrinks the compile by ~140LOC. This is step 31/n towards merging instructions and statements. The last step is updating the docs, which I will do as a separate patch in order to split it from this mostly mechanical patch. PiperOrigin-RevId: 227540453
2019-01-03 02:20:00 +08:00
// CHECK-LABEL: func @should_not_fuse_if_inst_in_loop_nest() {
func @should_not_fuse_if_inst_in_loop_nest() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%m = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
%c4 = constant 4 : index
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
store %cf7, %m[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Automated rollback of changelist 232728977. PiperOrigin-RevId: 232944889
2019-02-08 06:24:18 +08:00
if #set0(%c4) {
Adds a dependence check to test whether two accesses to the same memref access the same element. - Builds access functions and iterations domains for each access. - Builds dependence polyhedron constraint system which has equality constraints for equated access functions and inequality constraints for iteration domain loop bounds. - Runs elimination on the dependence polyhedron to test if no dependence exists between the accesses. - Adds a trivial LoopFusion transformation pass with a simple test policy to test dependence between accesses to the same memref in adjacent loops. - The LoopFusion pass will be extended in subsequent CLs. PiperOrigin-RevId: 219630898
2018-11-01 22:26:00 +08:00
}
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%v0 = load %m[%i1] : memref<10xf32>
Adds a dependence check to test whether two accesses to the same memref access the same element. - Builds access functions and iterations domains for each access. - Builds dependence polyhedron constraint system which has equality constraints for equated access functions and inequality constraints for iteration domain loop bounds. - Runs elimination on the dependence polyhedron to test if no dependence exists between the accesses. - Adds a trivial LoopFusion transformation pass with a simple test policy to test dependence between accesses to the same memref in adjacent loops. - The LoopFusion pass will be extended in subsequent CLs. PiperOrigin-RevId: 219630898
2018-11-01 22:26:00 +08:00
}
Recommit: Define a AffineOps dialect as well as an AffineIfOp operation. Replace all instances of IfInst with AffineIfOp and delete IfInst. PiperOrigin-RevId: 231342063
2019-01-29 13:23:53 +08:00
// IfOp in ForInst should prevent fusion.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: store %cst, %0[%i0] : memref<10xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i1 = 0 to 10 {
Automated rollback of changelist 232728977. PiperOrigin-RevId: 232944889
2019-02-08 06:24:18 +08:00
// CHECK-NEXT: if #set0(%c4) {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: %1 = load %0[%i1] : memref<10xf32>
// CHECK-NEXT: }
return
}
// -----
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5) -> (-d0 + d3)
// CHECK: [[MAP1:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5) -> (-d1 + d4)
// CHECK: [[MAP2:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5) -> (-d2 + d5)
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-LABEL: func @permute_and_fuse() {
func @permute_and_fuse() {
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
%m = alloc() : memref<10x20x30xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
for %i1 = 0 to 20 {
for %i2 = 0 to 30 {
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
store %cf7, %m[%i0, %i1, %i2] : memref<10x20x30xf32>
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
}
Adds a dependence check to test whether two accesses to the same memref access the same element. - Builds access functions and iterations domains for each access. - Builds dependence polyhedron constraint system which has equality constraints for equated access functions and inequality constraints for iteration domain loop bounds. - Runs elimination on the dependence polyhedron to test if no dependence exists between the accesses. - Adds a trivial LoopFusion transformation pass with a simple test policy to test dependence between accesses to the same memref in adjacent loops. - The LoopFusion pass will be extended in subsequent CLs. PiperOrigin-RevId: 219630898
2018-11-01 22:26:00 +08:00
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i3 = 0 to 30 {
for %i4 = 0 to 10 {
for %i5 = 0 to 20 {
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
%v0 = load %m[%i4, %i5, %i3] : memref<10x20x30xf32>
"foo"(%v0) : (f32) -> ()
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
}
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 30 {
// CHECK-NEXT: for %i1 = 0 to 10 {
// CHECK-NEXT: for %i2 = 0 to 20 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %1 = affine.apply [[MAP0]](%i1, %i2, %i0, %i1, %i2, %i0)
// CHECK-NEXT: %2 = affine.apply [[MAP1]](%i1, %i2, %i0, %i1, %i2, %i0)
// CHECK-NEXT: %3 = affine.apply [[MAP2]](%i1, %i2, %i0, %i1, %i2, %i0)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-NEXT: store %cst, %0[%1, %2, %3] : memref<1x1x1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %4 = affine.apply [[MAP0]](%i1, %i2, %i0, %i1, %i2, %i0)
// CHECK-NEXT: %5 = affine.apply [[MAP1]](%i1, %i2, %i0, %i1, %i2, %i0)
// CHECK-NEXT: %6 = affine.apply [[MAP2]](%i1, %i2, %i0, %i1, %i2, %i0)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-NEXT: %7 = load %0[%4, %5, %6] : memref<1x1x1xf32>
// CHECK-NEXT: "foo"(%7) : (f32) -> ()
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
Adds a dependence check to test whether two accesses to the same memref access the same element. - Builds access functions and iterations domains for each access. - Builds dependence polyhedron constraint system which has equality constraints for equated access functions and inequality constraints for iteration domain loop bounds. - Runs elimination on the dependence polyhedron to test if no dependence exists between the accesses. - Adds a trivial LoopFusion transformation pass with a simple test policy to test dependence between accesses to the same memref in adjacent loops. - The LoopFusion pass will be extended in subsequent CLs. PiperOrigin-RevId: 219630898
2018-11-01 22:26:00 +08:00
return
Loop Fusion pass update: introduce utilities to perform generalized loop fusion based on slicing; encompasses standard loop fusion. *) Adds simple greedy fusion algorithm to drive experimentation. This algorithm greedily fuses loop nests with single-writer/single-reader memref dependences to improve locality. *) Adds support for fusing slices of a loop nest computation: fusing one loop nest into another by adjusting the source loop nest's iteration bounds (after it is fused into the destination loop nest). This is accomplished by solving for the source loop nest's IVs in terms of the destination loop nests IVs and symbols using the dependece polyhedron, then creating AffineMaps of these functions for the loop bounds of the fused source loop. *) Adds utility function 'insertMemRefComputationSlice' which computes and inserts computation slice from loop nest surrounding a source memref access into the loop nest surrounding the destingation memref access. *) Adds FlatAffineConstraints::toAffineMap function which returns and AffineMap which represents an equality contraint where one dimension identifier is represented as a function of all others in the equality constraint. *) Adds multiple fusion unit tests. PiperOrigin-RevId: 225842944
2018-12-18 01:57:14 +08:00
}
Computation slice update: adds parameters to insertBackwardComputationSlice which specify the source loop nest depth at which to perform iteration space slicing, and the destination loop nest depth at which to insert the compution slice. Updates LoopFusion pass to take these parameters as command line flags for experimentation. PiperOrigin-RevId: 226514297
2018-12-22 03:06:23 +08:00
// -----
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK-DAG: [[MAP0:#map[0-9]+]] = (d0, d1) -> (d0 * 4 + d1)
// CHECK-DAG: [[MAP1:#map[0-9]+]] = (d0) -> (d0 floordiv 4)
// CHECK-DAG: [[MAP2:#map[0-9]+]] = (d0) -> (d0 mod 4)
Complete TODOs / cleanup for loop-fusion utility - this is CL 1/2 that does a clean up and gets rid of one limitation in an underlying method - as a result, fusion works for more cases. - fix bugs/incomplete impl. in toAffineMapFromEq - fusing across rank changing reshapes for example now just works For eg. given a rank 1 memref to rank 2 memref reshape (64 -> 8 x 8) like this, -loop-fusion -memref-dataflow-opt now completely fuses and inlines/store-forward to get rid of the temporary: INPUT // Rank 1 -> Rank 2 reshape for %i0 = 0 to 64 { %v = load %A[%i0] store %v, %B[%i0 floordiv 8, i0 mod 8] } for %i1 = 0 to 8 for %i2 = 0 to 8 %w = load %B[%i1, i2] "foo"(%w) : (f32) -> () OUTPUT $ mlir-opt -loop-fusion -memref-dataflow-opt fuse_reshape.mlir #map0 = (d0, d1) -> (d0 * 8 + d1) mlfunc @fuse_reshape(%arg0: memref<64xf32>) { for %i0 = 0 to 8 { for %i1 = 0 to 8 { %0 = affine_apply #map0(%i0, %i1) %1 = load %arg0[%0] : memref<64xf32> "foo"(%1) : (f32) -> () } } } AFAIK, there is no polyhedral tool / compiler that can perform such fusion - because it's not really standard loop fusion, but possible through a generalized slicing-based approach such as ours. PiperOrigin-RevId: 227918338
2019-01-05 06:52:21 +08:00
// Reshape from a 64 x f32 to 16 x 4 x f32.
// CHECK-LABEL: func @fuse_reshape_64_16_4
func @fuse_reshape_64_16_4(%in : memref<64xf32>) {
%out = alloc() : memref<16x4xf32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 64 {
Complete TODOs / cleanup for loop-fusion utility - this is CL 1/2 that does a clean up and gets rid of one limitation in an underlying method - as a result, fusion works for more cases. - fix bugs/incomplete impl. in toAffineMapFromEq - fusing across rank changing reshapes for example now just works For eg. given a rank 1 memref to rank 2 memref reshape (64 -> 8 x 8) like this, -loop-fusion -memref-dataflow-opt now completely fuses and inlines/store-forward to get rid of the temporary: INPUT // Rank 1 -> Rank 2 reshape for %i0 = 0 to 64 { %v = load %A[%i0] store %v, %B[%i0 floordiv 8, i0 mod 8] } for %i1 = 0 to 8 for %i2 = 0 to 8 %w = load %B[%i1, i2] "foo"(%w) : (f32) -> () OUTPUT $ mlir-opt -loop-fusion -memref-dataflow-opt fuse_reshape.mlir #map0 = (d0, d1) -> (d0 * 8 + d1) mlfunc @fuse_reshape(%arg0: memref<64xf32>) { for %i0 = 0 to 8 { for %i1 = 0 to 8 { %0 = affine_apply #map0(%i0, %i1) %1 = load %arg0[%0] : memref<64xf32> "foo"(%1) : (f32) -> () } } } AFAIK, there is no polyhedral tool / compiler that can perform such fusion - because it's not really standard loop fusion, but possible through a generalized slicing-based approach such as ours. PiperOrigin-RevId: 227918338
2019-01-05 06:52:21 +08:00
%v = load %in[%i0] : memref<64xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
%idx = affine.apply (d0) -> (d0 floordiv 4) (%i0)
%idy = affine.apply (d0) -> (d0 mod 4) (%i0)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
store %v, %out[%idx, %idy] : memref<16x4xf32>
Complete TODOs / cleanup for loop-fusion utility - this is CL 1/2 that does a clean up and gets rid of one limitation in an underlying method - as a result, fusion works for more cases. - fix bugs/incomplete impl. in toAffineMapFromEq - fusing across rank changing reshapes for example now just works For eg. given a rank 1 memref to rank 2 memref reshape (64 -> 8 x 8) like this, -loop-fusion -memref-dataflow-opt now completely fuses and inlines/store-forward to get rid of the temporary: INPUT // Rank 1 -> Rank 2 reshape for %i0 = 0 to 64 { %v = load %A[%i0] store %v, %B[%i0 floordiv 8, i0 mod 8] } for %i1 = 0 to 8 for %i2 = 0 to 8 %w = load %B[%i1, i2] "foo"(%w) : (f32) -> () OUTPUT $ mlir-opt -loop-fusion -memref-dataflow-opt fuse_reshape.mlir #map0 = (d0, d1) -> (d0 * 8 + d1) mlfunc @fuse_reshape(%arg0: memref<64xf32>) { for %i0 = 0 to 8 { for %i1 = 0 to 8 { %0 = affine_apply #map0(%i0, %i1) %1 = load %arg0[%0] : memref<64xf32> "foo"(%1) : (f32) -> () } } } AFAIK, there is no polyhedral tool / compiler that can perform such fusion - because it's not really standard loop fusion, but possible through a generalized slicing-based approach such as ours. PiperOrigin-RevId: 227918338
2019-01-05 06:52:21 +08:00
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 16 {
for %i2 = 0 to 4 {
Complete TODOs / cleanup for loop-fusion utility - this is CL 1/2 that does a clean up and gets rid of one limitation in an underlying method - as a result, fusion works for more cases. - fix bugs/incomplete impl. in toAffineMapFromEq - fusing across rank changing reshapes for example now just works For eg. given a rank 1 memref to rank 2 memref reshape (64 -> 8 x 8) like this, -loop-fusion -memref-dataflow-opt now completely fuses and inlines/store-forward to get rid of the temporary: INPUT // Rank 1 -> Rank 2 reshape for %i0 = 0 to 64 { %v = load %A[%i0] store %v, %B[%i0 floordiv 8, i0 mod 8] } for %i1 = 0 to 8 for %i2 = 0 to 8 %w = load %B[%i1, i2] "foo"(%w) : (f32) -> () OUTPUT $ mlir-opt -loop-fusion -memref-dataflow-opt fuse_reshape.mlir #map0 = (d0, d1) -> (d0 * 8 + d1) mlfunc @fuse_reshape(%arg0: memref<64xf32>) { for %i0 = 0 to 8 { for %i1 = 0 to 8 { %0 = affine_apply #map0(%i0, %i1) %1 = load %arg0[%0] : memref<64xf32> "foo"(%1) : (f32) -> () } } } AFAIK, there is no polyhedral tool / compiler that can perform such fusion - because it's not really standard loop fusion, but possible through a generalized slicing-based approach such as ours. PiperOrigin-RevId: 227918338
2019-01-05 06:52:21 +08:00
%w = load %out[%i1, %i2] : memref<16x4xf32>
"foo"(%w) : (f32) -> ()
}
}
return
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 =
// CHECK-NEXT: for %i1 =
Complete TODOs / cleanup for loop-fusion utility - this is CL 1/2 that does a clean up and gets rid of one limitation in an underlying method - as a result, fusion works for more cases. - fix bugs/incomplete impl. in toAffineMapFromEq - fusing across rank changing reshapes for example now just works For eg. given a rank 1 memref to rank 2 memref reshape (64 -> 8 x 8) like this, -loop-fusion -memref-dataflow-opt now completely fuses and inlines/store-forward to get rid of the temporary: INPUT // Rank 1 -> Rank 2 reshape for %i0 = 0 to 64 { %v = load %A[%i0] store %v, %B[%i0 floordiv 8, i0 mod 8] } for %i1 = 0 to 8 for %i2 = 0 to 8 %w = load %B[%i1, i2] "foo"(%w) : (f32) -> () OUTPUT $ mlir-opt -loop-fusion -memref-dataflow-opt fuse_reshape.mlir #map0 = (d0, d1) -> (d0 * 8 + d1) mlfunc @fuse_reshape(%arg0: memref<64xf32>) { for %i0 = 0 to 8 { for %i1 = 0 to 8 { %0 = affine_apply #map0(%i0, %i1) %1 = load %arg0[%0] : memref<64xf32> "foo"(%1) : (f32) -> () } } } AFAIK, there is no polyhedral tool / compiler that can perform such fusion - because it's not really standard loop fusion, but possible through a generalized slicing-based approach such as ours. PiperOrigin-RevId: 227918338
2019-01-05 06:52:21 +08:00
// CHECK-NOT: for
// CHECK: }
// CHECK-NEXT: }
// CHECK-NEXT: return
}
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
// -----
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK-DAG: [[MAP0:#map[0-9]+]] = (d0) -> (d0 floordiv 4)
// CHECK-DAG: [[MAP1:#map[0-9]+]] = (d0) -> (d0 mod 4)
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-DAG: [[MAP2:#map[0-9]+]] = (d0, d1) -> (d0 * 4 + d1)
// CHECK-DAG: [[MAP3:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
// Reshape a 16x4xf32 to 64xf32.
// CHECK-LABEL: func @fuse_reshape_16_4_64
func @fuse_reshape_16_4_64() {
%in = alloc() : memref<16x4xf32>
%out = alloc() : memref<64xf32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 16 {
for %i1 = 0 to 4 {
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
%v = load %in[%i0, %i1] : memref<16x4xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
%idx = affine.apply (d0, d1) -> (4*d0 + d1) (%i0, %i1)
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
store %v, %out[%idx] : memref<64xf32>
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 64 {
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
%w = load %out[%i2] : memref<64xf32>
"foo"(%w) : (f32) -> ()
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 64 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %2 = affine.apply [[MAP0]](%i0)
// CHECK-NEXT: %3 = affine.apply [[MAP1]](%i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %4 = load %1[%2, %3] : memref<16x4xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %5 = affine.apply [[MAP2]](%2, %3)
// CHECK-NEXT: %6 = affine.apply [[MAP3]](%i0, %5)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %4, %0[%6] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %7 = affine.apply [[MAP3]](%i0, %i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %8 = load %0[%7] : memref<1xf32>
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-NEXT: "foo"(%8) : (f32) -> ()
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
// All three loop nests below (6-d one, 2-d one, 2-d one is fused into a single
// 2-d loop nest).
func @R6_to_R2_reshape_square() -> memref<64x9xi32> {
%in = alloc() : memref<2x2x3x3x16x1xi32>
%out = alloc() : memref<64x9xi32>
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
%live_out = alloc() : memref<64x9xi32>
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
Update stale / target-specific information in comments - NFC PiperOrigin-RevId: 229800834
2019-01-18 04:51:48 +08:00
// Initialize input.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 2 {
for %i1 = 0 to 2 {
for %i2 = 0 to 3 {
for %i3 = 0 to 3 {
for %i4 = 0 to 16 {
for %i5 = 0 to 1 {
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
%val = "foo"(%i0, %i1, %i2, %i3, %i4, %i5) : (index, index, index, index, index, index) -> i32
store %val, %in[%i0, %i1, %i2, %i3, %i4, %i5] : memref<2x2x3x3x16x1xi32>
}
}
}
}
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %ii = 0 to 64 {
for %jj = 0 to 9 {
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
// Convert output coordinates to linear index.
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
%a0 = affine.apply (d0, d1) -> (d0 * 9 + d1) (%ii, %jj)
%0 = affine.apply (d0) -> (d0 floordiv (2 * 3 * 3 * 16 * 1))(%a0)
%1 = affine.apply (d0) -> ((d0 mod 288) floordiv (3 * 3 * 16 * 1))(%a0)
%2 = affine.apply (d0) -> (((d0 mod 288) mod 144) floordiv (3 * 16 * 1))(%a0)
%3 = affine.apply (d0) -> ((((d0 mod 288) mod 144) mod 48) floordiv (16 * 1))(%a0)
%4 = affine.apply (d0) -> ((((d0 mod 288) mod 144) mod 48) mod 16)(%a0)
%5 = affine.apply (d0) -> (((((d0 mod 144) mod 144) mod 48) mod 16) mod 1)(%a0)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
%v = load %in[%0, %1, %2, %3, %4, %5] : memref<2x2x3x3x16x1xi32>
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
store %v, %out[%ii, %jj] : memref<64x9xi32>
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i = 0 to 64 {
for %j = 0 to 9 {
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
%a = load %out[%i, %j] : memref<64x9xi32>
%b = muli %a, %a : i32
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
store %b, %live_out[%i, %j] : memref<64x9xi32>
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
}
}
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
return %live_out : memref<64x9xi32>
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
}
// Everything above is fused to a single 2-d loop nest, and the 6-d tensor %in
// is eliminated if -memref-dataflow-opt is also supplied.
//
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK-DAG: [[MAP0:#map[0-9]+]] = (d0, d1) -> ((d0 * 9 + d1) floordiv 288)
// CHECK-DAG: [[MAP1:#map[0-9]+]] = (d0, d1) -> (((d0 * 9 + d1) mod 288) floordiv 144)
// CHECK-DAG: [[MAP2:#map[0-9]+]] = (d0, d1) -> ((((d0 * 9 + d1) mod 288) mod 144) floordiv 48)
// CHECK-DAG: [[MAP3:#map[0-9]+]] = (d0, d1) -> (((((d0 * 9 + d1) mod 288) mod 144) mod 48) floordiv 16)
// CHECK-DAG: [[MAP4:#map[0-9]+]] = (d0, d1) -> (((((d0 * 9 + d1) mod 288) mod 144) mod 48) mod 16)
Bug fix for getConstantBoundOnDimSize - this was detected when memref-bound-check was run on the output of the loop-fusion pass - the addition (to represent ceildiv as a floordiv) had to be performed only for the constant term of the constraint - update test cases - memref-bound-check no longer returns an error on the output of this test case PiperOrigin-RevId: 236731137
2019-03-05 06:58:59 +08:00
// CHECK-DAG: [[MAP5:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5, d6, d7) -> (d2 - (d0 * 9 + d1) floordiv 288)
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK-DAG: [[MAP6:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5, d6, d7) -> (d3)
// CHECK-DAG: [[MAP7:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5, d6, d7) -> (d4)
// CHECK-DAG: [[MAP8:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5, d6, d7) -> (d5)
// CHECK-DAG: [[MAP9:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5, d6, d7) -> (d6)
// CHECK-DAG: [[MAP10:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5, d6, d7) -> (d7)
// CHECK-DAG: [[MAP11:#map[0-9]+]] = (d0, d1) -> (d0 * 9 + d1)
// CHECK-DAG: [[MAP12:#map[0-9]+]] = (d0) -> (d0 floordiv 288)
// CHECK-DAG: [[MAP13:#map[0-9]+]] = (d0) -> ((d0 mod 288) floordiv 144)
// CHECK-DAG: [[MAP14:#map[0-9]+]] = (d0) -> (((d0 mod 288) mod 144) floordiv 48)
// CHECK-DAG: [[MAP15:#map[0-9]+]] = (d0) -> ((((d0 mod 288) mod 144) mod 48) floordiv 16)
// CHECK-DAG: [[MAP16:#map[0-9]+]] = (d0) -> ((((d0 mod 288) mod 144) mod 48) mod 16)
// CHECK-DAG: [[MAP17:#map[0-9]+]] = (d0) -> (0)
// CHECK-DAG: [[MAP18:#map[0-9]+]] = (d0, d1, d2, d3) -> (-d0 + d2)
// CHECK-DAG: [[MAP19:#map[0-9]+]] = (d0, d1, d2, d3) -> (-d1 + d3)
//
// CHECK-LABEL: func @R6_to_R2_reshape
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK: %0 = alloc() : memref<1x2x3x3x16x1xi32>
// CHECK: %1 = alloc() : memref<1x1xi32>
// CHECK: %2 = alloc() : memref<64x9xi32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i0 = 0 to 64 {
// CHECK-NEXT: for %i1 = 0 to 9 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %3 = affine.apply [[MAP0]](%i0, %i1)
// CHECK-NEXT: %4 = affine.apply [[MAP1]](%i0, %i1)
// CHECK-NEXT: %5 = affine.apply [[MAP2]](%i0, %i1)
// CHECK-NEXT: %6 = affine.apply [[MAP3]](%i0, %i1)
// CHECK-NEXT: %7 = affine.apply [[MAP4]](%i0, %i1)
Fix test cases that were accessing out of bounds to start with (b/123072438) - detected with memref-bound-check - fixes b/123072438; while on this, fix another test case which was reported out of bounds PiperOrigin-RevId: 229978187
2019-01-19 04:55:46 +08:00
// CHECK-NEXT: %8 = "foo"(%3, %4, %5, %6, %7, %c0) : (index, index, index, index, index, index) -> i32
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %9 = affine.apply [[MAP5]](%i0, %i1, %3, %4, %5, %6, %7, %c0)
// CHECK-NEXT: %10 = affine.apply [[MAP6]](%i0, %i1, %3, %4, %5, %6, %7, %c0)
// CHECK-NEXT: %11 = affine.apply [[MAP7]](%i0, %i1, %3, %4, %5, %6, %7, %c0)
// CHECK-NEXT: %12 = affine.apply [[MAP8]](%i0, %i1, %3, %4, %5, %6, %7, %c0)
// CHECK-NEXT: %13 = affine.apply [[MAP9]](%i0, %i1, %3, %4, %5, %6, %7, %c0)
// CHECK-NEXT: %14 = affine.apply [[MAP10]](%i0, %i1, %3, %4, %5, %6, %7, %c0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %8, %0[%9, %10, %11, %12, %13, %14] : memref<1x2x3x3x16x1xi32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %15 = affine.apply [[MAP11]](%i0, %i1)
// CHECK-NEXT: %16 = affine.apply [[MAP12]](%15)
// CHECK-NEXT: %17 = affine.apply [[MAP13]](%15)
// CHECK-NEXT: %18 = affine.apply [[MAP14]](%15)
// CHECK-NEXT: %19 = affine.apply [[MAP15]](%15)
// CHECK-NEXT: %20 = affine.apply [[MAP16]](%15)
// CHECK-NEXT: %21 = affine.apply [[MAP17]](%15)
// CHECK-NEXT: %22 = affine.apply [[MAP5]](%i0, %i1, %16, %17, %18, %19, %20, %21)
// CHECK-NEXT: %23 = affine.apply [[MAP6]](%i0, %i1, %16, %17, %18, %19, %20, %21)
// CHECK-NEXT: %24 = affine.apply [[MAP7]](%i0, %i1, %16, %17, %18, %19, %20, %21)
// CHECK-NEXT: %25 = affine.apply [[MAP8]](%i0, %i1, %16, %17, %18, %19, %20, %21)
// CHECK-NEXT: %26 = affine.apply [[MAP9]](%i0, %i1, %16, %17, %18, %19, %20, %21)
// CHECK-NEXT: %27 = affine.apply [[MAP10]](%i0, %i1, %16, %17, %18, %19, %20, %21)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %28 = load %0[%22, %23, %24, %25, %26, %27] : memref<1x2x3x3x16x1xi32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %29 = affine.apply [[MAP18]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: %30 = affine.apply [[MAP19]](%i0, %i1, %i0, %i1)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-NEXT: store %28, %1[%29, %30] : memref<1x1xi32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %31 = affine.apply [[MAP18]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: %32 = affine.apply [[MAP19]](%i0, %i1, %i0, %i1)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-NEXT: %33 = load %1[%31, %32] : memref<1x1xi32>
// CHECK-NEXT: %34 = muli %33, %33 : i32
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %34, %2[%i0, %i1] : memref<64x9xi32>
Fix test cases that were accessing out of bounds to start with (b/123072438) - detected with memref-bound-check - fixes b/123072438; while on this, fix another test case which was reported out of bounds PiperOrigin-RevId: 229978187
2019-01-19 04:55:46 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: }
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: return %2 : memref<64x9xi32>
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
// -----
// CHECK-LABEL: func @fuse_symbolic_bounds
func @fuse_symbolic_bounds(%M : index, %N : index) {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
%N_plus_5 = affine.apply (d0) -> (d0 + 5)(%N)
Fix test cases that were accessing out of bounds to start with (b/123072438) - detected with memref-bound-check - fixes b/123072438; while on this, fix another test case which was reported out of bounds PiperOrigin-RevId: 229978187
2019-01-19 04:55:46 +08:00
%m = alloc(%M, %N_plus_5) : memref<? x ? x f32>
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
%c0 = constant 0.0 : f32
%s = constant 5 : index
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to %M {
for %i1 = 0 to (d0) -> (d0 + 5) (%N) {
Fix test cases that were accessing out of bounds to start with (b/123072438) - detected with memref-bound-check - fixes b/123072438; while on this, fix another test case which was reported out of bounds PiperOrigin-RevId: 229978187
2019-01-19 04:55:46 +08:00
store %c0, %m[%i0, %i1] : memref<? x ? x f32>
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to %M {
for %i3 = 0 to %N {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
%idy = affine.apply (d0)[s0] -> (d0 + s0) (%i3)[%s]
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
%v = load %m[%i2, %idy] : memref<? x ? x f32>
Extend loop-fusion's slicing utility + other fixes / updates - refactor toAffineFromEq and the code surrounding it; refactor code into FlatAffineConstraints::getSliceBounds - add FlatAffineConstraints methods to detect identifiers as mod's and div's of other identifiers - add FlatAffineConstraints::getConstantLower/UpperBound - Address b/122118218 (don't assert on invalid fusion depths cmdline flags - instead, don't do anything; change cmdline flags src-loop-depth -> fusion-src-loop-depth - AffineExpr/Map print method update: don't fail on null instances (since we have a wrapper around a pointer, it's avoidable); rationale: dump/print methods should never fail if possible. - Update memref-dataflow-opt to add an optimization to avoid a unnecessary call to IsRangeOneToOne when it's trivially going to be true. - Add additional test cases to exercise the new support - update a few existing test cases since the maps are now generated uniformly with all destination loop operands appearing for the backward slice - Fix projectOut - fix wrong range for getBestElimCandidate. - Fix for getConstantBoundOnDimSize() - didn't show up in any test cases since we didn't have any non-hyperrectangular ones. PiperOrigin-RevId: 228265152
2019-01-08 09:34:26 +08:00
}
}
return
}
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// -----
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK-DAG: [[MAP0:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// CHECK-LABEL: func @should_fuse_reduction_at_depth1
func @should_fuse_reduction_at_depth1() {
%a = alloc() : memref<10x100xf32>
%b = alloc() : memref<10xf32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
for %i1 = 0 to 100 {
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
%v0 = load %b[%i0] : memref<10xf32>
%v1 = load %a[%i0, %i1] : memref<10x100xf32>
%v2 = "maxf"(%v0, %v1) : (f32, f32) -> f32
store %v2, %b[%i0] : memref<10xf32>
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 10 {
for %i3 = 0 to 100 {
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
%v3 = load %b[%i2] : memref<10xf32>
%v4 = load %a[%i2, %i3] : memref<10x100xf32>
%v5 = subf %v4, %v3 : f32
store %v5, %b[%i2] : memref<10xf32>
}
}
// This test should fuse the src reduction loop at depth 1 in the destination
// loop nest, which improves locality and enables subsequence passes to
// decrease the reduction memref size and possibly place it in a faster
// memory space.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
// CHECK-NEXT: for %i1 = 0 to 100 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %2 = affine.apply [[MAP0]](%i0, %i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %3 = load %0[%2] : memref<1xf32>
// CHECK-NEXT: %4 = load %1[%i0, %i1] : memref<10x100xf32>
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// CHECK-NEXT: %5 = "maxf"(%3, %4) : (f32, f32) -> f32
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %6 = affine.apply [[MAP0]](%i0, %i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %5, %0[%6] : memref<1xf32>
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i2 = 0 to 100 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %7 = affine.apply [[MAP0]](%i0, %i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %8 = load %0[%7] : memref<1xf32>
// CHECK-NEXT: %9 = load %1[%i0, %i2] : memref<10x100xf32>
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-NEXT: %10 = subf %9, %8 : f32
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %11 = affine.apply [[MAP0]](%i0, %i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %10, %0[%11] : memref<1xf32>
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1, d2) -> (-d0 + d1)
// CHECK: [[MAP1:#map[0-9]+]] = (d0, d1, d2) -> (d2)
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// CHECK-LABEL: func @should_fuse_at_src_depth1_and_dst_depth1
func @should_fuse_at_src_depth1_and_dst_depth1() {
%a = alloc() : memref<100x16xf32>
%b = alloc() : memref<100x16xf32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 100 {
for %i1 = 0 to 16 {
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
%v0 = load %a[%i0, %i1] : memref<100x16xf32>
"op0"(%v0) : (f32) -> ()
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 16 {
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
%v1 = "op1"() : () -> (f32)
store %v1, %b[%i0, %i2] : memref<100x16xf32>
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i3 = 0 to 100 {
for %i4 = 0 to 16 {
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
%v2 = load %b[%i3, %i4] : memref<100x16xf32>
"op2"(%v2) : (f32) -> ()
}
}
// We can slice iterations of the '%i0' and '%i1' loops in the the source
// loop nest, but slicing at depth 2 and inserting the slice in the
// destination loop nest at depth2 causes extra computation. Instead,
// the fusion algorithm should detect that the source loop should be sliced
// at depth 1 and the slice should be inserted at depth 1.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 100 {
// CHECK-NEXT: for %i1 = 0 to 16 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %2 = load %1[%i0, %i1] : memref<100x16xf32>
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-NEXT: "op0"(%2) : (f32) -> ()
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i2 = 0 to 16 {
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-NEXT: %3 = "op1"() : () -> f32
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %4 = affine.apply [[MAP0]](%i0, %i0, %i2)
// CHECK-NEXT: %5 = affine.apply [[MAP1]](%i0, %i0, %i2)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %3, %0[%4, %5] : memref<1x16xf32>
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i3 = 0 to 16 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %6 = affine.apply [[MAP0]](%i0, %i0, %i3)
// CHECK-NEXT: %7 = affine.apply [[MAP1]](%i0, %i0, %i3)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %8 = load %0[%6, %7] : memref<1x16xf32>
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-NEXT: "op2"(%8) : (f32) -> ()
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1) -> (d0 * 10 + d1)
// CHECK: [[MAP1:#map[0-9]+]] = (d0, d1, d2) -> (d0 * -10 - d1 + d2)
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// CHECK-LABEL: func @should_fuse_src_depth1_at_dst_depth2
func @should_fuse_src_depth1_at_dst_depth2() {
%a = alloc() : memref<100xf32>
%c0 = constant 0.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 100 {
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
store %c0, %a[%i0] : memref<100xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
for %i2 = 0 to 10 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
%a0 = affine.apply (d0, d1) -> (d0 * 10 + d1) (%i1, %i2)
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
%v0 = load %a[%a0] : memref<100xf32>
}
}
// The source loop nest slice loop bound is a function of both destination
// loop IVs, so we should slice at depth 1 and insert the slice at depth 2.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
// CHECK-NEXT: for %i1 = 0 to 10 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %1 = affine.apply [[MAP0]](%i0, %i1)
// CHECK-NEXT: %2 = affine.apply [[MAP1]](%i0, %i1, %1)
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-NEXT: store %cst, %0[%2] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %3 = affine.apply [[MAP0]](%i0, %i1)
// CHECK-NEXT: %4 = affine.apply [[MAP1]](%i0, %i1, %3)
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-NEXT: %5 = load %0[%4] : memref<1xf32>
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
// CHECK-LABEL: func @fusion_at_depth0_not_currently_supported
func @fusion_at_depth0_not_currently_supported() {
%0 = alloc() : memref<10xf32>
%c0 = constant 0 : index
%cst = constant 0.000000e+00 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
store %cst, %0[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
%1 = load %0[%c0] : memref<10xf32>
}
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// NOTE: Should shrink memref size to 1 element access by load in dst loop
// nest, and make the store in the slice store to the same element.
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-DAG: %0 = alloc() : memref<1xf32>
LoopFusion: perform a series of loop interchanges to increase the loop depth at which slices of producer loop nests can be fused into constumer loop nests. *) Adds utility to LoopUtils to perform loop interchange of two AffineForOps. *) Adds utility to LoopUtils to sink a loop to a specified depth within a loop nest, using a series of loop interchanges. *) Computes dependences between all loads and stores in the loop nest, and classifies each loop as parallel or sequential. *) Computes loop interchange permutation required to sink sequential loops (and raise parallel loop nests) while preserving relative order among them. *) Checks each dependence against the permutation to make sure that dependences would not be violated by the loop interchange transformation. *) Calls loop interchange in LoopFusion pass on consumer loop nests before fusing in producers, sinking loops with loop carried dependences deeper into the consumer loop nest. *) Adds and updates related unit tests. PiperOrigin-RevId: 234158370
2019-02-16 01:32:18 +08:00
// CHECK: for %i0 = 0 to 10 {
Update direction vector computation to use FlatAffineConstraints::getLower/UpperBounds. Update FlatAffineConstraints::getLower/UpperBounds to project to the identifier for which bounds are being computed. This change enables computing bounds on an identifier which were previously dependent on the bounds of another identifier. PiperOrigin-RevId: 234017514
2019-02-15 05:25:46 +08:00
// CHECK-NEXT: store %cst, %0[%c0] : memref<1xf32>
// CHECK-NEXT: %1 = load %0[%c0_0] : memref<1xf32>
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9) -> (-d0 + d4)
// CHECK: [[MAP1:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9) -> (-d1 + d5)
// CHECK: [[MAP2:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9) -> (-d2 + d6)
// CHECK: [[MAP3:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9) -> (-d3 + d7)
// CHECK: [[MAP4:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9) -> (d8)
// CHECK: [[MAP5:#map[0-9]+]] = (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9) -> (d9)
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// CHECK-LABEL: func @should_fuse_deep_loop_nests
func @should_fuse_deep_loop_nests() {
%0 = alloc() : memref<2x2x3x3x16x10xf32, 2>
%1 = alloc() : memref<2x2x3x3x16x10xf32, 2>
%2 = alloc() : memref<3x3x3x3x16x10xf32, 2>
%c0 = constant 0 : index
%c1 = constant 1 : index
%c1_0 = constant 1 : index
%cst = constant 0.000000e+00 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 2 {
for %i1 = 0 to 2 {
for %i2 = 0 to 3 {
for %i3 = 0 to 3 {
for %i4 = 0 to 16 {
for %i5 = 0 to 10 {
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
%3 = load %0[%i0, %i1, %i2, %i3, %i4, %i5]
: memref<2x2x3x3x16x10xf32, 2>
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i6 = 0 to 16 {
for %i7 = 0 to 10 {
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
store %cst, %1[%i0, %i1, %i2, %i3, %i6, %i7]
: memref<2x2x3x3x16x10xf32, 2>
}
}
}
}
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i8 = 0 to 3 {
for %i9 = 0 to 3 {
for %i10 = 0 to 2 {
for %i11 = 0 to 2 {
for %i12 = 0 to 3 {
for %i13 = 0 to 3 {
for %i14 = 0 to 2 {
for %i15 = 0 to 2 {
for %i16 = 0 to 16 {
for %i17 = 0 to 10 {
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
%5 = load %0[%i14, %i15, %i12, %i13, %i16, %i17]
: memref<2x2x3x3x16x10xf32, 2>
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i18 = 0 to 16 {
for %i19 = 0 to 10 {
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
%6 = load %1[%i10, %i11, %i8, %i9, %i18, %i19]
: memref<2x2x3x3x16x10xf32, 2>
}
}
}
}
}
}
}
}
}
}
// The first four loops of the source loop nest can be sliced with iteration
// bounds which are a function of the first four loops of destination loop nest,
// where the destination loops nests have been interchanged.
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-DAG: %0 = alloc() : memref<1x1x1x1x16x10xf32, 2>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 3 {
// CHECK-NEXT: for %i1 = 0 to 3 {
// CHECK-NEXT: for %i2 = 0 to 2 {
// CHECK-NEXT: for %i3 = 0 to 2 {
// CHECK-NEXT: for %i4 = 0 to 3 {
// CHECK-NEXT: for %i5 = 0 to 3 {
// CHECK-NEXT: for %i6 = 0 to 16 {
// CHECK-NEXT: for %i7 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %3 = load %1[%i2, %i3, %i0, %i1, %i6, %i7] : memref<2x2x3x3x16x10xf32, 2>
Update fusion cost model + some additional infrastructure and debug information for -loop-fusion - update fusion cost model to fuse while tolerating a certain amount of redundant computation; add cl option -fusion-compute-tolerance evaluate memory footprint and intermediate memory reduction - emit debug info from -loop-fusion showing what was fused and why - introduce function to compute memory footprint for a loop nest - getMemRefRegion readability update - NFC PiperOrigin-RevId: 230541857
2019-01-24 01:16:24 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i8 = 0 to 16 {
// CHECK-NEXT: for %i9 = 0 to 10 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %4 = affine.apply [[MAP0]](%i2, %i3, %i0, %i1, %i2, %i3, %i0, %i1, %i8, %i9)
// CHECK-NEXT: %5 = affine.apply [[MAP1]](%i2, %i3, %i0, %i1, %i2, %i3, %i0, %i1, %i8, %i9)
// CHECK-NEXT: %6 = affine.apply [[MAP2]](%i2, %i3, %i0, %i1, %i2, %i3, %i0, %i1, %i8, %i9)
// CHECK-NEXT: %7 = affine.apply [[MAP3]](%i2, %i3, %i0, %i1, %i2, %i3, %i0, %i1, %i8, %i9)
// CHECK-NEXT: %8 = affine.apply [[MAP4]](%i2, %i3, %i0, %i1, %i2, %i3, %i0, %i1, %i8, %i9)
// CHECK-NEXT: %9 = affine.apply [[MAP5]](%i2, %i3, %i0, %i1, %i2, %i3, %i0, %i1, %i8, %i9)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %cst, %0[%4, %5, %6, %7, %8, %9] : memref<1x1x1x1x16x10xf32, 2>
Update fusion cost model + some additional infrastructure and debug information for -loop-fusion - update fusion cost model to fuse while tolerating a certain amount of redundant computation; add cl option -fusion-compute-tolerance evaluate memory footprint and intermediate memory reduction - emit debug info from -loop-fusion showing what was fused and why - introduce function to compute memory footprint for a loop nest - getMemRefRegion readability update - NFC PiperOrigin-RevId: 230541857
2019-01-24 01:16:24 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i10 = 0 to 2 {
// CHECK-NEXT: for %i11 = 0 to 2 {
// CHECK-NEXT: for %i12 = 0 to 16 {
// CHECK-NEXT: for %i13 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %10 = load %1[%i10, %i11, %i4, %i5, %i12, %i13] : memref<2x2x3x3x16x10xf32, 2>
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i14 = 0 to 16 {
// CHECK-NEXT: for %i15 = 0 to 10 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %11 = affine.apply [[MAP0]](%i2, %i3, %i0, %i1, %i2, %i3, %i0, %i1, %i14, %i15)
// CHECK-NEXT: %12 = affine.apply [[MAP1]](%i2, %i3, %i0, %i1, %i2, %i3, %i0, %i1, %i14, %i15)
// CHECK-NEXT: %13 = affine.apply [[MAP2]](%i2, %i3, %i0, %i1, %i2, %i3, %i0, %i1, %i14, %i15)
// CHECK-NEXT: %14 = affine.apply [[MAP3]](%i2, %i3, %i0, %i1, %i2, %i3, %i0, %i1, %i14, %i15)
// CHECK-NEXT: %15 = affine.apply [[MAP4]](%i2, %i3, %i0, %i1, %i2, %i3, %i0, %i1, %i14, %i15)
// CHECK-NEXT: %16 = affine.apply [[MAP5]](%i2, %i3, %i0, %i1, %i2, %i3, %i0, %i1, %i14, %i15)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %17 = load %0[%11, %12, %13, %14, %15, %16] : memref<1x1x1x1x16x10xf32, 2>
LoopFusion: automate selection of source loop nest slice depth and destination loop nest insertion depth based on a simple cost model (cost model can be extended/replaced at a later time). *) LoopFusion: Adds fusion cost function which compares the cost of the fused loop nest, with the cost of the two unfused loop nests to determine if it is profitable to fuse the candidate loop nests. The fusion cost function is run for various combinations for src/dst loop depths attempting find the minimum cost setting for src/dst loop depths which does not increase the computational cost when the loop nests are fused. Combinations of src/dst loop depth are evaluated attempting to maximize loop depth (i.e. take a bigger computation slice from the source loop nest, and insert it deeper in the destination loop nest for better locality). *) LoopFusion: Adds utility to compute op instance count for loop nests, sliced loop nests, and to compute the cost of a loop nest fused with another sliced loop nest. *) LoopFusion: canonicalizes slice bound AffineMaps (and updates related tests). *) Analysis::Utils: Splits getBackwardComputationSlice into two functions: one which calculates and returns the slice loop bounds for analysis by LoopFusion, and the other for insertion of the computation slice (ones fusion has calculated the min-cost src/dst loop depths). *) Test: Adds multiple unit tests to test the new functionality. PiperOrigin-RevId: 229219757
2019-01-15 03:26:25 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
// -----
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1, d2) -> (-d0 + d1)
// CHECK: [[MAP1:#map[0-9]+]] = (d0, d1, d2) -> (d2)
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
// CHECK-LABEL: func @should_fuse_at_depth1_and_reduce_slice_trip_count
func @should_fuse_at_depth1_and_reduce_slice_trip_count() {
%a = alloc() : memref<4x256xf32>
%b = alloc() : memref<4x256xf32>
%c0 = constant 0 : index
%cf0 = constant 0.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 4 {
for %i1 = 0 to 256 {
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
%v0 = load %b[%i0, %i1] : memref<4x256xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 256 {
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
store %cf0, %a[%i0, %i2] : memref<4x256xf32>
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %d0 = 0 to 4 {
for %d1 = 0 to 16 {
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
%v1 = load %a[%d0, %d1] : memref<4x256xf32>
}
}
// The cost of fusing at depth 2 is greater than the cost of fusing at depth 1
// for two reasons:
// 1) Inserting the unsliceable src loop %i1 to a higher depth removes
// redundant computation and reduces costs.
// 2) Inserting the sliceable src loop %i2 at depth 1, we can still reduce
// its trip count to 16 (from 256) reducing costs.
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// NOTE: the size of the private memref created for the fused loop nest
// is reduced from the original shape from 4x256 to 4x16 because of the
// data accessed by the load.
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-DAG: %0 = alloc() : memref<1x16xf32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 4 {
// CHECK-NEXT: for %i1 = 0 to 256 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %2 = load %1[%i0, %i1] : memref<4x256xf32>
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i2 = 0 to 16 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %3 = affine.apply [[MAP0]](%i0, %i0, %i2)
// CHECK-NEXT: %4 = affine.apply [[MAP1]](%i0, %i0, %i2)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %cst, %0[%3, %4] : memref<1x16xf32>
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i3 = 0 to 16 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %5 = affine.apply [[MAP0]](%i0, %i0, %i3)
// CHECK-NEXT: %6 = affine.apply [[MAP1]](%i0, %i0, %i3)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %7 = load %0[%5, %6] : memref<1x16xf32>
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
// CHECK-LABEL: func @should_fuse_at_depth1_with_trip_count_20
func @should_fuse_at_depth1_with_trip_count_20() {
%a = alloc() : memref<100xf32>
%c0 = constant 0 : index
%cf0 = constant 0.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 100 {
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
store %cf0, %a[%i0]: memref<100xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 5 {
for %i2 = 0 to 10 {
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
%v0 = load %a[%i2]: memref<100xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i3 = 0 to 10 {
for %i4 = 0 to 20 {
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
%v1 = load %a[%i4]: memref<100xf32>
}
}
}
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// NOTE: The size of the private memref created for fusion is shrunk to 20xf32
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-DAG: %0 = alloc() : memref<20xf32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 5 {
// CHECK-NEXT: for %i1 = 0 to 20 {
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// CHECK-NEXT: store %cst, %0[%i1] : memref<20xf32>
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i2 = 0 to 10 {
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// CHECK-NEXT: %1 = load %0[%i2] : memref<20xf32>
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i3 = 0 to 10 {
// CHECK-NEXT: for %i4 = 0 to 20 {
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// CHECK-NEXT: %2 = load %0[%i4] : memref<20xf32>
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
// CHECK-LABEL: func @should_fuse_at_depth1_with_trip_count_19
func @should_fuse_at_depth1_with_trip_count_19() {
%a = alloc() : memref<100xf32>
%c0 = constant 0 : index
%cf0 = constant 0.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 100 {
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
store %cf0, %a[%i0]: memref<100xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 5 {
for %i2 = 0 to 19 {
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
%v0 = load %a[%i2]: memref<100xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i3 = 0 to 10 {
for %i4 = 0 to 10 {
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
%v1 = load %a[%i4]: memref<100xf32>
}
}
}
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// NOTE: The size of the private memref created for fusion is shrunk to 19xf32
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-DAG: %0 = alloc() : memref<19xf32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 5 {
// CHECK-NEXT: for %i1 = 0 to 19 {
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// CHECK-NEXT: store %cst, %0[%i1] : memref<19xf32>
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i2 = 0 to 19 {
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// CHECK-NEXT: %1 = load %0[%i2] : memref<19xf32>
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i3 = 0 to 10 {
// CHECK-NEXT: for %i4 = 0 to 10 {
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// CHECK-NEXT: %2 = load %0[%i4] : memref<19xf32>
LoopFusion improvements: *) Adds support for fusing into consumer loop nests with multiple loads from the same memref. *) Adds support for reducing slice loop trip count by projecting out destination loop IVs greater than destination loop depth. *) Removes dependence on src loop depth and simplifies cost model computation. PiperOrigin-RevId: 229575126
2019-01-17 01:55:02 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// -----
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// CHECK-LABEL: func @should_fuse_with_private_memrefs_with_diff_shapes() {
func @should_fuse_with_private_memrefs_with_diff_shapes() {
%m = alloc() : memref<100xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 100 {
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
store %cf7, %m[%i0] : memref<100xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 17 {
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
%v0 = load %m[%i1] : memref<100xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 82 {
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
%v1 = load %m[%i2] : memref<100xf32>
}
// Should create two new private memrefs customized to the shapes accessed
// by loops %i1 and %i2.
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-DAG: %0 = alloc() : memref<1xf32>
// CHECK-DAG: %1 = alloc() : memref<1xf32>
Loop fusion for input reuse. *) Breaks fusion pass into multiple sub passes over nodes in data dependence graph: - first pass fuses single-use producers into their unique consumer. - second pass enables fusing for input-reuse by fusing sibling nodes which read from the same memref, but which do not share dependence edges. - third pass fuses remaining producers into their consumers (Note that the sibling fusion pass may have transformed a producer with multiple uses into a single-use producer). *) Fusion for input reuse is enabled by computing a sibling node slice using the load/load accesses to the same memref, and fusion safety is guaranteed by checking that the sibling node memref write region (to a different memref) is preserved. *) Enables output vector and output matrix computations from KFAC patches-second-moment operation to fuse into a single loop nest and reuse input from the image patches operation. *) Adds a generic loop utilitiy for finding all sequential loops in a loop nest. *) Adds and updates unit tests. PiperOrigin-RevId: 236350987
2019-03-02 03:50:25 +08:00
// CHECK: for %i0 = 0 to 17 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %2 = affine.apply [[MAP0]](%i0, %i0)
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
// CHECK-NEXT: store %cst, %1[%2] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %3 = affine.apply [[MAP0]](%i0, %i0)
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
// CHECK-NEXT: %4 = load %1[%3] : memref<1xf32>
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// CHECK-NEXT: }
Loop fusion for input reuse. *) Breaks fusion pass into multiple sub passes over nodes in data dependence graph: - first pass fuses single-use producers into their unique consumer. - second pass enables fusing for input-reuse by fusing sibling nodes which read from the same memref, but which do not share dependence edges. - third pass fuses remaining producers into their consumers (Note that the sibling fusion pass may have transformed a producer with multiple uses into a single-use producer). *) Fusion for input reuse is enabled by computing a sibling node slice using the load/load accesses to the same memref, and fusion safety is guaranteed by checking that the sibling node memref write region (to a different memref) is preserved. *) Enables output vector and output matrix computations from KFAC patches-second-moment operation to fuse into a single loop nest and reuse input from the image patches operation. *) Adds a generic loop utilitiy for finding all sequential loops in a loop nest. *) Adds and updates unit tests. PiperOrigin-RevId: 236350987
2019-03-02 03:50:25 +08:00
// CHECK-NEXT: for %i1 = 0 to 82 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %5 = affine.apply [[MAP0]](%i1, %i1)
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
// CHECK-NEXT: store %cst, %0[%5] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %6 = affine.apply [[MAP0]](%i1, %i1)
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
// CHECK-NEXT: %7 = load %0[%6] : memref<1xf32>
LoopFusion: Creates private MemRefs which are used only by operations in the fused loop. *) Enables reduction of private memref size based on MemRef region accessed by fused slice. *) Enables maximal fusion by creating a private memref to break a fusion-preventing dependence. *) Adds maximal fusion flag to enable fusing as much as possible (though it still fuses the minimum cost computation slice). PiperOrigin-RevId: 229936698
2019-01-19 00:56:27 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: return
return
Fix test cases that were accessing out of bounds to start with (b/123072438) - detected with memref-bound-check - fixes b/123072438; while on this, fix another test case which was reported out of bounds PiperOrigin-RevId: 229978187
2019-01-19 04:55:46 +08:00
}
Handle escaping memrefs in loop fusion pass: *) Do not remove loop nests which write to memrefs which escape the function. *) Do not remove memrefs which escape the function (e.g. are used in the return instruction). PiperOrigin-RevId: 230398630
2019-01-23 05:23:37 +08:00
// -----
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
// CHECK-LABEL: func @should_not_fuse_live_out_arg(%arg0: memref<10xf32>) {
func @should_not_fuse_live_out_arg(%arg0: memref<10xf32>) {
Handle escaping memrefs in loop fusion pass: *) Do not remove loop nests which write to memrefs which escape the function. *) Do not remove memrefs which escape the function (e.g. are used in the return instruction). PiperOrigin-RevId: 230398630
2019-01-23 05:23:37 +08:00
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Handle escaping memrefs in loop fusion pass: *) Do not remove loop nests which write to memrefs which escape the function. *) Do not remove memrefs which escape the function (e.g. are used in the return instruction). PiperOrigin-RevId: 230398630
2019-01-23 05:23:37 +08:00
store %cf7, %arg0[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Handle escaping memrefs in loop fusion pass: *) Do not remove loop nests which write to memrefs which escape the function. *) Do not remove memrefs which escape the function (e.g. are used in the return instruction). PiperOrigin-RevId: 230398630
2019-01-23 05:23:37 +08:00
%v0 = load %arg0[%i1] : memref<10xf32>
}
// This tests that the loop nest '%i0' should not be removed after fusion
// because it writes to memref argument '%arg0'.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
Handle escaping memrefs in loop fusion pass: *) Do not remove loop nests which write to memrefs which escape the function. *) Do not remove memrefs which escape the function (e.g. are used in the return instruction). PiperOrigin-RevId: 230398630
2019-01-23 05:23:37 +08:00
// CHECK-NEXT: store %cst, %arg0[%i0] : memref<10xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i1 = 0 to 10 {
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
// CHECK-NEXT: %0 = load %arg0[%i1] : memref<10xf32>
Handle escaping memrefs in loop fusion pass: *) Do not remove loop nests which write to memrefs which escape the function. *) Do not remove memrefs which escape the function (e.g. are used in the return instruction). PiperOrigin-RevId: 230398630
2019-01-23 05:23:37 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
// CHECK-LABEL: func @should_not_fuse_escaping_memref() -> memref<10xf32>
func @should_not_fuse_escaping_memref() -> memref<10xf32> {
Handle escaping memrefs in loop fusion pass: *) Do not remove loop nests which write to memrefs which escape the function. *) Do not remove memrefs which escape the function (e.g. are used in the return instruction). PiperOrigin-RevId: 230398630
2019-01-23 05:23:37 +08:00
%cf7 = constant 7.0 : f32
%m = alloc() : memref<10xf32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Handle escaping memrefs in loop fusion pass: *) Do not remove loop nests which write to memrefs which escape the function. *) Do not remove memrefs which escape the function (e.g. are used in the return instruction). PiperOrigin-RevId: 230398630
2019-01-23 05:23:37 +08:00
store %cf7, %m[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Handle escaping memrefs in loop fusion pass: *) Do not remove loop nests which write to memrefs which escape the function. *) Do not remove memrefs which escape the function (e.g. are used in the return instruction). PiperOrigin-RevId: 230398630
2019-01-23 05:23:37 +08:00
%v0 = load %m[%i1] : memref<10xf32>
}
// This tests that the loop nest '%i0' should not be removed after fusion
// because it writes to memref '%m' which is returned by the function.
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
// CHECK-DAG: %0 = alloc() : memref<10xf32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
// CHECK-NEXT: store %cst, %0[%i0] : memref<10xf32>
Handle escaping memrefs in loop fusion pass: *) Do not remove loop nests which write to memrefs which escape the function. *) Do not remove memrefs which escape the function (e.g. are used in the return instruction). PiperOrigin-RevId: 230398630
2019-01-23 05:23:37 +08:00
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i1 = 0 to 10 {
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
// CHECK-NEXT: %1 = load %0[%i1] : memref<10xf32>
Handle escaping memrefs in loop fusion pass: *) Do not remove loop nests which write to memrefs which escape the function. *) Do not remove memrefs which escape the function (e.g. are used in the return instruction). PiperOrigin-RevId: 230398630
2019-01-23 05:23:37 +08:00
// CHECK-NEXT: }
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
// CHECK-NEXT: return %0 : memref<10xf32>
Handle escaping memrefs in loop fusion pass: *) Do not remove loop nests which write to memrefs which escape the function. *) Do not remove memrefs which escape the function (e.g. are used in the return instruction). PiperOrigin-RevId: 230398630
2019-01-23 05:23:37 +08:00
return %m : memref<10xf32>
}
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// -----
// This should fuse with the %in becoming a 1x1x1.
func @R3_to_R2_reshape() {
%in = alloc() : memref<2x3x16xi32>
%c0 = constant 0 : index
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 2 {
for %i1 = 0 to 3 {
for %i2 = 0 to 16 {
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
%val = "foo"(%i0, %i1, %i2) : (index, index, index) -> i32
store %val, %in[%i0, %i1, %i2] : memref<2x3x16xi32>
}
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %ii = 0 to 32 {
for %jj = 0 to 3 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
%a0 = affine.apply (d0, d1) -> (d0 * 3 + d1) (%ii, %jj)
%idx = affine.apply (d0) -> (d0 floordiv (3 * 16)) (%a0)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
%v = load %in[%idx, %jj, %c0]
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
: memref<2x3x16xi32>
}
}
return
}
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1) -> ((d0 * 3 + d1) floordiv 48)
Bug fix for getConstantBoundOnDimSize - this was detected when memref-bound-check was run on the output of the loop-fusion pass - the addition (to represent ceildiv as a floordiv) had to be performed only for the constant term of the constraint - update test cases - memref-bound-check no longer returns an error on the output of this test case PiperOrigin-RevId: 236731137
2019-03-05 06:58:59 +08:00
// CHECK-NEXT: [[MAP2:#map[0-9]+]] = (d0, d1, d2, d3, d4) -> (d2 - (d0 * 3 + d1) floordiv 48)
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK-NEXT: [[MAP3:#map[0-9]+]] = (d0, d1, d2, d3, d4) -> (-d1 + d3)
// CHECK-NEXT: [[MAP4:#map[0-9]+]] = (d0, d1, d2, d3, d4) -> (d4)
// CHECK-NEXT: [[MAP5:#map[0-9]+]] = (d0, d1) -> (d0 * 3 + d1)
// CHECK-NEXT: [[MAP6:#map[0-9]+]] = (d0) -> (d0 floordiv 48)
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
Allocate private/local buffers for slices accurately during fusion - the size of the private memref created for the slice should be based on the memref region accessed at the depth at which the slice is being materialized, i.e., symbolic in the outer IVs up until that depth, as opposed to the region accessed based on the entire domain. - leads to a significant contraction of the temporary / intermediate memref whenever the memref isn't reduced to a single scalar (through store fwd'ing). Other changes - update to promoteIfSingleIteration - avoid introducing unnecessary identity map affine_apply from IV; makes it much easier to write and read test cases and pass output for all passes that use promoteIfSingleIteration; loop-fusion test cases become much simpler - fix replaceAllMemrefUsesWith bug that was exposed by the above update - 'domInstFilter' could be one of the ops erased due to a memref replacement in it. - fix getConstantBoundOnDimSize bug: a division by the coefficient of the identifier was missing (the latter need not always be 1); add lbFloorDivisors output argument - rename getBoundingConstantSizeAndShape -> getConstantBoundingSizeAndShape PiperOrigin-RevId: 230405218
2019-01-23 05:58:52 +08:00
// CHECK-LABEL: func @R3_to_R2_reshape()
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-DAG: %0 = alloc() : memref<1x1x1xi32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 32 {
// CHECK-NEXT: for %i1 = 0 to 3 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %1 = affine.apply [[MAP0]](%i0, %i1)
Update direction vector computation to use FlatAffineConstraints::getLower/UpperBounds. Update FlatAffineConstraints::getLower/UpperBounds to project to the identifier for which bounds are being computed. This change enables computing bounds on an identifier which were previously dependent on the bounds of another identifier. PiperOrigin-RevId: 234017514
2019-02-15 05:25:46 +08:00
// CHECK-NEXT: %2 = "foo"(%1, %i1, %c0) : (index, index, index) -> i32
// CHECK-NEXT: %3 = affine.apply [[MAP2]](%i0, %i1, %1, %i1, %c0)
// CHECK-NEXT: %4 = affine.apply [[MAP3]](%i0, %i1, %1, %i1, %c0)
// CHECK-NEXT: %5 = affine.apply [[MAP4]](%i0, %i1, %1, %i1, %c0)
// CHECK-NEXT: store %2, %0[%3, %4, %5] : memref<1x1x1xi32>
// CHECK-NEXT: %6 = affine.apply [[MAP5]](%i0, %i1)
// CHECK-NEXT: %7 = affine.apply [[MAP6]](%6)
// CHECK-NEXT: %8 = affine.apply [[MAP2]](%i0, %i1, %7, %i1, %c0_0)
// CHECK-NEXT: %9 = affine.apply [[MAP3]](%i0, %i1, %7, %i1, %c0_0)
// CHECK-NEXT: %10 = affine.apply [[MAP4]](%i0, %i1, %7, %i1, %c0_0)
// CHECK-NEXT: %11 = load %0[%8, %9, %10] : memref<1x1x1xi32>
Update replaceAllMemRefUsesWith to generate single result affine_apply's for index remapping - generate a sequence of single result affine_apply's for the index remapping (instead of one multi result affine_apply) - update dma-generate and loop-fusion test cases; while on this, change test cases to use single result affine apply ops - some fusion comment fix/cleanup PiperOrigin-RevId: 230985830
2019-01-26 08:00:50 +08:00
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
Fix single producer check in loop fusion pass. PiperOrigin-RevId: 230565482
2019-01-24 03:11:43 +08:00
// -----
// CHECK-LABEL: func @should_not_fuse_multi_output_producer() {
func @should_not_fuse_multi_output_producer() {
%a = alloc() : memref<10xf32>
%b = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Fix single producer check in loop fusion pass. PiperOrigin-RevId: 230565482
2019-01-24 03:11:43 +08:00
store %cf7, %a[%i0] : memref<10xf32>
store %cf7, %b[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Fix single producer check in loop fusion pass. PiperOrigin-RevId: 230565482
2019-01-24 03:11:43 +08:00
%v0 = load %a[%i1] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
Fix single producer check in loop fusion pass. PiperOrigin-RevId: 230565482
2019-01-24 03:11:43 +08:00
// CHECK-NEXT: store %cst, %0[%i0] : memref<10xf32>
// CHECK-NEXT: store %cst, %1[%i0] : memref<10xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i1 = 0 to 10 {
Fix single producer check in loop fusion pass. PiperOrigin-RevId: 230565482
2019-01-24 03:11:43 +08:00
// CHECK-NEXT: %2 = load %0[%i1] : memref<10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// -----
// CHECK-LABEL: func @fusion_preventing_deps_on_middle_loop() {
func @fusion_preventing_deps_on_middle_loop() {
%a = alloc() : memref<10xf32>
%b = alloc() : memref<10xf32>
%c = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
%v0 = load %a[%i0] : memref<10xf32>
store %v0, %b[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
store %cf7, %a[%i1] : memref<10xf32>
%v1 = load %c[%i1] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
%v2 = load %b[%i2] : memref<10xf32>
store %v2, %c[%i2] : memref<10xf32>
}
// Loops '%i0' and '%i2' cannot fuse along producer/consumer edge on memref
// '%b', because of the WAR dep from '%i0' to '%i1' on memref '%a' and
// because of the WAR dep from '%i1' to '%i2' on memref '%c'.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %3 = load %0[%i0] : memref<10xf32>
// CHECK-NEXT: store %3, %1[%i0] : memref<10xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i1 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %cst, %0[%i1] : memref<10xf32>
// CHECK-NEXT: %4 = load %2[%i1] : memref<10xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i2 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %5 = load %1[%i2] : memref<10xf32>
// CHECK-NEXT: store %5, %2[%i2] : memref<10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-LABEL: func @should_fuse_and_move_to_preserve_war_dep() {
func @should_fuse_and_move_to_preserve_war_dep() {
%a = alloc() : memref<10xf32>
%b = alloc() : memref<10xf32>
%c = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
%v0 = load %b[%i0] : memref<10xf32>
store %v0, %a[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 3 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
%v2 = load %c[%i1] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 5 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
store %cf7, %b[%i2] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i3 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
%v1 = load %a[%i3] : memref<10xf32>
store %cf7, %c[%i3] : memref<10xf32>
}
// Dependence graph:
//
// %i0 ---------
// | |
// --- %i1 | %b | %a
// | | |
// %c | %i2 <-- |
// | |
// --> %i3 <--------
//
// It is possible to fuse loop '%i0' into '%i3' and preserve dependences
// if the fused loop nest is inserted between loops '%i1' and '%i2'.
// CHECK-DAG: %0 = alloc() : memref<1xf32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 3 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %3 = load %2[%i0] : memref<10xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i1 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %4 = load %1[%i1] : memref<10xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %5 = affine.apply [[MAP0]](%i1, %i1)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %4, %0[%5] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %6 = affine.apply [[MAP0]](%i1, %i1)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %7 = load %0[%6] : memref<1xf32>
// CHECK-NEXT: store %cst, %2[%i1] : memref<10xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i2 = 0 to 5 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %cst, %1[%i2] : memref<10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
// CHECK-LABEL: func @fusion_preventing_dep_on_constant() {
func @fusion_preventing_dep_on_constant() {
%a = alloc() : memref<10xf32>
%b = alloc() : memref<10xf32>
%c = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
%v0 = load %b[%i0] : memref<10xf32>
store %cf7, %a[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
store %cf7, %b[%i1] : memref<10xf32>
}
%cf11 = constant 11.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
%v2 = load %a[%i2] : memref<10xf32>
store %cf11, %c[%i2] : memref<10xf32>
}
// Loops '%i0' and '%i2' cannot fuse along producer/consumer edge on memref
// '%a', because of the WAR dep from '%i0' to '%i1' on memref '%b' and
// because of the SSA value dep from '%cf11' def to use in '%i2'.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %3 = load %1[%i0] : memref<10xf32>
// CHECK-NEXT: store %cst, %0[%i0] : memref<10xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i1 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %cst, %1[%i1] : memref<10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: %cst_0 = constant 1.100000e+01 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i2 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %4 = load %0[%i2] : memref<10xf32>
// CHECK-NEXT: store %cst_0, %2[%i2] : memref<10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
Update tests using affine maps to not rely on specific map numbers in the output IR. This is necessary to remove the dependency on ForInst not numbering the AffineMap bounds it has custom formatting for. PiperOrigin-RevId: 231634812
2019-01-31 03:33:32 +08:00
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-LABEL: func @should_fuse_and_preserve_dep_on_constant() {
func @should_fuse_and_preserve_dep_on_constant() {
%a = alloc() : memref<10xf32>
%b = alloc() : memref<10xf32>
%c = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
%cf11 = constant 11.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
%v0 = load %b[%i0] : memref<10xf32>
store %cf7, %a[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
store %cf7, %b[%i1] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
%v2 = load %a[%i2] : memref<10xf32>
store %cf11, %c[%i2] : memref<10xf32>
}
// Loops '%i0' and '%i2' can fuse along producer/consumer edge on memref
// '%a', and preserve the WAR dep from '%i0' to '%i1' on memref '%b', and
// the SSA value dep from '%cf11' def to use in '%i2'.
// CHECK: %cst_0 = constant 1.100000e+01 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i0 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %3 = load %1[%i0] : memref<10xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %4 = affine.apply [[MAP0]](%i0, %i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %cst, %0[%4] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %5 = affine.apply [[MAP0]](%i0, %i0)
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: %6 = load %0[%5] : memref<1xf32>
// CHECK-NEXT: store %cst_0, %2[%i0] : memref<10xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i1 = 0 to 10 {
Support fusing loop nests which require insertion into a new instruction Block position while preserving dependences, opening up additional fusion opportunities. - Adds SSA Value edges to the data dependence graph used in the loop fusion pass. PiperOrigin-RevId: 231417649
2019-01-30 01:36:41 +08:00
// CHECK-NEXT: store %cst, %1[%i1] : memref<10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
// -----
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1, d2) -> (d1)
// CHECK: [[MAP1:#map[0-9]+]] = (d0, d1, d2) -> (-d0 + d2)
// CHECK: [[MAP2:#map[0-9]+]] = (d0, d1) -> (d0 * 16 - d1 + 15)
// CHECK: [[MAP3:#map[0-9]+]] = (d0, d1) -> (d0 * 16 + d1)
// CHECK-LABEL: func @should_fuse_at_depth_above_loop_carried_dependence(%arg0: memref<64x4xf32>, %arg1: memref<64x4xf32>) {
func @should_fuse_at_depth_above_loop_carried_dependence(%arg0: memref<64x4xf32>, %arg1: memref<64x4xf32>) {
%out = alloc() : memref<64x4xf32>
%0 = constant 0.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 64 {
for %i1 = 0 to 4 {
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
store %0, %out[%i0, %i1] : memref<64x4xf32>
}
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 4 {
for %i3 = 0 to 4 {
for %i4 = 0 to 16 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
%1 = affine.apply (d0, d1) -> (d0 * 16 - d1 + 15)(%i3, %i4)
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
%2 = load %arg1[%1, %i2] : memref<64x4xf32>
"op0"(%2) : (f32) -> ()
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i5 = 0 to 4 {
for %i6 = 0 to 16 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
%3 = affine.apply (d0, d1) -> (d0 * 16 - d1 + 15)(%i5, %i6)
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
%4 = load %arg0[%3, %i3] : memref<64x4xf32>
"op1"(%4) : (f32) -> ()
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i7 = 0 to 16 {
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
%5 = "op2"() : () -> (f32)
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
%6 = affine.apply (d0, d1) -> (d0 * 16 + d1)(%i5, %i7)
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
%7 = load %out[%6, %i2] : memref<64x4xf32>
%8 = addf %7, %5 : f32
store %8, %out[%6, %i2] : memref<64x4xf32>
}
}
}
}
// We can fuse source loop nest '%i0' into dst loop nest '%i2', but the
// depth at which we can insert the src loop nest slice into the dst loop
// lest must be decreased because of a loop carried dependence on loop '%i3'.
// As a result, the source loop nest is inserted at dst loop nest depth 1,
// just above the loop with the carried depenence. In addition, the source
// loop nest iteration bounds on its loop '%i1' are reduced to 1, so the
// memref size can be reduced to 128x1xf32.
// CHECK: %0 = alloc() : memref<64x1xf32>
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 4 {
// CHECK-NEXT: for %i1 = 0 to 64 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %1 = affine.apply [[MAP0]](%i0, %i1, %i0)
// CHECK-NEXT: %2 = affine.apply [[MAP1]](%i0, %i1, %i0)
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
// CHECK-NEXT: store %cst, %0[%1, %2] : memref<64x1xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i2 = 0 to 4 {
// CHECK-NEXT: for %i3 = 0 to 16 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %3 = affine.apply [[MAP2]](%i2, %i3)
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
// CHECK-NEXT: %4 = load %arg1[%3, %i0] : memref<64x4xf32>
// CHECK-NEXT: "op0"(%4) : (f32) -> ()
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i4 = 0 to 4 {
// CHECK-NEXT: for %i5 = 0 to 16 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %5 = affine.apply [[MAP2]](%i4, %i5)
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
// CHECK-NEXT: %6 = load %arg0[%5, %i2] : memref<64x4xf32>
// CHECK-NEXT: "op1"(%6) : (f32) -> ()
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i6 = 0 to 16 {
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
// CHECK-NEXT: %7 = "op2"() : () -> f32
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %8 = affine.apply [[MAP3]](%i4, %i6)
// CHECK-NEXT: %9 = affine.apply [[MAP0]](%i0, %8, %i0)
// CHECK-NEXT: %10 = affine.apply [[MAP1]](%i0, %8, %i0)
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
// CHECK-NEXT: %11 = load %0[%9, %10] : memref<64x1xf32>
// CHECK-NEXT: %12 = addf %11, %7 : f32
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %13 = affine.apply [[MAP0]](%i0, %8, %i0)
// CHECK-NEXT: %14 = affine.apply [[MAP1]](%i0, %8, %i0)
LoopFusion: insert the source loop nest slice at a depth in the destination loop nest which preserves dependences (above any loop carried or other dependences). This is accomplished by updating the maximum destination loop depth based on dependence checks between source loop nest loads and stores which access the memref on which the source loop nest has a store op. In addition, prevent fusing in source loop nests which write to memrefs which escape or are live out. PiperOrigin-RevId: 231684492
2019-01-31 07:53:41 +08:00
// CHECK-NEXT: store %12, %0[%13, %14] : memref<64x1xf32>
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
// -----
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
// CHECK-LABEL: func @should_fuse_after_private_memref_creation() {
func @should_fuse_after_private_memref_creation() {
%a = alloc() : memref<10xf32>
%b = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i0 = 0 to 10 {
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
store %cf7, %a[%i0] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i1 = 0 to 10 {
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
%v0 = load %a[%i1] : memref<10xf32>
store %v0, %b[%i1] : memref<10xf32>
}
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
for %i2 = 0 to 10 {
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
%v1 = load %a[%i2] : memref<10xf32>
store %v1, %b[%i2] : memref<10xf32>
}
// On the first visit to '%i2', the fusion algorithm can not fuse loop nest
// '%i0' into '%i2' because of the dependences '%i0' and '%i2' each have on
// '%i1'. However, once the loop nest '%i0' is fused into '%i1' with a
// private memref, the dependence between '%i0' and '%i1' on memref '%a' no
// longer exists, so '%i0' can now be fused into '%i2'.
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK: for %i0 = 0 to 10 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %3 = affine.apply [[MAP0]](%i0, %i0)
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
// CHECK-NEXT: store %cst, %1[%3] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %4 = affine.apply [[MAP0]](%i0, %i0)
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
// CHECK-NEXT: %5 = load %1[%4] : memref<1xf32>
// CHECK-NEXT: store %5, %2[%i0] : memref<10xf32>
// CHECK-NEXT: }
Automated rollback of changelist 232717775. PiperOrigin-RevId: 232807986
2019-02-07 13:54:18 +08:00
// CHECK-NEXT: for %i1 = 0 to 10 {
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %6 = affine.apply [[MAP0]](%i1, %i1)
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
// CHECK-NEXT: store %cst, %0[%6] : memref<1xf32>
NFC: Rename affine_apply to affine.apply. This is the first step to adding a namespace to the affine dialect. PiperOrigin-RevId: 232707862
2019-02-07 03:08:18 +08:00
// CHECK-NEXT: %7 = affine.apply [[MAP0]](%i1, %i1)
Loop fusion improvements: *) After a private memref buffer is created for a fused loop nest, dependences on the old memref are reduced, which can open up fusion opportunities. In these cases, users of the old memref are added back to the worklist to be reconsidered for fusion. *) Fixed a bug in fusion insertion point dependence check where the memref being privatized was being skipped from the check. PiperOrigin-RevId: 232477853
2019-02-05 22:57:08 +08:00
// CHECK-NEXT: %8 = load %0[%7] : memref<1xf32>
// CHECK-NEXT: store %8, %2[%i1] : memref<10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
LoopFusion: perform a series of loop interchanges to increase the loop depth at which slices of producer loop nests can be fused into constumer loop nests. *) Adds utility to LoopUtils to perform loop interchange of two AffineForOps. *) Adds utility to LoopUtils to sink a loop to a specified depth within a loop nest, using a series of loop interchanges. *) Computes dependences between all loads and stores in the loop nest, and classifies each loop as parallel or sequential. *) Computes loop interchange permutation required to sink sequential loops (and raise parallel loop nests) while preserving relative order among them. *) Checks each dependence against the permutation to make sure that dependences would not be violated by the loop interchange transformation. *) Calls loop interchange in LoopFusion pass on consumer loop nests before fusing in producers, sinking loops with loop carried dependences deeper into the consumer loop nest. *) Adds and updates related unit tests. PiperOrigin-RevId: 234158370
2019-02-16 01:32:18 +08:00
// -----
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1) -> (-d0 + d1)
// CHECK-LABEL: func @should_fuse_after_one_loop_interchange() {
func @should_fuse_after_one_loop_interchange() {
%a = alloc() : memref<10xf32>
%cf0 = constant 0.0 : f32
for %i0 = 0 to 10 {
store %cf0, %a[%i0] : memref<10xf32>
}
for %i1 = 0 to 5 {
for %i2 = 0 to 10 {
%v0 = load %a[%i2] : memref<10xf32>
store %v0, %a[%i2] : memref<10xf32>
}
}
// The dependence between the load and store is carried on loop '%i1', and
// cannot be fused with loop '%i0' without violating this dependence.
// Once loops '%i1' and %i2' are interchanged, loop '%i0' can be fused
// at loop depth 1, because the loop carrying the dependence has been
// interchanged and is now at depth 2.
// CHECK: for %i0 = 0 to 10 {
// CHECK-NEXT: %1 = affine.apply [[MAP0]](%i0, %i0)
// CHECK-NEXT: store %cst, %0[%1] : memref<1xf32>
// CHECK-NEXT: for %i1 = 0 to 5 {
// CHECK-NEXT: %2 = affine.apply [[MAP0]](%i0, %i0)
// CHECK-NEXT: %3 = load %0[%2] : memref<1xf32>
// CHECK-NEXT: %4 = affine.apply [[MAP0]](%i0, %i0)
// CHECK-NEXT: store %3, %0[%4] : memref<1xf32>
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
// CHECK: [[MAP0:#map[0-9]+]] = (d0, d1, d2, d3) -> (-d0 + d2)
// CHECK: [[MAP1:#map[0-9]+]] = (d0, d1, d2, d3) -> (-d1 + d3)
// CHECK-LABEL: func @should_fuse_after_two_loop_interchanges() {
func @should_fuse_after_two_loop_interchanges() {
%a = alloc() : memref<6x8xf32>
%cf0 = constant 0.0 : f32
for %i0 = 0 to 6 {
for %i1 = 0 to 8 {
store %cf0, %a[%i0, %i1] : memref<6x8xf32>
}
}
for %i2 = 0 to 4 {
for %i3 = 0 to 6 {
for %i4 = 0 to 2 {
for %i5 = 0 to 8 {
%v0 = load %a[%i3, %i5] : memref<6x8xf32>
%v1 = addf %v0, %v0 : f32
store %v1, %a[%i3, %i5] : memref<6x8xf32>
}
}
}
}
// The dependence between the load and store is carried on loops '%i2' and
// '%i4', and cannot be fused with loop '%i0' without violating this
// dependence.
// Once loop '%i2' is interchanged with loop '%i3', and again with loop
// '%i5', then loop '%i0' can be fused at loop depth 2, because the loop
// carring the dependences have been interchanged with loops at depth > 2.
// CHECK: for %i0 = 0 to 6 {
// CHECK-NEXT: for %i1 = 0 to 8 {
// CHECK-NEXT: %1 = affine.apply [[MAP0]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: %2 = affine.apply [[MAP1]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: store %cst, %0[%1, %2] : memref<1x1xf32>
// CHECK-NEXT: for %i2 = 0 to 4 {
// CHECK-NEXT: for %i3 = 0 to 2 {
// CHECK-NEXT: %3 = affine.apply [[MAP0]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: %4 = affine.apply [[MAP1]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: %5 = load %0[%3, %4] : memref<1x1xf32>
// CHECK-NEXT: %6 = addf %5, %5 : f32
// CHECK-NEXT: %7 = affine.apply [[MAP0]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: %8 = affine.apply [[MAP1]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: store %6, %0[%7, %8] : memref<1x1xf32>
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
Support fusing producer loop nests which write to a memref which is live out, provided that the write region of the consumer loop nest to the same memref is a super set of the producer's write region. PiperOrigin-RevId: 234240958
2019-02-16 09:12:19 +08:00
// -----
func @should_fuse_live_out_writer(%arg0 : memref<10xf32>) -> memref<10xf32> {
%cst = constant 0.000000e+00 : f32
for %i0 = 0 to 10 {
store %cst, %arg0[%i0] : memref<10xf32>
}
for %i1 = 0 to 10 {
%1 = load %arg0[%i1] : memref<10xf32>
store %1, %arg0[%i1] : memref<10xf32>
}
return %arg0 : memref<10xf32>
// CHECK: %cst = constant 0.000000e+00 : f32
// CHECK-NEXT: for %i0 = 0 to 10 {
// CHECK-NEXT: store %cst, %arg0[%i0] : memref<10xf32>
// CHECK-NEXT: %0 = load %arg0[%i0] : memref<10xf32>
// CHECK-NEXT: store %0, %arg0[%i0] : memref<10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: return %arg0 : memref<10xf32>
}
Extend/improve getSliceBounds() / complete TODO + update unionBoundingBox - compute slices precisely where the destination iteration depends on multiple source iterations (instead of over-approximating to the whole source loop extent) - update unionBoundingBox to deal with input with non-matching symbols - reenable disabled backend test case PiperOrigin-RevId: 234714069
2019-02-20 10:17:19 +08:00
// -----
// The fused slice has 16 iterations from along %i0.
// CHECK-DAG: [[MAP_LB:#map[0-9]+]] = (d0) -> (d0 * 16)
// CHECK-DAG: [[MAP_UB:#map[0-9]+]] = (d0) -> (d0 * 16 + 16)
#map = (d0, d1) -> (d0 * 16 + d1)
// CHECK-LABEL: slice_tile
Make sure that fusion test cases don't have out of bounds accesses - fix out of bounds test case - -memref-bound-check on the test/Transforms/loop-fusion.mlir no longer reports any errors, before or after -loop-fusion is run PiperOrigin-RevId: 236757658
2019-03-05 09:26:46 +08:00
func @slice_tile(%arg0: memref<128x8xf32>, %arg1: memref<32x8xf32>, %0 : f32) -> memref<32x8xf32> {
Extend/improve getSliceBounds() / complete TODO + update unionBoundingBox - compute slices precisely where the destination iteration depends on multiple source iterations (instead of over-approximating to the whole source loop extent) - update unionBoundingBox to deal with input with non-matching symbols - reenable disabled backend test case PiperOrigin-RevId: 234714069
2019-02-20 10:17:19 +08:00
for %i0 = 0 to 32 {
for %i1 = 0 to 8 {
Make sure that fusion test cases don't have out of bounds accesses - fix out of bounds test case - -memref-bound-check on the test/Transforms/loop-fusion.mlir no longer reports any errors, before or after -loop-fusion is run PiperOrigin-RevId: 236757658
2019-03-05 09:26:46 +08:00
store %0, %arg1[%i0, %i1] : memref<32x8xf32>
Extend/improve getSliceBounds() / complete TODO + update unionBoundingBox - compute slices precisely where the destination iteration depends on multiple source iterations (instead of over-approximating to the whole source loop extent) - update unionBoundingBox to deal with input with non-matching symbols - reenable disabled backend test case PiperOrigin-RevId: 234714069
2019-02-20 10:17:19 +08:00
}
}
for %i = 0 to 2 {
for %j = 0 to 8 {
for %k = 0 to 8 {
for %kk = 0 to 16 {
%1 = affine.apply #map(%k, %kk)
Make sure that fusion test cases don't have out of bounds accesses - fix out of bounds test case - -memref-bound-check on the test/Transforms/loop-fusion.mlir no longer reports any errors, before or after -loop-fusion is run PiperOrigin-RevId: 236757658
2019-03-05 09:26:46 +08:00
%2 = load %arg0[%1, %j] : memref<128x8xf32>
Extend/improve getSliceBounds() / complete TODO + update unionBoundingBox - compute slices precisely where the destination iteration depends on multiple source iterations (instead of over-approximating to the whole source loop extent) - update unionBoundingBox to deal with input with non-matching symbols - reenable disabled backend test case PiperOrigin-RevId: 234714069
2019-02-20 10:17:19 +08:00
%3 = "foo"(%2) : (f32) -> f32
}
for %ii = 0 to 16 {
%6 = affine.apply #map(%i, %ii)
Make sure that fusion test cases don't have out of bounds accesses - fix out of bounds test case - -memref-bound-check on the test/Transforms/loop-fusion.mlir no longer reports any errors, before or after -loop-fusion is run PiperOrigin-RevId: 236757658
2019-03-05 09:26:46 +08:00
%7 = load %arg1[%6, %j] : memref<32x8xf32>
Extend/improve getSliceBounds() / complete TODO + update unionBoundingBox - compute slices precisely where the destination iteration depends on multiple source iterations (instead of over-approximating to the whole source loop extent) - update unionBoundingBox to deal with input with non-matching symbols - reenable disabled backend test case PiperOrigin-RevId: 234714069
2019-02-20 10:17:19 +08:00
%8 = addf %7, %7 : f32
Make sure that fusion test cases don't have out of bounds accesses - fix out of bounds test case - -memref-bound-check on the test/Transforms/loop-fusion.mlir no longer reports any errors, before or after -loop-fusion is run PiperOrigin-RevId: 236757658
2019-03-05 09:26:46 +08:00
store %8, %arg1[%6, %j] : memref<32x8xf32>
Extend/improve getSliceBounds() / complete TODO + update unionBoundingBox - compute slices precisely where the destination iteration depends on multiple source iterations (instead of over-approximating to the whole source loop extent) - update unionBoundingBox to deal with input with non-matching symbols - reenable disabled backend test case PiperOrigin-RevId: 234714069
2019-02-20 10:17:19 +08:00
}
}
}
}
Make sure that fusion test cases don't have out of bounds accesses - fix out of bounds test case - -memref-bound-check on the test/Transforms/loop-fusion.mlir no longer reports any errors, before or after -loop-fusion is run PiperOrigin-RevId: 236757658
2019-03-05 09:26:46 +08:00
return %arg1 : memref<32x8xf32>
Extend/improve getSliceBounds() / complete TODO + update unionBoundingBox - compute slices precisely where the destination iteration depends on multiple source iterations (instead of over-approximating to the whole source loop extent) - update unionBoundingBox to deal with input with non-matching symbols - reenable disabled backend test case PiperOrigin-RevId: 234714069
2019-02-20 10:17:19 +08:00
}
// CHECK: for %i0 = 0 to 2 {
// CHECK-NEXT: for %i1 = 0 to 8 {
// CHECK-NEXT: for %i2 = [[MAP_LB]](%i0) to [[MAP_UB]](%i0) {
// CHECK-NEXT: store %arg2, %arg1[%i2, %i1] : memref<32x8xf32>
// CHECK-NEXT: }
// CHECK-NEXT: for %i3 = 0 to 8 {
// CHECK-NEXT: for %i4 = 0 to 16 {
// CHECK-NEXT: %0 = affine.apply #map{{[0-9]+}}(%i3, %i4)
Make sure that fusion test cases don't have out of bounds accesses - fix out of bounds test case - -memref-bound-check on the test/Transforms/loop-fusion.mlir no longer reports any errors, before or after -loop-fusion is run PiperOrigin-RevId: 236757658
2019-03-05 09:26:46 +08:00
// CHECK-NEXT: %1 = load %arg0[%0, %i1] : memref<128x8xf32>
Extend/improve getSliceBounds() / complete TODO + update unionBoundingBox - compute slices precisely where the destination iteration depends on multiple source iterations (instead of over-approximating to the whole source loop extent) - update unionBoundingBox to deal with input with non-matching symbols - reenable disabled backend test case PiperOrigin-RevId: 234714069
2019-02-20 10:17:19 +08:00
// CHECK-NEXT: %2 = "foo"(%1) : (f32) -> f32
// CHECK-NEXT: }
// CHECK-NEXT: for %i5 = 0 to 16 {
// CHECK-NEXT: %3 = affine.apply #map{{[0-9]+}}(%i0, %i5)
// CHECK-NEXT: %4 = load %arg1[%3, %i1] : memref<32x8xf32>
// CHECK-NEXT: %5 = addf %4, %4 : f32
// CHECK-NEXT: store %5, %arg1[%3, %i1] : memref<32x8xf32>
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return %arg1 : memref<32x8xf32>
// CHECK-NEXT:}
Fix bug in memref region computation with slice loop bounds. Adds loop IV values to ComputationSliceState which are used in FlatAffineConstraints::addSliceBounds, to ensure that constraints are only added for loop IV values which are present in the constraint system. PiperOrigin-RevId: 235952912
2019-02-28 03:01:49 +08:00
// -----
// Test case which illustrates fix for b/126454413
func @test_add_slice_bounds() {
%a = alloc() : memref<10xf32>
%b = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
%c0 = constant 0 : index
for %i0 = 0 to 10 {
for %i1 = 0 to 10 {
for %i2 = 0 to 10 {
%a0 = affine.apply (d0) -> (d0) (%i0)
%a1 = affine.apply (d0) -> (d0) (%i0)
%a2 = affine.apply (d0, d1) -> (d0 - d1) (%a0, %a1)
store %cf7, %a[%a2] : memref<10xf32>
}
}
}
for %i3 = 0 to 10 {
for %i4 = 0 to 10 {
for %i5 = 0 to 10 {
%v0 = load %a[%c0] : memref<10xf32>
}
}
}
// CHECK: for %i0 = 0 to 10 {
// CHECK-NEXT: for %i1 = 0 to 10 {
// CHECK-NEXT: for %i2 = 0 to 10 {
// CHECK-NEXT: %2 = affine.apply #map2(%i0)
// CHECK-NEXT: %3 = affine.apply #map2(%i0)
// CHECK-NEXT: %4 = affine.apply #map3(%2, %3)
// CHECK-NEXT: store %cst, %0[%4] : memref<10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: for %i3 = 0 to 10 {
// CHECK-NEXT: for %i4 = 0 to 10 {
// CHECK-NEXT: for %i5 = 0 to 10 {
// CHECK-NEXT: %5 = load %0[%c0] : memref<10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: }
return
}
Loop fusion for input reuse. *) Breaks fusion pass into multiple sub passes over nodes in data dependence graph: - first pass fuses single-use producers into their unique consumer. - second pass enables fusing for input-reuse by fusing sibling nodes which read from the same memref, but which do not share dependence edges. - third pass fuses remaining producers into their consumers (Note that the sibling fusion pass may have transformed a producer with multiple uses into a single-use producer). *) Fusion for input reuse is enabled by computing a sibling node slice using the load/load accesses to the same memref, and fusion safety is guaranteed by checking that the sibling node memref write region (to a different memref) is preserved. *) Enables output vector and output matrix computations from KFAC patches-second-moment operation to fuse into a single loop nest and reuse input from the image patches operation. *) Adds a generic loop utilitiy for finding all sequential loops in a loop nest. *) Adds and updates unit tests. PiperOrigin-RevId: 236350987
2019-03-02 03:50:25 +08:00
// -----
// CHECK-DAG: [[MAP0:#map[0-9]+]] = (d0, d1, d2, d3) -> (-d0 + d2)
// CHECK-DAG: [[MAP1:#map[0-9]+]] = (d0, d1, d2, d3) -> (-d1 + d3)
func @should_fuse_init_loops_siblings_then_shared_producer(%arg0: memref<10x10xf32>, %arg1: memref<10x10xf32>) {
%0 = alloc() : memref<10x10xf32>
%cst = constant 0.000000e+00 : f32
%cst_0 = constant 1.000000e+00 : f32
%cst_1 = constant 7.000000e+00 : f32
for %i0 = 0 to 10 {
for %i1 = 0 to 10 {
store %cst_1, %0[%i0, %i1] : memref<10x10xf32>
}
}
for %i2 = 0 to 3 {
for %i3 = 0 to 3 {
store %cst, %arg0[%i2, %i3] : memref<10x10xf32>
}
}
for %i4 = 0 to 3 {
for %i5 = 0 to 3 {
%1 = load %0[%i4, %i5] : memref<10x10xf32>
%2 = load %arg0[%i4, %i5] : memref<10x10xf32>
%3 = mulf %1, %2 : f32
store %3, %arg0[%i4, %i5] : memref<10x10xf32>
}
}
for %i6 = 0 to 3 {
for %i7 = 0 to 3 {
store %cst_0, %arg1[%i6, %i7] : memref<10x10xf32>
}
}
for %i8 = 0 to 3 {
for %i9 = 0 to 3 {
%4 = load %0[%i8, %i9] : memref<10x10xf32>
%5 = load %arg1[%i8, %i9] : memref<10x10xf32>
%6 = addf %4, %5 : f32
store %6, %arg1[%i8, %i9] : memref<10x10xf32>
}
}
// Pass 1: should fuse single-use producer loop nests into their unique user,
// so '%i2' will fuse into '%i4' and '%i6' will fuse into '%i8'.
// Pass 2: should fuse sibling loop nests which share no dependence edges,
// so should fuse '%i4' into '%i8'.
// Pass 3: should fuse single-use producer loop nest '%i0' into '%i8'. Note
// that loop nest '%i0' now has a single user after Pass 2 fused its
// two users together).
// CHECK: for %i0 = 0 to 3 {
// CHECK-NEXT: for %i1 = 0 to 3 {
// CHECK-NEXT: %1 = affine.apply [[MAP0]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: %2 = affine.apply [[MAP1]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: store %cst_1, %0[%1, %2] : memref<1x1xf32>
// CHECK-NEXT: store %cst, %arg0[%i0, %i1] : memref<10x10xf32>
// CHECK-NEXT: %3 = affine.apply [[MAP0]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: %4 = affine.apply [[MAP1]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: %5 = load %0[%3, %4] : memref<1x1xf32>
// CHECK-NEXT: %6 = load %arg0[%i0, %i1] : memref<10x10xf32>
// CHECK-NEXT: %7 = mulf %5, %6 : f32
// CHECK-NEXT: store %7, %arg0[%i0, %i1] : memref<10x10xf32>
// CHECK-NEXT: store %cst_0, %arg1[%i0, %i1] : memref<10x10xf32>
// CHECK-NEXT: %8 = affine.apply [[MAP0]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: %9 = affine.apply [[MAP1]](%i0, %i1, %i0, %i1)
// CHECK-NEXT: %10 = load %0[%8, %9] : memref<1x1xf32>
// CHECK-NEXT: %11 = load %arg1[%i0, %i1] : memref<10x10xf32>
// CHECK-NEXT: %12 = addf %10, %11 : f32
// CHECK-NEXT: store %12, %arg1[%i0, %i1] : memref<10x10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}
// -----
// CHECK-DAG: [[MAP2:#map[0-9]+]] = (d0, d1, d2) -> (d1)
// CHECK-DAG: [[MAP3:#map[0-9]+]] = (d0, d1, d2) -> (-d0 + d2)
func @two_matrix_vector_products() {
%in_matrix = alloc() : memref<10x10xf32>
%in_vec0 = alloc() : memref<10xf32>
%in_vec1 = alloc() : memref<10xf32>
%out_vec0 = alloc() : memref<10xf32>
%out_vec1 = alloc() : memref<10xf32>
%cf7 = constant 7.0 : f32
// Populate input matrix.
for %i0 = 0 to 10 {
for %i1 = 0 to 10 {
store %cf7, %in_matrix[%i0, %i1] : memref<10x10xf32>
}
}
// out_vec0 = in_matrix x in_vec0
for %i2 = 0 to 10 {
for %i3 = 0 to 10 {
%v0 = load %in_matrix[%i2, %i3] : memref<10x10xf32>
%v1 = load %in_vec0[%i3] : memref<10xf32>
%v2 = mulf %v0, %v1 : f32
%v3 = load %out_vec0[%i3] : memref<10xf32>
%v4 = addf %v2, %v3 : f32
store %v4, %out_vec0[%i3] : memref<10xf32>
}
}
// out_vec1 = in_matrix x in_vec1
for %i4 = 0 to 10 {
for %i5 = 0 to 10 {
%v5 = load %in_matrix[%i4, %i5] : memref<10x10xf32>
%v6 = load %in_vec1[%i5] : memref<10xf32>
%v7 = mulf %v5, %v6 : f32
%v8 = load %out_vec1[%i5] : memref<10xf32>
%v9 = addf %v7, %v8 : f32
store %v9, %out_vec1[%i5] : memref<10xf32>
}
}
// CHECK: for %i0 = 0 to 10 {
// CHECK-NEXT: for %i1 = 0 to 10 {
// CHECK-NEXT: %5 = affine.apply [[MAP2]](%i0, %i1, %i0)
// CHECK-NEXT: %6 = affine.apply [[MAP3]](%i0, %i1, %i0)
// CHECK-NEXT: store %cst, %0[%5, %6] : memref<10x1xf32>
// CHECK-NEXT: }
// CHECK-NEXT: for %i2 = 0 to 10 {
// CHECK-NEXT: %7 = affine.apply [[MAP2]](%i0, %i2, %i0)
// CHECK-NEXT: %8 = affine.apply [[MAP3]](%i0, %i2, %i0)
// CHECK-NEXT: %9 = load %0[%7, %8] : memref<10x1xf32>
// CHECK-NEXT: %10 = load %1[%i0] : memref<10xf32>
// CHECK-NEXT: %11 = mulf %9, %10 : f32
// CHECK-NEXT: %12 = load %3[%i0] : memref<10xf32>
// CHECK-NEXT: %13 = addf %11, %12 : f32
// CHECK-NEXT: store %13, %3[%i0] : memref<10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: for %i3 = 0 to 10 {
// CHECK-NEXT: %14 = affine.apply [[MAP2]](%i0, %i3, %i0)
// CHECK-NEXT: %15 = affine.apply [[MAP3]](%i0, %i3, %i0)
// CHECK-NEXT: %16 = load %0[%14, %15] : memref<10x1xf32>
// CHECK-NEXT: %17 = load %2[%i0] : memref<10xf32>
// CHECK-NEXT: %18 = mulf %16, %17 : f32
// CHECK-NEXT: %19 = load %4[%i0] : memref<10xf32>
// CHECK-NEXT: %20 = addf %18, %19 : f32
// CHECK-NEXT: store %20, %4[%i0] : memref<10xf32>
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
return
Bug fix for getConstantBoundOnDimSize - this was detected when memref-bound-check was run on the output of the loop-fusion pass - the addition (to represent ceildiv as a floordiv) had to be performed only for the constant term of the constraint - update test cases - memref-bound-check no longer returns an error on the output of this test case PiperOrigin-RevId: 236731137
2019-03-05 06:58:59 +08:00
}
Adds loop attribute as a temporary work around to prevent slice fusion of loop nests containing instructions with side effects (the proper solution will be do use memref read/write regions in the future). PiperOrigin-RevId: 236733739
2019-03-05 07:14:12 +08:00
// -----
// CHECK-DAG: [[MAP3:#map[0-9]+]] = (d0, d1, d2) -> (-d0 + d1)
// CHECK-DAG: [[MAP4:#map[0-9]+]] = (d0, d1, d2) -> (d2)
func @should_not_slice_past_slice_barrier() {
%0 = alloc() : memref<100x16xf32>
for %i0 = 0 to 100 {
for %i1 = 0 to 16 {
%1 = "op1"() : () -> f32
store %1, %0[%i0, %i1] : memref<100x16xf32>
} {slice_fusion_barrier: true}
}
for %i2 = 0 to 100 {
for %i3 = 0 to 16 {
%2 = load %0[%i2, %i3] : memref<100x16xf32>
"op2"(%2) : (f32) -> ()
}
}
// The 'slice_fusion_barrier' attribute on '%i1' prevents slicing the
// iteration space of '%i1' and any enclosing loop nests.
// CHECK: for %i0 = 0 to 100 {
// CHECK-NEXT: for %i1 = 0 to 16 {
// CHECK-NEXT: %1 = "op1"() : () -> f32
// CHECK-NEXT: %2 = affine.apply [[MAP3]](%i0, %i0, %i1)
// CHECK-NEXT: %3 = affine.apply [[MAP4]](%i0, %i0, %i1)
// CHECK-NEXT: store %1, %0[%2, %3] : memref<1x16xf32>
// CHECK-NEXT: } {slice_fusion_barrier: true}
// CHECK-NEXT: for %i2 = 0 to 16 {
// CHECK-NEXT: %4 = affine.apply [[MAP3]](%i0, %i0, %i2)
// CHECK-NEXT: %5 = affine.apply [[MAP4]](%i0, %i0, %i2)
// CHECK-NEXT: %6 = load %0[%4, %5] : memref<1x16xf32>
// CHECK-NEXT: "op2"(%6) : (f32) -> ()
// CHECK-NEXT: }
// CHECK-NEXT: }
return
}
Fix and improve detectAsMod - fix for the mod detection - simplify/avoid the mod at construction (if the dividend is already known to be less than the divisor), since the information is available at hand there PiperOrigin-RevId: 236882988
2019-03-06 03:08:41 +08:00
// -----
#map0 = (d0, d1) -> (d0 * 16 + d1)
func @fuse_across_dim_mismatch(%arg0: memref<4x4x16x1xf32>, %arg1: memref<144x9xf32>, %arg2: memref<9xf32>) {
%1 = alloc() : memref<144x4xf32>
%2 = constant 0.0 : f32
for %i2 = 0 to 9 {
for %i3 = 0 to 4 {
for %i5 = 0 to 16 {
%7 = affine.apply #map0(%i2, %i5)
store %2, %1[%7, %i3] : memref<144x4xf32>
}
}
}
for %i6 = 0 to 9 {
for %i7 = 0 to 9 {
for %i8 = 0 to 4 {
for %i10 = 0 to 16 {
%10 = affine.apply #map0(%i6, %i10)
%11 = load %1[%10, %i8] : memref<144x4xf32>
}
}
}
}
return
}
// MAXIMAL: #map4 = (d0, d1) -> (d0 * 16 + d1)
// MAXIMAL-NEXT: #map5 = (d0, d1, d2, d3, d4) -> (d0 * -16 - d1 + d3)
// MAXIMAL-NEXT: #map6 = (d0, d1, d2, d3, d4) -> (-d2 + d4)
// MAXIMAL-LABEL: func @fuse_across_dim_mismatch
// MAXIMAL: %0 = alloc() : memref<1x1xf32>
// MAXIMAL: for %i0 = 0 to 9 {
// MAXIMAL-NEXT: for %i1 = 0 to 9 {
// MAXIMAL-NEXT: for %i2 = 0 to 4 {
// MAXIMAL-NEXT: for %i3 = 0 to 16 {
// MAXIMAL-NEXT: %1 = affine.apply #map4(%i0, %i3)
// MAXIMAL-NEXT: %2 = affine.apply #map5(%i0, %i3, %i2, %1, %i2)
// MAXIMAL-NEXT: %3 = affine.apply #map6(%i0, %i3, %i2, %1, %i2)
// MAXIMAL-NEXT: store %cst, %0[%2, %3] : memref<1x1xf32>
// MAXIMAL-NEXT: %4 = affine.apply #map4(%i0, %i3)
// MAXIMAL-NEXT: %5 = affine.apply #map5(%i0, %i3, %i2, %4, %i2)
// MAXIMAL-NEXT: %6 = affine.apply #map6(%i0, %i3, %i2, %4, %i2)
// MAXIMAL-NEXT: %7 = load %0[%5, %6] : memref<1x1xf32>
// MAXIMAL-NEXT: }
// MAXIMAL-NEXT: }
// MAXIMAL-NEXT: }
// MAXIMAL-NEXT: }
// -----
#map3 = (d0, d1) -> ((d0 * 72 + d1) floordiv 2304)
#map4 = (d0, d1) -> (((d0 * 72 + d1) mod 2304) floordiv 1152)
#map5 = (d0, d1) -> (((((d0 * 72 + d1) mod 2304) mod 1152) floordiv 9) floordiv 8)
#map6 = (d0, d1) -> (((((d0 * 72 + d1) mod 2304) mod 1152) mod 9) floordiv 3)
#map7 = (d0, d1) -> (((((d0 * 72 + d1) mod 2304) mod 1152) mod 9) mod 3)
#map10 = (d0, d1) -> (d0 * 16 + d1)
#map11 = (d0, d1) -> (d0 * 16 + d1)
#map12 = (d0, d1) -> (d0 * 16 - d1 + 15)
func @fuse_across_varying_dims_complex() {
%c0 = constant 0 : index
%0 = alloc() : memref<2x2x3x3x16x1xf32>
%1 = alloc() : memref<64x9xf32>
%2 = alloc() : memref<144x4xf32>
for %i0 = 0 to 64 {
for %i1 = 0 to 9 {
%4 = affine.apply #map3(%i0, %i1)
%5 = affine.apply #map4(%i0, %i1)
%6 = affine.apply #map5(%i0, %i1)
%7 = affine.apply #map6(%i0, %i1)
%8 = affine.apply #map7(%i0, %i1)
%9 = load %0[%4, %5, %7, %8, %6, %c0] : memref<2x2x3x3x16x1xf32>
store %9, %1[%i0, %i1] : memref<64x9xf32>
}
}
for %i2 = 0 to 9 {
for %i3 = 0 to 4 {
for %i4 = 0 to 16 {
%10 = affine.apply #map10(%i3, %i4)
%11 = load %1[%10, %i2] : memref<64x9xf32>
}
for %i5 = 0 to 16 {
%13 = "bar"() : () -> f32
%14 = affine.apply #map11(%i2, %i5)
store %13, %2[%14, %i3] : memref<144x4xf32>
}
}
}
for %i6 = 0 to 9 {
for %i7 = 0 to 9 {
for %i8 = 0 to 4 {
for %i9 = 0 to 16 {
%15 = affine.apply #map12(%i8, %i9)
%16 = load %1[%15, %i7] : memref<64x9xf32>
}
}
}
}
return
}
// MAXIMAL: #map5 = (d0, d1) -> ((d0 * 72 + d1) floordiv 2304)
// MAXIMAL-NEXT: #map6 = (d0, d1) -> (((d0 * 72 + d1) mod 2304) floordiv 1152)
// MAXIMAL-NEXT: #map7 = (d0, d1) -> (((((d0 * 72 + d1) mod 2304) mod 1152) floordiv 9) floordiv 8)
// MAXIMAL-NEXT: #map8 = (d0, d1) -> (((((d0 * 72 + d1) mod 2304) mod 1152) mod 9) floordiv 3)
// MAXIMAL-NEXT: #map9 = (d0, d1) -> (((((d0 * 72 + d1) mod 2304) mod 1152) mod 9) mod 3)
// MAXIMAL-NEXT: #map10 = (d0, d1, d2) -> (d1)
// MAXIMAL-NEXT: #map11 = (d0, d1, d2) -> (-d0 + d2)
// MAXIMAL-NEXT: #map12 = (d0, d1) -> (d0 * 16 + d1)
// MAXIMAL-NEXT: #map13 = (d0, d1) -> (d0 * 16 - d1 + 15)
Add missing run command to fusion test cases - follow up to cl/236882988 PiperOrigin-RevId: 236947383
2019-03-06 08:54:34 +08:00
// MAXIMAL-LABEL: func @fuse_across_varying_dims_complex
Fix and improve detectAsMod - fix for the mod detection - simplify/avoid the mod at construction (if the dividend is already known to be less than the divisor), since the information is available at hand there PiperOrigin-RevId: 236882988
2019-03-06 03:08:41 +08:00
// MAXIMAL-NEXT: %0 = alloc() : memref<64x1xf32>
// MAXIMAL-NEXT: %c0 = constant 0 : index
// MAXIMAL-NEXT: %1 = alloc() : memref<2x2x3x3x16x1xf32>
// MAXIMAL-NEXT: %2 = alloc() : memref<144x4xf32>
// MAXIMAL-NEXT: for %i0 = 0 to 9 {
// MAXIMAL-NEXT: for %i1 = 0 to 9 {
// MAXIMAL-NEXT: for %i2 = 0 to 4 {
// MAXIMAL-NEXT: for %i3 = 0 to 16 {
// MAXIMAL-NEXT: for %i4 = 0 to 64 {
// MAXIMAL-NEXT: %3 = affine.apply #map5(%i4, %i0)
// MAXIMAL-NEXT: %4 = affine.apply #map6(%i4, %i0)
// MAXIMAL-NEXT: %5 = affine.apply #map7(%i4, %i0)
// MAXIMAL-NEXT: %6 = affine.apply #map8(%i4, %i0)
// MAXIMAL-NEXT: %7 = affine.apply #map9(%i4, %i0)
// MAXIMAL-NEXT: %8 = load %1[%3, %4, %6, %7, %5, %c0] : memref<2x2x3x3x16x1xf32>
// MAXIMAL-NEXT: %9 = affine.apply #map10(%i0, %i4, %i0)
// MAXIMAL-NEXT: %10 = affine.apply #map11(%i0, %i4, %i0)
// MAXIMAL-NEXT: store %8, %0[%9, %10] : memref<64x1xf32>
// MAXIMAL-NEXT: }
// MAXIMAL-NEXT: for %i5 = 0 to 4 {
// MAXIMAL-NEXT: for %i6 = 0 to 16 {
// MAXIMAL-NEXT: %11 = affine.apply #map12(%i5, %i6)
// MAXIMAL-NEXT: %12 = affine.apply #map10(%i0, %11, %i0)
// MAXIMAL-NEXT: %13 = affine.apply #map11(%i0, %11, %i0)
// MAXIMAL-NEXT: %14 = load %0[%12, %13] : memref<64x1xf32>
// MAXIMAL-NEXT: }
// MAXIMAL-NEXT: for %i7 = 0 to 16 {
// MAXIMAL-NEXT: %15 = "bar"() : () -> f32
// MAXIMAL-NEXT: %16 = affine.apply #map12(%i0, %i7)
// MAXIMAL-NEXT: store %15, %2[%16, %i5] : memref<144x4xf32>
// MAXIMAL-NEXT: }
// MAXIMAL-NEXT: }
// MAXIMAL-NEXT: %17 = affine.apply #map13(%i2, %i3)
// MAXIMAL-NEXT: %18 = affine.apply #map10(%i0, %17, %i0)
// MAXIMAL-NEXT: %19 = affine.apply #map11(%i0, %17, %i0)
// MAXIMAL-NEXT: %20 = load %0[%18, %19] : memref<64x1xf32>
// MAXIMAL-NEXT: }
// MAXIMAL-NEXT: }
// MAXIMAL-NEXT: }
// MAXIMAL-NEXT: }
Use FlatAffineConstraints::unionBoundingBox to perform slice bounds union for loop fusion pass (WIP). Adds utility to convert slice bounds to a FlatAffineConstraints representation. Adds utility to FlatAffineConstraints to promote loop IV symbol identifiers to dim identifiers. PiperOrigin-RevId: 236973261
2019-03-06 12:33:30 +08:00
// -----
// CHECK-DAG: [[MAP3:#map[0-9]+]] = (d0) -> (d0 - 10)
func @should_fuse_with_slice_union() {
%a = alloc() : memref<100xf32>
%c0 = constant 0 : index
%cf0 = constant 0.0 : f32
for %i0 = 0 to 100 {
store %cf0, %a[%i0]: memref<100xf32>
}
for %i1 = 10 to 20 {
%v0 = load %a[%i1]: memref<100xf32>
for %i2 = 15 to 25 {
%v1 = load %a[%i2]: memref<100xf32>
}
}
// The union of two slice bounds (calculated between the store and each of
// the loads) is computed and used in the fusion cost calculation, index
// remapping, and private memref size. The result is that the temporary
// memref is reduced from 100xf32 to 15xf32 and properly indexed by
// the fused loops based on the union calculation.
// CHECK: for %i0 = 10 to 20 {
// CHECK-NEXT: for %i1 = 10 to 25 {
// CHECK-NEXT: %1 = affine.apply [[MAP3]](%i1)
// CHECK-NEXT: store %cst, %0[%1] : memref<15xf32>
// CHECK-NEXT: }
// CHECK-NEXT: %2 = affine.apply [[MAP3]](%i0)
// CHECK-NEXT: %3 = load %0[%2] : memref<15xf32>
// CHECK-NEXT: for %i2 = 15 to 25 {
// CHECK-NEXT: %4 = affine.apply [[MAP3]](%i2)
// CHECK-NEXT: %5 = load %0[%4] : memref<15xf32>
// CHECK-NEXT: }
// CHECK-NEXT: }
// CHECK-NEXT: return
return
}