Commit Graph

7466 Commits

Author SHA1 Message Date
thomasraoux b90b66bcbe [mlir] Missed clang-format 2021-05-07 13:57:34 -07:00
thomasraoux d0453a8933 [mlir][vector] Extend pattern to trim lead unit dimension to Splat Op
Differential Revision: https://reviews.llvm.org/D102091
2021-05-07 13:54:41 -07:00
Alexander Belyaev 3444996b4c [mlir] Add a pattern to bufferize std.index_cast.
Differential Revision: https://reviews.llvm.org/D102088
2021-05-07 21:32:02 +02:00
Alexander Belyaev a3f22d020b [mlir] Add a pattern to bufferize linalg.tensor_reshape.
Differential Revision: https://reviews.llvm.org/D102089
2021-05-07 21:31:17 +02:00
Emilio Cota 21db1e3b01 [mlir][docs] remove stale statement about index type in vectors
b614ada0e8 ("[mlir] add support for index type in vectors.") removed
this limitation.

Differential Revision: https://reviews.llvm.org/D102081
2021-05-07 19:25:17 +00:00
thomasraoux a970e69d6b [mlir][vector] add pattern to cast away leading unit dim for elementwise op
Differential Revision: https://reviews.llvm.org/D102034
2021-05-07 07:54:09 -07:00
thomasraoux 565ee6afc7 [mlir][spirv] add support lowering of extract_slice to scalar type
Differential Revision: https://reviews.llvm.org/D102041
2021-05-07 07:52:02 -07:00
Tobias Gysi f31531a30b [mlir][linalg] Remove redundant indexOp builder.
Remove the builder signature taking a signed dimension identifier.

Reviewed By: ergawy

Differential Revision: https://reviews.llvm.org/D102055
2021-05-07 14:22:12 +00:00
Tres Popp faab8c140a [mlir] Rename BufferAliasAnalysis to BufferViewFlowAnalysis
This it to make more clear the difference between this and
an AliasAnalysis.

For example, given a sequence of subviews that create values
A -> B -> C -> d:
BufferViewFlowAnalysis::resolve(B) => {B, C, D}
AliasAnalysis::resolve(B) => {A, B, C, D}

Differential Revision: https://reviews.llvm.org/D100838
2021-05-07 16:12:54 +02:00
KareemErgawy-TomTom e4dee7e730 [MLIR][SPIRV] Properly (de-)serialize BranchConditionalOp.
Implements proper (de-)serialization logic for BranchConditionalOp when
such ops have true/false target operands.

Reviewed By: antiagainst

Differential Revision: https://reviews.llvm.org/D101602
2021-05-07 09:00:50 +02:00
Tobias Gysi 26e916334e [mlir][linalg] Add IndexedGenericOp to GenericOp canonicalization.
Replace all `linalg.indexed_generic` ops by `linalg.generic` ops that access the iteration indices using the `linalg.index` op.

Differential Revision: https://reviews.llvm.org/D101612
2021-05-07 06:00:16 +00:00
MaheshRavishankar 05a89312d8 [mlir][Linalg] Allow folding to rank-zero tensor when using rank-reducing subtensors.
The pattern to convert subtensor ops to their rank-reduced versions
(by dropping unit-dims in the result) can also convert to a zero-rank
tensor. Handle that case.
This also fixes a OOB access bug in the existing pattern for such
cases.

Differential Revision: https://reviews.llvm.org/D101949
2021-05-06 19:03:55 -07:00
Rob Suderman d3e987c389 [mlir][tosa] Added div op, variadic concat. Removed placeholder. Spec v0.22 alignment.
Nearly complete alignment to spec v0.22
- Adds Div op
- Concat inputs now variadic
- Removes Placeholder op

Note: TF side PR https://github.com/tensorflow/tensorflow/pull/48921 deletes Concat legalizations to avoid breaking TensorFlow CI. This must be merged only after the TF PR has merged.

Reviewed By: rsuderman

Differential Revision: https://reviews.llvm.org/D101958
2021-05-06 15:55:58 -07:00
Amy Zhuang 5dc1ed3f62 [mlir] Update dstNode after DenseMap insertion in loop fusion pass.
Reviewed By: vinayaka-polymage

Differential Revision: https://reviews.llvm.org/D101794
2021-05-06 15:23:59 -07:00
Lei Zhang 41bc54cc56 [mlir][spirv] NFC: Replace OwningSPIRVModuleRef with OwningOpRef
Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D102009
2021-05-06 17:17:44 -04:00
River Riddle 6304c0836a [mlir] Store the flag for dynamic operand storage in the low bits
It is currently stored in the high bits, which is disallowed on certain
platforms (e.g. android). This revision switches the representation to use
the low bits instead, fixing crashes/breakages on those platforms.

Differential Revision: https://reviews.llvm.org/D101969
2021-05-06 12:45:35 -07:00
thomasraoux 71eb32d97e [mlir][vector] Fix typo 2021-05-06 10:12:31 -07:00
thomasraoux 52525cb20f [mlir][linalg][NFC] Make reshape folding control more fine grain
This expose a lambda control instead of just a boolean to control unit
dimension folding.
This however gives more control to user to pick a good heuristic.
Folding reshapes helps fusion opportunities but may generate sub-optimal
generic ops.

Differential Revision: https://reviews.llvm.org/D101917
2021-05-06 10:11:39 -07:00
Denys Shabalin 1f109f9d9c Fix array attribute in bindings for linalg.init_tensor
Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D101998
2021-05-06 18:25:59 +02:00
thomasraoux 933551eaeb [mlir][NFC] Fix warning in VectorTransforms.cpp 2021-05-06 08:11:42 -07:00
thomasraoux 0b303da6f8 [mlir][vector] add pattern to cast away lead unit dimension for broadcast op
Differential Revision: https://reviews.llvm.org/D101955
2021-05-06 08:02:17 -07:00
Christian Sigg a0d019fc89 [mlir] Add support for ops with regions in 'gpu-async-region' rewriter.
Reviewed By: herhut

Differential Revision: https://reviews.llvm.org/D101757
2021-05-06 13:21:28 +02:00
Navdeep Kumar 875eb523c1 [MLIR][GPU][NVVM] Add warp synchronous matrix-multiply accumulate ops
Add warp synchronous matrix-multiply accumulate ops in GPU and NVVM
dialect. Add following three ops to GPU dialect :-
  1.) subgroup_mma_load_matrix
  2.) subgroup_mma_store_matrix
  3.) subgroup_mma_compute
Add following three ops to NVVM dialect :-
  1.) wmma.m16n16k16.load.[a,b,c].[f16,f32].row.stride
  2.) wmma.m16n16k16.store.d.[f16,f32].row.stride
  3.) wmma.m16n16k16.mma.row.row.[f16,f32].[f16,f32]

Reviewed By: bondhugula, ftynse, ThomasRaoux

Differential Revision: https://reviews.llvm.org/D95330
2021-05-06 12:06:25 +05:30
Emilio Cota 3c952ab25f [mlir] Check generated IR of math_polynomial_approx.mlir
Instead of just checking that we emit something.

Differential Revision: https://reviews.llvm.org/D101940
2021-05-05 16:42:48 -07:00
MaheshRavishankar 4b2d7ef3ea [mlir][Linalg] Fix test to use new reshape op form.
Differential Revision: https://reviews.llvm.org/D101956
2021-05-05 16:06:58 -07:00
MaheshRavishankar b6060b7673 [mlir][Linalg] Fix element type of results when folding reshapes.
Fixing a minor bug which lead to element type of the output being
modified when folding reshapes with generic op.

Differential Revision: https://reviews.llvm.org/D101942
2021-05-05 15:40:41 -07:00
Emilio Cota 0edc4bc84a [mlir] Add polynomial approximation for math::ExpM1
This approximation matches the one in Eigen.

```
name                      old cpu/op  new cpu/op  delta
BM_mlir_Expm1_f32/10      90.9ns ± 4%  52.2ns ± 4%  -42.60%    (p=0.000 n=74+87)
BM_mlir_Expm1_f32/100      837ns ± 3%   231ns ± 4%  -72.43%    (p=0.000 n=79+69)
BM_mlir_Expm1_f32/1k      8.43µs ± 3%  1.58µs ± 5%  -81.30%    (p=0.000 n=77+83)
BM_mlir_Expm1_f32/10k     83.8µs ± 3%  15.4µs ± 5%  -81.65%    (p=0.000 n=83+69)
BM_eigen_s_Expm1_f32/10   68.8ns ±17%  72.5ns ±14%   +5.40%  (p=0.000 n=118+115)
BM_eigen_s_Expm1_f32/100   694ns ±11%   717ns ± 2%   +3.34%   (p=0.000 n=120+75)
BM_eigen_s_Expm1_f32/1k   7.69µs ± 2%  7.97µs ±11%   +3.56%   (p=0.000 n=95+117)
BM_eigen_s_Expm1_f32/10k  88.0µs ± 1%  89.3µs ± 6%   +1.45%   (p=0.000 n=74+106)
BM_eigen_v_Expm1_f32/10   44.3ns ± 6%  45.0ns ± 8%   +1.45%   (p=0.018 n=81+111)
BM_eigen_v_Expm1_f32/100   351ns ± 1%   360ns ± 9%   +2.58%    (p=0.000 n=73+99)
BM_eigen_v_Expm1_f32/1k   3.31µs ± 1%  3.42µs ± 9%   +3.37%   (p=0.000 n=71+100)
BM_eigen_v_Expm1_f32/10k  33.7µs ± 8%  34.1µs ± 9%   +1.04%    (p=0.007 n=99+98)
```

Reviewed By: ezhulenev

Differential Revision: https://reviews.llvm.org/D101852
2021-05-05 14:31:34 -07:00
Rob Suderman 7abb56c78b [mlir][tosa] Add tosa.depthwise lowering to existing linalg.depthwise_conv
Implements support for undialated depthwise convolution using the existing
depthwise convolution operation. Once convolutions migrate to yaml defined
versions we can rewrite for cleaner implementation.

Reviewed By: mravishankar

Differential Revision: https://reviews.llvm.org/D101579
2021-05-05 13:30:05 -07:00
Philipp Krones 632ebc4ab4 [MC] Untangle MCContext and MCObjectFileInfo
This untangles the MCContext and the MCObjectFileInfo. There is a circular
dependency between MCContext and MCObjectFileInfo. Currently this dependency
also exists during construction: You can't contruct a MOFI without a MCContext
without constructing the MCContext with a dummy version of that MOFI first.
This removes this dependency during construction. In a perfect world,
MCObjectFileInfo wouldn't depend on MCContext at all, but only be stored in the
MCContext, like other MC information. This is future work.

This also shifts/adds more information to the MCContext making it more
available to the different targets. Namely:

- TargetTriple
- ObjectFileType
- SubtargetInfo

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D101462
2021-05-05 10:03:02 -07:00
Javier Setoain 95861216ac [mlir][ArmSVE] Add masked arithmetic operations
These instructions map to SVE-specific instrinsics that accept a
predicate operand to support control flow in vector code.

Differential Revision: https://reviews.llvm.org/D100982
2021-05-05 17:41:58 +01:00
Sergei Grechanik d80b04ab00 [mlir][Affine][Vector] Support vectorizing reduction loops
This patch adds support for vectorizing loops with 'iter_args'
implementing known reductions along the vector dimension. Comparing to
the non-vector-dimension case, two additional things are done during
vectorization of such loops:
- The resulting vector returned from the loop is reduced to a scalar
  using `vector.reduce`.
- In some cases a mask is applied to the vector yielded at the end of
  the loop to prevent garbage values from being written to the
  accumulator.

Vectorization of reduction loops is disabled by default. To enable it, a
map from loops to array of reduction descriptors should be explicitly passed to
`vectorizeAffineLoops`, or `vectorize-reductions=true` should be passed
to the SuperVectorize pass.

Current limitations:
- Loops with a non-unit step size are not supported.
- n-D vectorization with n > 1 is not supported.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D100694
2021-05-05 09:03:59 -07:00
Tobias Gysi 4a6ee23d83 [mlir][linalg] Fix bug in the fusion on tensors index op handling.
The old index op handling let the new index operations point back to the
producer block. As a result, after fusion some index operations in the
fused block had back references to the old producer block resulting in
illegal IR. The patch now relies on a block and value mapping to avoid
such back references.

Differential Revision: https://reviews.llvm.org/D101887
2021-05-05 14:46:08 +00:00
Uday Bondhugula 62851ea7ea [MLIR] Rename free function `verify` on OffsetSizeAndStrideOpInterface
Using a free function verify(<Op>) is error prone. Rename it.

Differential Revision: https://reviews.llvm.org/D101886
2021-05-05 17:44:15 +05:30
Alexander Belyaev 2865d114f9 [mlir] Use ReassociationIndices instead of affine maps in linalg.reshape.
Differential Revision: https://reviews.llvm.org/D101861
2021-05-05 12:59:57 +02:00
Javier Setoain 001d601ac4 [mlir][ArmSVE] Add basic arithmetic operations
While we figure out how to best add Standard support for scalable
vectors, these instructions provide a workaround for basic arithmetic
between scalable vectors.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D100837
2021-05-05 09:50:18 +02:00
William S. Moses f4a2dbfe29 [MLIR][SCF] Combine adjacent scf.if with same condition
Differential Revision: https://reviews.llvm.org/D101798
2021-05-05 00:39:58 -04:00
Aart Bik a2c9d4bb04 [mlir][sparse] Introduce proper sparsification passes
This revision migrates more code from Linalg into the new permanent home of
SparseTensor. It replaces the test passes with proper compiler passes.

NOTE: the actual removal of the last glue and clutter in Linalg will follow

Reviewed By: bixia

Differential Revision: https://reviews.llvm.org/D101811
2021-05-04 17:10:09 -07:00
River Riddle c1c1df6347 [mlir] Fix region successor bug in forward dataflow analysis
We weren't properly visiting region successors when the terminator wasn't return like, which could create incorrect results in the analysis. This revision ensures that we properly visit region successors, to avoid optimistically assuming a value is constant when it isn't.

Differential Revision: https://reviews.llvm.org/D101783
2021-05-04 14:50:37 -07:00
Rob Suderman 1f7adf8cb1 [mlir][tosa] Fix tosa.concat by inserting linalg.fill after linalg.init
All linalg.init operations must be fed into a linalg operation before
subtensor. The inserted linalg.fill guarantees it executes correctly.

Reviewed By: mravishankar

Differential Revision: https://reviews.llvm.org/D101848
2021-05-04 14:26:28 -07:00
William S. Moses cb395b84b0 [MLIR] Add not icmp canonicalization documentation
See: https://reviews.llvm.org/D101710
2021-05-04 11:44:25 -04:00
William S. Moses 8e211bf1c8 [MLIR][SCF] Assume uses of condition in the body of scf.while is true
Differential Revision: https://reviews.llvm.org/D101801
2021-05-04 11:39:07 -04:00
William S. Moses 93297e4bac [MLIR] Replace a not of a comparison with appropriate comparison
Differential Revision: https://reviews.llvm.org/D101710
2021-05-04 11:23:29 -04:00
Tobias Gysi 05d2297b86 [mlir][linalg] Always lower index operations during loop lowering.
Ensure the index operations are lowered on all linalg loop lowering paths.

Differential Revision: https://reviews.llvm.org/D101827
2021-05-04 14:30:59 +00:00
Adrian Kuegel 93537fabce [mlir] Add lowering from math.expm1 to LLVM.
Differential Revision: https://reviews.llvm.org/D96776
2021-05-04 14:22:10 +02:00
Matthias Springer aa58281979 [mlir] Fix bug in TransferOpReduceRank when all dims are broadcasts
TransferReadOps that are a scalar read + broadcast are handled by TransferReadToVectorLoadLowering.

Differential Revision: https://reviews.llvm.org/D101808
2021-05-04 11:21:44 +09:00
natashaknk 07ce5c99d7 [mlir][tosa] Add lowerings for tosa.equal and tosa.arithmetic_right_shift
Lowerings equal and arithmetic_right_shift for elementwise ops to linalg dialect using linalg.generic

Reviewed By: rsuderman

Differential Revision: https://reviews.llvm.org/D101804
2021-05-03 18:26:49 -07:00
Eugene Zhulenev 9b67096fe9 [mlir] Linalg: add vector transfer lowering patterns to the contraction lowering
This fixes a performance regression in vec-mat vectorization

Reviewed By: asaadaldien

Differential Revision: https://reviews.llvm.org/D101795
2021-05-03 16:21:51 -07:00
Emilio Cota 1c0374e770 [mlir] Add polynomial approximation for math::Log1p
This approximation matches the one in Eigen.

```
name                      old cpu/op  new cpu/op  delta
BM_mlir_Log1p_f32/10      83.2ns ± 7%  34.8ns ± 5%  -58.19%    (p=0.000 n=84+71)
BM_mlir_Log1p_f32/100      664ns ± 4%   129ns ± 4%  -80.57%    (p=0.000 n=82+82)
BM_mlir_Log1p_f32/1k      6.75µs ± 4%  0.81µs ± 3%  -88.07%    (p=0.000 n=88+79)
BM_mlir_Log1p_f32/10k     76.5µs ± 3%   7.8µs ± 4%  -89.84%    (p=0.000 n=80+80)
BM_eigen_s_Log1p_f32/10   70.1ns ±14%  72.6ns ±14%   +3.49%  (p=0.000 n=116+112)
BM_eigen_s_Log1p_f32/100   706ns ± 9%   717ns ± 3%   +1.60%   (p=0.018 n=117+80)
BM_eigen_s_Log1p_f32/1k   8.26µs ± 1%  8.26µs ± 1%     ~       (p=0.567 n=84+86)
BM_eigen_s_Log1p_f32/10k  92.1µs ± 5%  92.6µs ± 6%   +0.60%  (p=0.047 n=115+115)
BM_eigen_v_Log1p_f32/10   31.8ns ±24%  34.9ns ±17%   +9.72%    (p=0.000 n=98+96)
BM_eigen_v_Log1p_f32/100   169ns ±10%   177ns ± 5%   +4.66%   (p=0.000 n=119+81)
BM_eigen_v_Log1p_f32/1k   1.42µs ± 4%  1.46µs ± 8%   +2.70%   (p=0.000 n=93+113)
BM_eigen_v_Log1p_f32/10k  14.4µs ± 5%  14.9µs ± 8%   +3.61%  (p=0.000 n=115+110)
```

Reviewed By: ezhulenev, ftynse

Differential Revision: https://reviews.llvm.org/D101765
2021-05-03 15:11:37 -07:00
MaheshRavishankar a6e09391bb [mlir][Linalg] Add a utility method to get reassociations maps for reshape.
Given the source and destination shapes, if they are static, or if the
expanded/collapsed dimensions are unit-extent, it is possible to
compute the reassociation maps that can be used to reshape one type
into another. Add a utility method to return the reassociation maps
when possible.

This utility function can be used to fuse a sequence of reshape ops,
given the type of the source of the producer and the final result
type. This pattern supercedes a more constrained folding pattern added
to DropUnitDims pass.

Differential Revision: https://reviews.llvm.org/D101343
2021-05-03 14:40:15 -07:00
Aart Bik 90d18e106b [mlir][sparse] fixed typo: sparse -> sparse_tensor
Test passes either way, but this is full name of dialect

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D101774
2021-05-03 14:19:09 -07:00