Rationale:
Allocating the temporary buffers for access pattern expansion on the stack
(using alloca) is a bit too agressive, since it easily runs out of stack space
for large enveloping tensor dimensions. This revision changes the dynamic
allocation of these buffers with explicit alloc/dealloc pairs.
Reviewed By: bixia, wrengr
Differential Revision: https://reviews.llvm.org/D123253
Support returning arbitrary tensors from functions. Even those that are
not equivalent. To that end, additional information is gathered during
the analysis phase. In particular, which function args are aliasing with
which return values.
Also fix bugs in the current implementation when returning equivalent
tensors. Various unit tests are added to ensure that we have better test
coverage.
Note: Returning non-equivalent tensors is only allowed when
allowReturnAllocs is enabled. This functionality is useful for unit
testing and compatibility with other bufferizations such as the sparse
compiler. This is also towards using ModuleBufferization as a
replacement for --func-bufferize.
Differential Revision: https://reviews.llvm.org/D119120
* Bufferize FuncOp bodies and boundaries in the same loop. This is in preparation of moving FuncOp bufferization into an external model implementation.
* As a side effect, stop bufferization earlier if there was an error. (Do not continue bufferization, fewer error messages.)
* Run equivalence analysis of CallOps before the main analysis. This is needed so that equialvence info is propagated properly.
Differential Revision: https://reviews.llvm.org/D123208
https://reviews.llvm.org/D122641 introduced fixes to the ExpandShapeOp verifier
but also introduced an artificial layout limitation that prevents the consideration of transposed layouts.
This revision fixes the omissions and reimplements the logic using saturated arithmetic which is more
idiomatic and avoids leaking internal implementation details.
Tests cases are added for transposed layouts.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D122845
ListOption currently uses llvm:🆑:list under the hood, but the usages
of ListOption are generally a tad different from llvm:🆑:list. This
commit codifies this by making ListOption implicitly comma separated,
and removes the explicit flag set for all of the current list options.
The new parsing for comma separation of ListOption also adds in support
for skipping over delimited sub-ranges (i.e. {}, [], (), "", ''). This
more easily supports nested options that use those as part of the
format, and this constraint (balanced delimiters) is already codified
in the syntax of pass pipelines.
See https://discourse.llvm.org/t/list-of-lists-pass-option/5950 for
related discussion
Differential Revision: https://reviews.llvm.org/D122879
Prior to this change there were a number of places where the allocation and deallocation of SparseTensorCOO objects were not cleanly paired, leading to inconsistencies regarding whether each function released its tensor/coo arguments or not, as well as making it easy to run afoul of memory leaks, use-after-free, or double-free errors. This change cleans up the codegen vs runtime boundary to resolve those issues. Now, the only time the runtime library frees an object is either (a) because it's a function explicitly designed to do so, or (b) because the allocated object is entirely local to the function and would be a memory leak if not released. Thus, now the codegen takes complete responsibility for releasing any objects it caused to be allocated.
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D122435
This reverts commit 59bbc7a085.
This exposes an issue breaking the contract of
`applyPatternsAndFoldGreedily` where we "converge" without applying
remaining patterns.
This avoids accidentally reversing the order of constants during successive
application, e.g. when running the canonicalizer. This helps reduce the number
of iterations, and also avoids unnecessary changes to input IR.
Fixes#51892
Differential Revision: https://reviews.llvm.org/D122692
Bubble up extract_slice above Linalg operation.
A sequence of operations
%0 = linalg.<op> ... arg0, arg1, ...
%1 = tensor.extract_slice %0 ...
can be replaced with
%0 = tensor.extract_slice %arg0
%1 = tensor.extract_slice %arg1
%2 = linalg.<op> ... %0, %1, ...
This results in the reduce computation of the linalg operation.
The implementation uses the tiling utility functions. One difference
from the tiling process is that we don't need to insert the checking
code for the out-of-bound accesses. The use of the slice itself
represents that the code writer is sure about the boundary condition.
To avoid adding the boundary condtion check code, `omitPartialTileCheck`
is introduced for the tiling utility functions.
Differential Revision: https://reviews.llvm.org/D122437
Infer a tighter MemRef type instead of always falling back to the most dynamic MemRef type. This is inefficient and caused op verification errors.
Differential Revision: https://reviews.llvm.org/D122649
* Complete rewrite of the verifier.
* CollapseShapeOp verifier will be updated in a subsequent commit.
* Update and expand op documentation.
* Add a new builder that infers the result type based on the source type, result shape and reassociation indices. In essence, only the result layout map is inferred.
Differential Revision: https://reviews.llvm.org/D122641
For example, we could do the following eliminations:
fold vector.shuffle V1, V2, [0, 1, 2, 3] : <4xi32>, <2xi32> -> V1
fold vector.shuffle V1, V2, [4, 5] : <4xi32>, <2xi32> -> V2
Differential Revision: https://reviews.llvm.org/D122706
linalg.generic can also take scalars instead of tensors, which
tensor.cast doesn't support. We don't have an easy way to cast between
scalars and tensors so just keep the linalg.generic in those cases.
Differential Revision: https://reviews.llvm.org/D122575
This revision supports padding only a subset of the iteration dimensions via an additional padding-dimensions parameter. This control allows us to pad an operation in multiple steps. For example, one may want to pad only the output dimensions of a producer matmul fused into a consumer loop nest, before tiling and padding its reduction dimension.
Depends On D122309
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D122560
Pass the padding options using arrays instead of lambdas. In particular pass the padding value as string and use the argument parser to create the padding value. Arrays are a more natural choice that matches the current use cases and avoids converting arrays to lambdas.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D122309
This patch adds the ReductionClauseInterface and also adds reduction
support for `omp.parallel` operation.
Reviewed By: kiranchandramohan
Differential Revision: https://reviews.llvm.org/D122402
This patch adds MLIR NVVM support for the various NVPTX `mma.sync`
operations. There are a number of possible data type, shape,
and other attribute combinations supported by the operation, so a
custom assebmly format is added and attributes are inferred where
possible.
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D122410
Use "enable-vla-vectorization=vla" to generate a vector length agnostic
loops during vectorization. This option works for vectorization strategy 2.
Differential Revision: https://reviews.llvm.org/D118379
The way vector.create_mask is currently lowered is
vector-length-dependent, and therefore incompatible with scalable vector
types. This patch adds an alternative lowering path for create_mask
operations that return a scalable vector mask.
Differential Revision: https://reviews.llvm.org/D118248
This transformation allow to break up a reduction dimension in a
parallel and a reduction dimension. This is followed by a separate
reduction op. This allows to generate tree reduction which is beneficial
on target allowing to take advantage parallelism.
Differential Revision: https://reviews.llvm.org/D122045
Previously, only LinalgOps whose operands are defined by an ExtractSliceOp could be padded. The revision supports walking a use-def chain of LinalgOps to find an ExtractSliceOp.
Reviewed By: hanchung
Differential Revision: https://reviews.llvm.org/D122116
This revision introduces a heuristic to stop fusion for shape-only tensors. A shape-only tensor only defines the shape of the consumer computation while the data is not used. Pure producer consumer fusion thus shall not fuse the producer of a shape-only tensor. In particular, since the shape-only tensor will have other uses that actually consume the data.
The revision enables fusion for consumers that have two uses of the same tensor. One as input operand and one as shape-only output operand. In these cases, we want to fuse only the input operand and avoid output fusion via iteration argument.
Reviewed By: hanchung
Differential Revision: https://reviews.llvm.org/D120981
Create the AffineMinOp used to compute the padding width in canonical form and update the tests.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D122311
These attributes were added because of oilist required them earlier. It
no longer requires them and so these attributes can be safely removed
from the operations.
Reviewed By: kiranchandramohan
Differential Revision: https://reviews.llvm.org/D122289
This patch adds omp.single according to Section 2.8.2 of OpenMP 5.0.
Also added tests for the same.
Reviewed By: peixin
Differential Revision: https://reviews.llvm.org/D122288
Co-authored-by: Kiran Kumar T P <kirankumar.tp@amd.com>
This patch
- adds assembly format for `omp.wsloop` operation
- removes the `parseClauses` clauses as it is not required anymore
This is expected to be the final patch in a series of patches for replacing
parsers for clauses with `oilist`.
Reviewed By: Mogball
Differential Revision: https://reviews.llvm.org/D121367
Make MaxSI, MaxUI, MinSI and MinUI commutative, so they will be canonicalized to have its constants appear as the second operand. And the constant folder will match more cases.
Differential Revision: https://reviews.llvm.org/D122225
This is work towards: https://github.com/llvm/llvm-project/issues/51652
This differential doesn't yet make use of the new kSparseToSparse, just introduces it. The differential that finally makes use of them is D122061, which is the final differential in the chain that fixes bug 51652.
Depends On D122054
Reviewed By: aartbik
Differential Revision: https://reviews.llvm.org/D122055
The revision introduces a affine.min and affine.max canonicalization pattern that orders the result expressions. It flattens the result expressions to arrays of dimension and symbol coefficients plus one constant coefficient and rearranges them in lexicographic order.
Without the pattern, CSE will not eliminate two affine.min / affine.max operation if the results are ordered differently. For example, the operations
```
%1 = affine.min affine_map<(d0) -> (8, -d0 + 27)>(%arg4)
%2 = affine.min affine_map<(d0) -> (-d0 + 27, 8)>(%arg4)
```
doe not CSE. After applying the pattern, the two operations are equivalent
```
%1 = affine.min affine_map<(d0) -> (8, -d0 + 27)>(%arg4)
%2 = affine.min affine_map<(d0) -> (8, -d0 + 27)>(%arg4)
```
which enables CSE.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D121819
Computing dropped unit-dims when all the unit dims are dropped, does
not need to check for strides being dropped.
This also enables canonicalization of reduced-rank subviews.
Reviewed By: gysit
Differential Revision: https://reviews.llvm.org/D121766
The current nested if merging has a bug. Specifically, consider the following code:
```
%r = scf.if %arg3 -> (i32) {
scf.if %arg1 {
"test.op"() : () -> ()
}
scf.yield %arg0 : i32
} else {
scf.yield %arg2 : i32
}
```
When the above gets merged, it will become:
```
%r = scf.if %arg3 && %arg1-> (i32) {
"test.op"() : () -> ()
scf.yield %arg0 : i32
} else {
scf.yield %arg2 : i32
}
```
However, this means that when only %arg3 is true, we will incorrectly return %arg2 instead
of %arg0. This change updates the behavior of the pass to only enable nested if merging where
the outer yield contains only values from the inner if, or values defined outside of the if.
In the case of the latter, they can turned into a select of only the outer if condition, thus
maintaining correctness.
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D122108
This support has never really worked well, and is incredibly clunky to
use (it effectively creates two argument APIs), and clunky to generate (it isn't
clear how we should actually expose this from PDL frontends). Treating these
as just attribute arguments is much much cleaner in every aspect of the stack.
If we need to optimize lots of constant parameters, it would be better to
investigate internal representation optimizations (e.g. batch attribute creation),
that do not affect the user (we want a clean external API).
Differential Revision: https://reviews.llvm.org/D121569
This patch slightly updates the behavior of scf.if->select to
place any hoisted select statements prior to the remaining scf.if body.
This allows better composition with other canonicalization passes, such as
scf.if nested merging.
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D122027
Aart Bik