Commit Graph

3006 Commits

Author SHA1 Message Date
Nicolas Vasilache 35b65be041 [mlir][Vector] Add transformation + pattern to split vector.transfer_read into full and partial copies.
This revision adds a transformation and a pattern that rewrites a "maybe masked" `vector.transfer_read %view[...], %pad `into a pattern resembling:

```
   %1:3 = scf.if (%inBounds) {
      scf.yield %view : memref<A...>, index, index
    } else {
      %2 = vector.transfer_read %view[...], %pad : memref<A...>, vector<...>
      %3 = vector.type_cast %extra_alloc : memref<...> to
      memref<vector<...>> store %2, %3[] : memref<vector<...>> %4 =
      memref_cast %extra_alloc: memref<B...> to memref<A...> scf.yield %4 :
      memref<A...>, index, index
   }
   %res= vector.transfer_read %1#0[%1#1, %1#2] {masked = [false ... false]}
```
where `extra_alloc` is a top of the function alloca'ed buffer of one vector.

This rewrite makes it possible to realize the "always full tile" abstraction where vector.transfer_read operations are guaranteed to read from a padded full buffer.
The extra work only occurs on the boundary tiles.

Differential Revision: https://reviews.llvm.org/D84631
2020-08-03 04:53:43 -04:00
Jacques Pienaar 86a78546b9 [mlir] Add shape.with_shape op
This is an operation that can returns a new ValueShape with a different shape. Useful for composing shape function calls and reusing existing shape transfer functions.

Just adding the op in this change.

Differential Revision: https://reviews.llvm.org/D84217
2020-07-31 14:46:48 -07:00
River Riddle 2a6c8b2e95 [mlir][PassIncGen] Refactor how pass registration is generated
The current output is a bit clunky and requires including files+macros everywhere, or manually wrapping the file inclusion in a registration function. This revision refactors the pass backend to automatically generate `registerFooPass`/`registerFooPasses` functions that wrap the pass registration. `gen-pass-decls` now takes a `-name` input that specifies a tag name for the group of passes that are being generated. For each pass, the generator now produces a `registerFooPass` where `Foo` is the name of the definition specified in tablegen. It also generates a `registerGroupPasses`, where `Group` is the tag provided via the `-name` input parameter, that registers all of the passes present.

Differential Revision: https://reviews.llvm.org/D84983
2020-07-31 13:20:37 -07:00
Rahul Joshi eb8c72cb0d [MLIR][NFC] Add FuncOp::getArgumentTypes()
Differential Revision: https://reviews.llvm.org/D85038
2020-07-31 13:18:03 -07:00
Thomas Raoux cfb955ac37 [mlir][spirv] Relax restriction on pointer type for CooperativeMatrix load/store
This change allow CooperativeMatrix Load/Store operations to use pointer type
that may not match the matrix element type. This allow us to declare buffer
with a larger type size than the matrix element type. This follows SPIR-V spec
and this is needed to be able to use cooperative matrix in combination with
shared local memory efficiently.

Differential Revision: https://reviews.llvm.org/D84993
2020-07-31 08:02:21 -07:00
Frederik Gossen 6983cf3a57 [MLIR][Shape] Allow unsafe `shape.broadcast`
In a context in which `shape.broadcast` is known not to produce an error value,
we want it to operate solely on extent tensors. The operation's behavior is
then undefined in the error case as the result type cannot hold this value.

Differential Revision: https://reviews.llvm.org/D84933
2020-07-31 14:18:06 +00:00
Jakub Lichman eef1bfb2d2 [mlir][Linalg] Conv {1,2,3}D ops defined with TC syntax
Replaced definition of named ND ConvOps with tensor comprehension
syntax which reduces boilerplate code significantly. Furthermore,
new ops to support TF convolutions added (without strides and dilations).

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D84628
2020-07-31 13:20:17 +02:00
Alexander Belyaev 4d6eec8e70 [mlir] Add TFFramework dialect to DialectSymbolRegistry.
Differential Revision: https://reviews.llvm.org/D84918
2020-07-31 11:00:54 +02:00
Rahul Joshi a34a8d5260 [MLIR][NFC] Add SymbolUse::UseRange::empty()
Differential Revision: https://reviews.llvm.org/D84984
2020-07-30 15:18:58 -07:00
Alexander Belyaev 6b8c641d8e [mlir] NFC: Expose `getElementPtrType` and `getSizes` methods of AllocOpLowering.
Differential Revision: https://reviews.llvm.org/D84917
2020-07-30 20:18:29 +02:00
Christian Sigg 13a3d88666 [MLIR] Don't pass separate LowerToLLVMOptions when we already pass a LLVMTypeConverter which contains those options already.
This also prevents passing inconsistent options.

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D84915
2020-07-30 14:55:23 +02:00
Abhishek Varma 76d07503f0 [MLIR] Introduce inter-procedural memref layout normalization
-- Introduces a pass that normalizes the affine layout maps to the identity layout map both within and across functions by rewriting function arguments and call operands where necessary.
-- Memref normalization is now implemented entirely in the module pass '-normalize-memrefs' and the limited intra-procedural version has been removed from '-simplify-affine-structures'.
-- Run using -normalize-memrefs.
-- Return ops are not handled and would be handled in the subsequent revisions.

Signed-off-by: Abhishek Varma <abhishek.varma@polymagelabs.com>

Differential Revision: https://reviews.llvm.org/D84490
2020-07-30 18:12:56 +05:30
George Mitenkov 1880532036 [MLIR][SPIRVToLLVM] Conversion of GLSL ops to LLVM intrinsics
This patch introduces new intrinsics in LLVM dialect:
-  `llvm.intr.floor`
-  `llvm.intr.maxnum`
-  `llvm.intr.minnum`
-  `llvm.intr.smax`
-  `llvm.intr.smin`
These intrinsics correspond to SPIR-V ops from GLSL
extended instruction set (`spv.GLSL.Floor`, `spv.GLSL.FMax`,
`spv.GLSL.FMin`,  `spv.GLSL.SMax` and `spv.GLSL.SMin`
respectively). Also conversion patterns for them were added.

Reviewed By: antiagainst

Differential Revision: https://reviews.llvm.org/D84661
2020-07-30 11:22:44 +03:00
Tres Popp f05308a277 [MLIR][NFC] Move Shape::WitnessType Declaration.
This moves it from ShapeOps.td to ShapeBase.td

Differential Revision: https://reviews.llvm.org/D84845
2020-07-29 20:00:28 +02:00
Jakub Lichman 1aaf8aa53d [mlir][Linalg] Conv1D, Conv2D and Conv3D added as named ops
This commit is part of a greater project which aims to add
full end-to-end support for convolutions inside mlir. The
reason behind having conv ops for each rank rather than
having one generic ConvOp is to enable better optimizations
for every N-D case which reflects memory layout of input/kernel
buffers better and simplifies code as well. We expect plain linalg.conv
to be progressively retired.

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D83879
2020-07-29 16:39:56 +02:00
Tres Popp ad793ed903 Forward extent tensors through shape.broadcast.
Differential Revision: https://reviews.llvm.org/D84832
2020-07-29 15:49:10 +02:00
Stephan Herhut 823ffef009 [mlir][Standard] Allow unranked memrefs as operands to dim and rank
`std.dim` currently only accepts ranked memrefs and `std.rank` is limited to
tensors.

Differential Revision: https://reviews.llvm.org/D84790
2020-07-29 14:42:58 +02:00
Alex Zinenko aec38c619d [mlir] LLVMType: make getUnderlyingType private
The current modeling of LLVM IR types in MLIR is based on the LLVMType class
that wraps a raw `llvm::Type *` and delegates uniquing, printing and parsing to
LLVM itself. This is model makes thread-safe type manipulation hard and is
being progressively replaced with a cleaner MLIR model that replicates the type
system. In the new model, LLVMType will no longer have an underlying LLVM IR
type. Restrict access to this type in the current model in preparation for the
change.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D84389
2020-07-29 13:43:38 +02:00
Stephan Herhut 5d9f33aaa0 [MLIR][Shape] Add conversion for missing ops to standard
This adds conversions for const_size and to_extent_tensor. Also, cast-like operations are now folded away if the source and target types are the same.

Differential Revision: https://reviews.llvm.org/D84745
2020-07-29 12:46:18 +02:00
Frederik Gossen 2e7baf6197 [MLIR][Shape] Allow `shape.add` to operate on indices
Differential Revision: https://reviews.llvm.org/D84441
2020-07-29 10:23:37 +00:00
Rahul Joshi 706d992ced [NFC] Add getArgumentTypes() to Region
- Add getArgumentTypes() to Region (missed from before)
- Adopt Region argument API in `hasMultiplyAddBody`
- Fix 2 typos in comments

Differential Revision: https://reviews.llvm.org/D84807
2020-07-28 18:27:42 -07:00
Anand Kodnani 834133c950 [MLIR] Vector store to load forwarding
The MemRefDataFlow pass does store to load forwarding
only for affine store/loads. This patch updates the pass
to use affine read/write interface which enables vector
forwarding.

Reviewed By: dcaballe, bondhugula, ftynse

Differential Revision: https://reviews.llvm.org/D84302
2020-07-28 11:30:54 -07:00
Nicolas Vasilache 64cdd5b3da [mlir][Vector] Drop declarative transforms
For the purpose of vector transforms, the Tablegen-based infra is subsumed by simple C++ pattern application. Deprecate declarative transforms whose complexity does not pay for itself.

Differential Revision: https://reviews.llvm.org/D84753
2020-07-28 13:11:16 -04:00
lorenzo chelini 946be75b9e [MLIR][Linalg] Retire C++ DotOp in favor of a linalg-ods-gen'd op
- replace DotOp, now that DRR rules have been dropped.

- Capture arguments mismatch in the parser. The number of parsed arguments must
  equal the number of expected arguments.

Reviewed By: ftynse, nicolasvasilache

Differential Revision: https://reviews.llvm.org/D82952
2020-07-28 12:34:19 +02:00
MaheshRavishankar fbe911ee75 [mlir][AffineToStandard] Make LowerAffine pass Op-agnostic.
The LowerAffine psas was a FunctionPass only for legacy
reasons. Making this Op-agnostic allows it to be used from command
line when affine expressions are within operations other than
`std.func`.

Differential Revision: https://reviews.llvm.org/D84590
2020-07-27 12:14:17 -07:00
Alex Zinenko a51829913d [mlir] Support for mutable types
Introduce support for mutable storage in the StorageUniquer infrastructure.
This makes MLIR have key-value storage instead of just uniqued key storage. A
storage instance now contains a unique immutable key and a mutable value, both
stored in the arena allocator that belongs to the context. This is a
preconditio for supporting recursive types that require delayed initialization,
in particular LLVM structure types.  The functionality is exercised in the test
pass with trivial self-recursive type. So far, recursive types can only be
printed in parsed in a closed type system. Removing this restriction is left
for future work.

Differential Revision: https://reviews.llvm.org/D84171
2020-07-27 13:07:44 +02:00
George Mitenkov 36618274f3 [MLIR][LLVMDialect] Added volatile and nontemporal attributes to load/store
This patch introduces 2 new optional attributes to `llvm.load`
and `llvm.store` ops: `volatile` and `nontemporal`. These attributes
are translated into proper LLVM as a `volatile` marker and a metadata node
respectively. They are also helpful with SPIR-V to LLVM dialect conversion
since they are the mappings for `Volatile` and `NonTemporal` Memory Operands.

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D84396
2020-07-27 10:55:56 +03:00
Mehdi Amini 1c93f09bf3 Remove declaration of constexpr member kDynamicSize in MemRefType
This member is already publicly declared on the base class. The
redundant declaration is mangled differently though and in some
unoptimized build it requires a definition to also exist. However we
have a definition for the base ShapedType class, removing the
declaration here will redirect every use to the base class member
instead.

Differential Revision: https://reviews.llvm.org/D84615
2020-07-27 04:50:08 +00:00
Jacques Pienaar 595d214f47 [mlir][shape] Further operand and result type generalization
Previous changes generalized some of the operands and results. Complete
a larger group of those to simplify progressive lowering. Also update
some of the declarative asm form due to generalization. Tried to keep it
mostly mechanical.
2020-07-25 21:41:31 -07:00
Frederik Gossen 07f227c0eb [MLIR][Shape] Allow `num_elements` to operate on extent tensors
Re-landing with dependent change landed and error condition relaxed.
Beyond the change to error condition exactly https://reviews.llvm.org/D84445.
2020-07-25 15:02:29 -07:00
Jacques Pienaar 5142448a5e [MLIR][Shape] Refactor verification
Based on https://reviews.llvm.org/D84439 but less restrictive, else we
don't allow shape_of to be able to produce a ranked output and doesn't
allow for iterative refinement here. We can consider making it more
restrictive later.
2020-07-25 14:55:19 -07:00
Jacques Pienaar 7bfecd7739 Revert "[MLIR][Shape] Allow `num_elements` to operate on extent tensors"
This reverts commit 55ced04d6b.

Forgot to submit depend change first.
2020-07-25 14:47:57 -07:00
Frederik Gossen 55ced04d6b [MLIR][Shape] Allow `num_elements` to operate on extent tensors
Differential Revision: https://reviews.llvm.org/D84445
2020-07-25 14:41:05 -07:00
Frederik Gossen 670ae4b6da [MLIR][Shape] Fold `shape.mul`
Implement constant folding for `shape.mul`.

Differential Revision: https://reviews.llvm.org/D84438
2020-07-24 13:30:45 +00:00
Frederik Gossen 783a351785 [MLIR][Shape] Allow `shape.mul` to operate in indices
Differential Revision: https://reviews.llvm.org/D84437
2020-07-24 13:25:40 +00:00
Frederik Gossen 5984d74139 [MLIR][Shape] Allow `get_extent` to operate on extent tensors and indices
Differential Revision: https://reviews.llvm.org/D84435
2020-07-24 11:13:17 +00:00
Frederik Gossen 7f600da828 [MLIR][Shape] Allow `shape.any` to operate on extent tensors
Differential Revision: https://reviews.llvm.org/D84433
2020-07-24 11:03:10 +00:00
Frederik Gossen 23a65648c0 [MLIR][Shape] Allow `shape.rank` to operate on extent tensors
Differential Revision: https://reviews.llvm.org/D84429
2020-07-24 10:43:39 +00:00
Frederik Gossen d4e4d5d780 [MLIR][Shape] Allow for `shape_of` to return extent tensors
The operation `shape.shape_of` now returns an extent tensor `tensor<?xindex>` in
cases when no error are possible. All consuming operation will eventually accept
both, shapes and extent tensors.

Differential Revision: https://reviews.llvm.org/D84160
2020-07-24 08:40:40 +00:00
Frederik Gossen 0e1a42efd8 [MLIR][Shape] Allow `shape.get_extent` to operate on extent tensors
`shape.get_extent` now accepts extent tensors `tensor<?xindex>` as an argument.

Differential Revision: https://reviews.llvm.org/D84158
2020-07-24 08:34:37 +00:00
Frederik Gossen 14d3cef012 [MLIR][Shape] Generalze `shape.const_shape` to extent tensors
The operation `shape.const_shape` was used for constants of type shape only.
We can now also use it to create constant extent tensors.

Differential Revision: https://reviews.llvm.org/D84157
2020-07-24 08:06:24 +00:00
George Mitenkov 99d03f0391 [MLIR][LLVMDialect] Added branch weights attribute to CondBrOp
This patch introduces branch weights metadata to `llvm.cond_br` op in
LLVM Dialect. It is modelled as optional `ElementsAttr`, for example:
```
llvm.cond_br %cond weights(dense<[1, 3]> : vector<2xi32>), ^bb1, ^bb2
```
When exporting to proper LLVM, this attribute is transformed into metadata
node. The test for metadata creation is added to `../Target/llvmir.mlir`.

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D83658
2020-07-24 10:11:13 +03:00
George Mitenkov 1563973f41 [MLIR][SPIRV] Updated documentation for variableOp
This is an update of the documentation for `spv.Variable`.
Removed `bind` and `built_in` that are now used with `spv.globalVariable`
instead.

Reviewed By: antiagainst

Differential Revision: https://reviews.llvm.org/D84196
2020-07-24 09:38:22 +03:00
River Riddle 4589dd924d [mlir][DialectConversion] Enable deeper integration of type conversions
This revision adds support for much deeper type conversion integration into the conversion process, and enables auto-generating cast operations when necessary. Type conversions are now largely automatically managed by the conversion infra when using a ConversionPattern with a provided TypeConverter. This removes the need for patterns to do type cast wrapping themselves and moves the burden to the infra. This makes it much easier to perform partial lowerings when type conversions are involved, as any lingering type conversions will be automatically resolved/legalized by the conversion infra.

To support this new integration, a few changes have been made to the type materialization API on TypeConverter. Materialization has been split into three separate categories:
* Argument Materialization: This type of materialization is used when converting the type of block arguments when calling `convertRegionTypes`. This is useful for contextually inserting additional conversion operations when converting a block argument type, such as when converting the types of a function signature.
* Source Materialization: This type of materialization is used to convert a legal type of the converter into a non-legal type, generally a source type. This may be called when uses of a non-legal type persist after the conversion process has finished.
* Target Materialization: This type of materialization is used to convert a non-legal, or source, type into a legal, or target, type. This type of materialization is used when applying a pattern on an operation, but the types of the operands have not yet been converted.

Differential Revision: https://reviews.llvm.org/D82831
2020-07-23 19:40:31 -07:00
Jakub Lichman e4dd964df0 [mlir] Loop bounds inference in linalg.generic op improved to support bounds for convolution
Loop bound inference is right now very limited as it supports only permutation maps and thus
it is impossible to implement convolution with linalg.generic as it requires more advanced
loop bound inference. This commits solves it for the convolution case.

Depends On D83158

Differential Revision: https://reviews.llvm.org/D83191
2020-07-23 11:01:54 +02:00
aartbik 1485fd295b [mlir] [VectorOps] Improve scatter/gather CPU performance
Replaced the linearized address with the proper LLVM way of
defining vector of base + indices in SIMD style. This yields
much better code. Some prototype results with microbencmarking
sparse matrix x vector with 50% sparsity (about 2-3x faster):

         LINEARIZED     IMPROVED
GFLOPS  sdot  saxpy     sdot saxpy
16x16    1.6   1.4       4.4  2.1
32x32    1.7   1.6       5.8  5.9
64x64    1.7   1.7       6.4  6.4
128x128  1.7   1.7       5.9  5.9
256x256  1.6   1.6       6.1  6.0
512x512  1.4   1.4       4.9  4.7

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D84368
2020-07-22 23:47:36 -07:00
Diego Caballero 3fff5acd8f [mlir][VectorOps] Expose SuperVectorizer as a utility
This patch refactors a small part of the Super Vectorizer code to
a utility so that it can be used independently from the pass. This
aligns vectorization with other utilities that we already have for loop
transformations, such as fusion, interchange, tiling, etc.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D84289
2020-07-22 14:22:15 -07:00
aartbik 19dbb230a2 [mlir] [VectorOps] Add scatter/gather operations to Vector dialect
Introduces the scatter/gather operations to the Vector dialect
(important memory operations for sparse computations), together
with a first reference implementation that lowers to the LLVM IR
dialect to enable running on CPU (and other targets that support
the corresponding LLVM IR intrinsics).

The operations can be used directly where applicable, or can be used
during progressively lowering to bring other memory operations closer to
hardware ISA support for a gather/scatter. The semantics of the operation
closely correspond to those of the corresponding llvm intrinsics.

Note that the operation allows for a dynamic index vector (which is
important for sparse computations). However, this first reference
lowering implementation "serializes" the address computation when
base + index_vector is converted to a vector of pointers. Exploring
how to use SIMD properly during these step is TBD. More general
memrefs and idiomatic versions of striding are also TBD.

Reviewed By: arpith-jacob

Differential Revision: https://reviews.llvm.org/D84039
2020-07-21 10:57:40 -07:00
Diego Caballero f8b72fba86 [MLIR][EDSC] Add fptrunc and fpext to EDSC
Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D84216
2020-07-21 08:55:18 -07:00
Alex Zinenko aa84e6e579 [mlir] Fix undefined behavior in Linalg utils getViewSizes
The utility function getViewSizes in Linalg has been recently updated to
support a different form of Linalg operations. In doing so, the code looking
like `smallvector.push_back(smallvector[i])` was introduced. Unlike std
vectors, this can lead to undefined behavior if the vector must grow upon
insertion: `smallvector[i]` returns a reference to the element, `push_back`
takes a const reference to the element, and then grows the vector storage
before accessing the referenced value. After the resize, the reference may
become dangling, which leads to undefined behavior detected by ASAN as
use-after-free. Work around the issue by forcing the value to be copied by
putting it into a temporary variable.
2020-07-21 09:57:41 +02:00