Commit Graph

11290 Commits

Author SHA1 Message Date
Denys Shabalin 89d4904541 [mlir] Fix declaration of nano time function in benchmark infra
In d4555698f8, the name of nano precision timer function has changed from `nano_time` to `nanoTime`, but benchmarks were not updated to reflect that. This change addresses the discrepancy.

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D125217
2022-05-13 13:22:18 +02:00
Groverkss 7b323af52a [MLIR] Fix areIdsUnique in AffineStructures
This patch fixes a bug in areIdsUnique where it ignores the [start, end] range.

No test case is added since there are no use cases through IR from where it
can be tested, and it is hard to create a unittest since we do not currently
have Values in unittests.

Reviewed By: arjunp

Differential Revision: https://reviews.llvm.org/D124735
2022-05-13 16:15:13 +05:30
Tres Popp 1de73629aa Add cmake dependency for TensorToLinalg 2022-05-13 12:33:14 +02:00
Tres Popp 1dce51b888 [mlir] Add TensorToLinalgPass
This pass is to handle computationally complex operations like
tensor.pad which are not simply lowered to the exact same operation in
the memref dialect.

Differential Revision: https://reviews.llvm.org/D125384
2022-05-13 12:17:22 +02:00
Alex Zinenko 7deed49ab9 [mlir] use dynamic sections in MLIR Doxygen
Due to an apparent bug in the Doxygen version <1.8.16 used to generate
documentation for MLIR, parts of the navigation (specifically, the lists
of inherited methods for classes) are unusable due to dynsections.js
missing from the output generated by Doxygen. Setting this flag makes
Doxygen always produce the file.
2022-05-13 11:43:52 +02:00
Matthias Springer e9fa559097 [mlir][sparse][NFC] Use RewriterBase/OpBuilder when possible
Most functions do not need a PatternRewriter or ConversionPatternRewriter.

Differential Revision: https://reviews.llvm.org/D125466
2022-05-13 11:37:26 +02:00
Matthias Springer 8f42939a07 [mlir][bufferize][NFC] Make getContiguousMemRefType a static function
No need to expose this as public API anymore.

Differential Revision: https://reviews.llvm.org/D125361
2022-05-13 11:27:43 +02:00
wren romano 753fe330c1 [mlir][sparse] Factoring out an enumerator over elements of SparseTensorStorage
Work towards fixing: https://github.com/llvm/llvm-project/issues/51652

Depends On D122928

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D122060
2022-05-12 17:05:56 -07:00
Aart Bik 6f3c7dfb77 [mlir][sparse] add sparse sign integration test
Implements a floating-point sign operator (using the new semi-ring ops)
that accomodates +/-Inf and +/-NaN in consistent way.

Reviewed By: bixia

Differential Revision: https://reviews.llvm.org/D125494
2022-05-12 15:56:36 -07:00
River Riddle 80c28a400c [mlir] Bump mlir-vscode to 0.0.7
The syntax highlighting for mlir has gotten a significant facelift,
we also have some recent bug fixes for server launches.
2022-05-12 15:52:46 -07:00
River Riddle 86f5caeee9 [mlir] Significantly overhaul the textmate grammar
The current grammar is really crusty, only supports a handful of
cases, and is also out-of-date after various refactorings. This commit
refactors the textmate grammar to handle significantly more cases,
and now provides proper coloring for a majority of cases (including
dialect attributes, operations, types, etc.)

Differential Revision: https://reviews.llvm.org/D125458
2022-05-12 15:50:32 -07:00
River Riddle 86e1c2f097 [mlir] Fix pipeline-parsing.mlir on windows
We shouldn't be making assumptions about the result of llvm::getTypeName,
which may have different results for anonymous namespaces depending
on the platform.
2022-05-12 13:40:16 -07:00
River Riddle c2fb9c29b4 [mlir:Pass] Add support for op-agnostic pass managers
This commit refactors the current pass manager support to allow for
operation agnostic pass managers. This allows for a series of passes
to be executed on any viable pass manager root operation, instead
of one specific operation type. Op-agnostic/generic pass managers
only allow for adding op-agnostic passes.

These types of pass managers are extremely useful when constructing
pass pipelines that can apply to many different types of operations,
e.g., the default inliner simplification pipeline. With the advent of
interface/trait passes, this support can be used to define FunctionOpInterface
pass managers, or other pass managers that effectively operate on
specific interfaces/traits/etc (see #52916 for an example).

Differential Revision: https://reviews.llvm.org/D123536
2022-05-12 13:12:59 -07:00
Ashay Rane 5380e30e04
[mlir] translate memref.reshape ops that have static shapes
This patch references code for translating memref.reinterpret_cast ops
to add translation rules for memref.reshape ops that have a static shape
argument.  Since reshape ops don't have offsets, sizes, or strides, this
patch simply sets the allocated and aligned pointers of the MemRef
descriptor.

Reviewed By: ftynse, cathyzhyi

Differential Revision: https://reviews.llvm.org/D125039
2022-05-12 11:57:20 -07:00
Benjamin Kramer 434385ba41 [DenseElementAttr] Silence warning in -DNDEBUG builds. NFC. 2022-05-12 17:59:39 +02:00
Chris Lattner 3cce374ee6 Various improvements suggested by river NFC.
Differential Revision: https://reviews.llvm.org/D125471
2022-05-12 16:18:23 +01:00
Chris Lattner f21896f2c6 [DenseElementAttr] Simplify the public API for creating these.
Instead of requiring the client to compute the "isSplat" bit,
compute it internally.  This makes the logic more consistent
and defines away a lot of "elements.size()==1" in the clients.

This addresses Issue #55185

Differential Revision: https://reviews.llvm.org/D125447
2022-05-12 16:18:23 +01:00
Thomas Raoux d02f10d96d [mlir][vector] Add lowering pattern for vector.warp_execute_on_lane_0 op
Add lowering of the vector.warp_execute_on_lane_0 into scf.if plus memory
transfer for the operands and yield values.

This also add an integration test running on GPU warp. The same tests can be
later re-used with different comment lines to tests distribution
transformations.

This is mostly from @springerm contribution.

Differential Revision: https://reviews.llvm.org/D125430
2022-05-12 13:27:43 +00:00
Benjamin Kramer 303638248a [mlir][linalg] Add lowering of named ops on complex numbers
This lets linalg.dot and friends lower to a complex muladd using ops
from the complex dialect.

Differential Revision: https://reviews.llvm.org/D125461
2022-05-12 13:37:34 +02:00
Benjamin Kramer 27dad99622 [mlir][LLVM] Make the nested type restriction on complex constants less aggressive
Complex nested in other types is perfectly fine, just nested structs
aren't supported. Instead of checking whether there's nesting just check
whether the struct we're dealing with is a complex number.

Differential Revision: https://reviews.llvm.org/D125381
2022-05-12 11:47:01 +02:00
Daniil Dudkin 70c463efc8 [mlir][NFC] Fix `GpuKernelOutliningPass` copy constructor warnings
1. Call copy constructor of the base class
2. Assign value of the option directly

Reviewed By: dcaballe, rriddle

Differential Revision: https://reviews.llvm.org/D125101
2022-05-12 11:41:18 +03:00
Nikita Popov f02716a806 [MLIR] Fix build without native arch
D125214 split off a MLIRExecutionEngineUtils library that is used
by MLIRGPUTransforms. However, currently the entire ExecutionEngine
directory is skipped if the LLVM_NATIVE_ARCH target is not available.

Move the check for LLVM_NATIVE_ARCH, such that MLIRExecutionEngineUtils
always gets built, and only the JIT-related libraries are omitted
without native arch.

Differential Revision: https://reviews.llvm.org/D125357
2022-05-12 09:50:51 +02:00
Matthias Springer 82ea0d8b82 [mlir][bufferize] Support alloc hoisting across function boundaries
This change integrates the BufferResultsToOutParamsPass into One-Shot Module Bufferization. This improves memory management (deallocation) when buffers are returned from a function.

Note: This currently only works with statically-sized tensors. The generated code is not very efficient yet and there are opportunities for improvment (fewer copies). By default, this new functionality is deactivated.

Differential Revision: https://reviews.llvm.org/D125376
2022-05-12 09:44:07 +02:00
Matthias Springer 2fe40c34ea [mlir][bufferize] Fix op filter
Bufferization has an optional filter to exclude certain ops from analysis+bufferization. There were a few remaining places in the codebase where the filter was not checked.

Differential Revision: https://reviews.llvm.org/D125356
2022-05-12 09:33:07 +02:00
Matthias Springer 011f1b1c1f [mlir][bufferize] Add helpers for templatized DENY filters
We already have templatized ALLOW filters but the DENY filters were missing.

Differential Revision: https://reviews.llvm.org/D125358
2022-05-12 09:18:21 +02:00
River Riddle 1155c1fe65 [mlir:Parser] Emit a better diagnostic when a custom operation is unknown
When a custom operation is unknown and does not have a dialect prefix, we currently
emit an error using the name of the operation with the default dialect prefix. This
leads to a confusing error message, especially when operations get moved between dialects.
For example, `func` was recently moved out of `builtin` and to the `func` dialect. The current
error message we get is:

```
func @foo()
^ custom op 'builtin.func' is unknown
```

This could lead users to believe that there is supposed to be a `builtin.func`,
because there used to be. This commit adds a better error message that does
not assume that the operation is supposed to be in the default dialect:

```
func @foo()
^ custom op 'func' is unknown (tried 'builtin.func' as well)
```

Differential Revision: https://reviews.llvm.org/D125351
2022-05-11 22:54:44 -07:00
Mahesh Ravishankar 8be7e6f56a [mlir][Linalg] Combine canonicalizers that deal with removing dead/redundant args.
`linalg.generic` ops have canonicalizers that either remove arguments
not used in the payload, or redundant arguments. Combine these and
enhance the canonicalization to also remove results that have no use.
This is effectively dead code elimination for Linalg ops.

Differential Revision: https://reviews.llvm.org/D123632
2022-05-12 05:22:30 +00:00
Mogball 0ffef0c23b [mlir][ods] (NFC) don't use std::function for map_range 2022-05-12 05:15:03 +00:00
Mogball 19906262c9 [mlir] (NFC) Use assembly format for test.graph_region 2022-05-12 04:19:25 +00:00
bzcheeseman bc22b5c9a2 [MLIR][Operation] Simplify Operation casting, NFC
We can simplify the code needed to implement dyn_cast/cast/isa support for MLIR operations with documented interfaces via the CastInfo structures. This will also provide an example of how to use CastInfo.

Depends on D123901

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D124963
2022-05-12 00:17:01 -04:00
grosul1 a4b227c28a [mlir] Fix loop unrolling: properly replace the arguments of the epilogue loop.
Using "replaceUsesOfWith" is incorrect because the same initializer value may appear multiple times.

For example, if the epilogue is needed when this loop is unrolled
```
%x:2 = scf.for ... iter_args(%arg1 = %c1, %arg2 = %c1) {
  ...
}
```
then both epilogue's arguments will be incorrectly renamed to use the same result index (note #1 in both cases):
```
%x_unrolled:2 = scf.for ... iter_args(%arg1 = %c1, %arg2 = %c1) {
  ...
}
%x_epilogue:2 = scf.for ... iter_args(%arg1 = %x_unrolled#1, %arg2 = %x_unrolled#1) {
  ...
}
```
2022-05-12 01:54:39 +00:00
Chris Lattner 86445e8c63 [AsmParser] Adopt emitWrongTokenError more, improving QoI
This is a full audit of emitError calls, I took the opportunity
to remove extranous parens and fix a couple cases where we'd
generate multiple diagnostics for the same error.

Differential Revision: https://reviews.llvm.org/D125355
2022-05-11 20:41:12 +01:00
River Riddle 5a9a438a54 [TableGen] Refactor TableGenParseFile to no longer use a callback
Now that TableGen no longer relies on global Record state, we can allow
for the client to own the RecordKeeper and SourceMgr. Given that TableGen
internally still relies on the global llvm::SrcMgr, this method unfortunately
still isn't thread-safe.

Differential Revision: https://reviews.llvm.org/D125277
2022-05-11 11:55:33 -07:00
Matthias Springer 248e113e9f [mlir][bufferize][NFC] Move helper functions to BufferizationOptions
Move helper functions for creating allocs/deallocs/memcpys to BufferizationOptions.

Differential Revision: https://reviews.llvm.org/D125375
2022-05-11 16:23:22 +02:00
Chris Lattner 34b6f206cb [AsmParser] Improve error recovery again.
Change the parsing logic to use StringRef instead of lower level
char* logic.  Also, if emitting a diagnostic on the first token
in the file, we make sure to use that position instead of the
very start of the file.

Differential Revision: https://reviews.llvm.org/D125353
2022-05-11 08:25:36 +01:00
Chia-hung Duan 96e642652b [mlir] Print some message for op-printing verification
Before dump, Insetad of switching to generic form silently after
verification failure. Print some debug logs to help identify why an op
may be printed in a different way.

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D125136
2022-05-10 22:48:47 +00:00
Thomas Raoux 15bcc36eed [mlir][gpu] Move async copy ops to NVGPU and add caching hints
Move async copy operations to NVGPU as they only exist on NV target and are
designed to match ptx semantic. This allows us to also add more fine grain
caching hint attribute to the op.
Add hint to bypass L1 and hook it up to NVVM op.

Differential Revision: https://reviews.llvm.org/D125244
2022-05-10 22:30:24 +00:00
Nicolas Vasilache 1f23211cb1 [mlir][SCF] Retire `cloneWithNewYields` helper function.
This is now subsumed by `replaceLoopWithNewYields`.

Differential Revision: https://reviews.llvm.org/D125309
2022-05-10 18:44:11 +00:00
Mahesh Ravishankar 567fd523bf [mlir][SCF] Add utility method to add new yield values to a loop.
The current implementation of `cloneWithNewYields` has a few issues
- It clones the loop body of the original loop to create a new
  loop. This is very expensive.
- It performs `erase` operations which are incompatible when this
  method is called from within a pattern rewrite. All erases need to
  go through `PatternRewriter`.

To address these a new utility method `replaceLoopWithNewYields` is added
which
- moves the operations from the original loop into the new loop.
- replaces all uses of the original loop with the corresponding
  results of the new loop
- use a call back to allow caller to generate the new yield values.
- the original loop is modified to just yield the basic block
  arguments corresponding to the iter_args of the loop. This
  represents a no-op loop. The loop itself is dead (since all its uses
  are replaced), but is not removed. The caller is expected to erase
  the op. Consequently, this method can be called from within a
  `matchAndRewrite` method of a `PatternRewriter`.

The `cloneWithNewYields` could be replaces with
`replaceLoopWithNewYields`, but that seems to trigger a failure during
walks, potentially due to the operations being moved. That is left as
a TODO.

Differential Revision: https://reviews.llvm.org/D125147
2022-05-10 18:44:11 +00:00
Krzysztof Drewniak 814b605095 [mlir][AMDGPU] Add AMDGPU conversion patterns to ConvertGPUToROCDL
This ensures that attributes such as the index bitwidth propagate
correctly to the AMDGPUToROCDL patterns.

Differential Revision: https://reviews.llvm.org/D125320
2022-05-10 16:49:11 +00:00
Ashay Rane 53ff0daa7e
[mlir] Fail early if AnalysisState::getBuffer() returns failure
This patch updates calls to AnalysisState::getBuffer() so that we return
early with a failure if the call does not succeed.

Reviewed By: springerm

Differential Revision: https://reviews.llvm.org/D125251
2022-05-10 08:08:38 -07:00
Krzysztof Drewniak f1f05a91ca [MLIR][AMDGPU] Add AMDGPU dialect, wrappers around raw buffer intrinsics
By analogy with the NVGPU dialect, introduce an AMDGPU dialect for
AMD-specific intrinsic wrappers.

The dialect initially includes wrappers around the raw buffer intrinsics.

On AMD GPUs, a memref can be converted to a "buffer descriptor" that
allows more precise control of memory access, such as by allowing for
out of bounds loads/stores to be replaced by 0/ignored without adding
additional conditional logic, which is important for performance.

The repository currently contains a limited conversion from
transfer_read/transfer_write to Mubuf intrinsics, which are an older,
deprecated intrinsic for the same functionality.

The new amdgpu.raw_buffer_* ops allow these operations to be used
explicitly and for including metadata such as whether the target
chipset is an RDNA chip or not (which impacts the interpretation of
some bits in the buffer descriptor), while still maintaining an
MLIR-like interface.

(This change also exposes the floating-point atomic add intrinsic.)

Reviewed By: ThomasRaoux

Differential Revision: https://reviews.llvm.org/D122765
2022-05-10 14:59:58 +00:00
Chris Lattner ad3b358180 [MLIR Parser] Improve QoI for "expected token" errors
A typical problem with missing a token is that the missing
token is at the end of a line.  The problem with this is that
the error message gets reported on the start of the following
line (which is where the next / invalid token is) which can
be confusing.

Handle this by noticing this case and backing up to the end of
the previous line.

Differential Revision: https://reviews.llvm.org/D125295
2022-05-10 15:44:17 +01:00
Adrian Kuegel 64c8574209 [mlir] Remove unused using declaration (NFC) 2022-05-10 12:58:01 +02:00
Nikita Popov 03ab30686d [MLIR] Split off MLIRExecutionEngineUtils to fix libMLIR.so build (PR54242)
Building libMLIR.so currently fails with:

> /usr/bin/ld: /tmp/ccNzulEA.ltrans39.ltrans.o: in function `(anonymous namespace)::SerializeToHsacoPass::optimizeLlvm(llvm::Module&, llvm::TargetMachine&)':
> /builddir/build/BUILD/llvm-project-15.0.0.src/mlir/lib/Dialect/GPU/Transforms/SerializeToHsaco.cpp:328: undefined reference to `mlir::makeOptimizingTransformer(unsigned int, unsigned int, llvm::TargetMachine*)'

This is because MLIRGPUTransforms depends on MLIRExecutionEngine in
61bb2e4ea8/mlir/lib/Dialect/GPU/Transforms/SerializeToHsaco.cpp (L328),
but MLIRExecutionEngine is marked as excluded from libMLIR.so.

However, this code doesn't require the full execution engine: It
only performs middle-end optimization, and does not need any of
the JIT/codegen infrastructure. As such, split off a separate
library MLIRExecutionEngineUtils, which only contains that part
and is not excluded from libMLIR.so.

Fixes https://github.com/llvm/llvm-project/issues/54242.

Differential Revision: https://reviews.llvm.org/D125214
2022-05-10 10:17:52 +02:00
Stella Stamenova 784a5bccfd [mlir] Fix python bindings build on Windows in Debug
Currently, building mlir with the python bindings enabled on Windows in Debug is broken because pybind11, python and cmake don't like to play together. This change normalizes how the three interact, so that the builds can now run and succeed.

The main issue is that python and cmake both make assumptions about which libraries are needed in a Windows build based on the flavor.
- cmake assumes that a debug (or a debug-like) flavor of the build will always require pythonX_d.lib and provides no option/hint to tell it to use a different library. cmake does find both the debug and release versions, but then uses the debug library.
- python (specifically pyconfig.h and by extension python.h) hardcodes the dependency on pythonX_d.lib or pythonX.lib depending on whether `_DEBUG` is defined. This is NOT transparent - it does not show up anywhere in the build logs until the link step fails with `pythonX_d.lib is missing` (or `pythonX.lib is missing`)
- pybind11 tries to "fix" this by implementing a workaround - unless Py_DEBUG is defined, `_DEBUG` is explicitly undefined right before including python headers. This also requires some windows headers to be included differently, so while clever, this is a non-trivial workaround.

mlir itself includes the pybind11 headers (which contain the workaround) AS WELL AS python.h, essentially always requiring both pythonX.lib and pythonX_d.lib for linking. cmake explicitly only adds one or the other, so the build fails.

This change does a couple of things:
- In the cmake files, explicitly add the release version of the python library on Windows builds regardless of flavor. Since Py_DEBUG is not defined, pybind11 will always require release and it will be satisfied
- To satisfy python as well, this change removes any explicit inclusions of Python.h on Windows instead relying on the fact that pybind11 headers will bring in what is needed

There are a few additional things that we could do but I rejected as unnecessary at this time:
- define Py_DEBUG based on the CMAKE_BUILD_TYPE - this will *mostly* work, we'd have to think through multiconfig generators like VS, but it's possible. There doesn't seem to be a need to link against debug python at the moment, so I chose not to overcomplicate the build and always default to release
- similar to above, but define Py_DEBUG based on the CMAKE_BUILD_TYPE *as well as* the presence of the debug python library (`Python3_LIBRARY_DEBUG`). Similar to above, this seems unnecessary right now. I think it's slightly better than above because most people don't actually have the debug version of python installed, so this would prevent breaks in that case.
- similar to the two above, but add a cmake variable to control the logic
- implement the pybind11 workaround directly in mlir (specifically in Interop.h) so that Python.h can still be included directly. This seems prone to error and a pain to maintain in lock step with pybind11
- reorganize how the pybind11 headers are included and place at least one of them in Interop.h directly, so that the header has all of its dependencies included as was the original intention. I decided against this because it really doesn't need pybind11 logic and it's always included after pybind11 is, so we don't necessarily need the python includes

Reviewed By: stellaraccident

Differential Revision: https://reviews.llvm.org/D125284
2022-05-09 19:46:47 -07:00
Mathieu Fehr 67d0bc27c0 [mlir][doc] Move documentation of extensible dialects
Merge the documentation of the definition of extensible dialects
with the definition of dialects.

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D125200
2022-05-09 16:01:12 -07:00
River Riddle 867cd5007d [mlir-LSP] Ensure existing documents are process synchronously
This prevents races where we accidentally launched multiple servers.
2022-05-09 15:23:23 -07:00
Thomas Raoux 09fc685ce6 [mlir][nvvm] Add attribute to nvvm.cpAsyncOp to control l1 bypass
Add attribute to be able to generate the intrinsic version of async copy
generating a copy with l1 bypass. This correspond to
cp.async.cg.shared.global in ptx.

Differential Revision: https://reviews.llvm.org/D125241
2022-05-09 19:34:48 +00:00
Stella Stamenova 057863a9bc [mlir] Fix build & test of mlir python bindings on Windows
There are a couple of issues with the python bindings on Windows:
- `create_symlink` requires special permissions on Windows - using `copy_if_different` instead allows the build to complete and then be usable
- the path to the `python_executable` is likely to contain spaces if python is installed in Program Files. llvm's python substitution adds extra quotes in order to account for this case, but mlir's own python substitution does not
- the location of the shared libraries is different on windows
- if the type is not specified for numpy arrays, they appear to be treated as strings

I've implemented the smallest possible changes for each of these in the patch, but I would actually prefer a slightly more comprehensive fix for the python_executable and the shared libraries.

For the python substitution, I think it makes sense to leverage the existing %python instead of adding %PYTHON and instead add a new variable for the case when preloading is needed. This would also make it clearer which tests are which and should be skipped on platforms where the preloading won't work.

For the shared libraries, I think it would make sense to pass the correct path and extension (possibly even the names) to the python script since these are known by lit and don't have to be hardcoded in the test at all.

Reviewed By: stellaraccident

Differential Revision: https://reviews.llvm.org/D125122
2022-05-09 11:10:20 -07:00
Jakub Tucholski 167bbfcb9d [mlir] Refactoring dialect and test code to use parseCommaSeparatedList
Issue #55173

Reviewed By: lattner, rriddle

Differential Revision: https://reviews.llvm.org/D124791
2022-05-09 12:36:54 -04:00
Jerry Wu ad7c49bef7 [mlir][linalg] Fix padding size calculation for Conv2d ops.
This patch fixed the padding size calculation for Conv2d ops when the stride > 1. It contains the changes below:

- Use addBound to add constraint for AffineApplyOp in getUpperBoundForIndex. So the result value can be mapped and retrieved later.

- Fixed the bound from AffineMinOp by adding as a closed bound. Originally the bound was added as an open upper bound, which results in the incorrect bounds when we multiply the values. For example:

```
%0 = affine.min affine_map<()[s0] -> (4, -s0 + 11)>()[iv0]
%1 = affine.apply affine_map<()[s0] -> (s0 * 2)>()[%0]

If we add the affine.min as an open bound, addBound will internally transform it into the close bound "%0 <= 3". The following sliceBounds will derive the bound of %1 as "%1 <= 6" and return the open bound "%1 < 7", while the correct bound should be "%1 <= 8".
```

- In addition to addBound, I also changed sliceBounds to support returning closed upper bound, since for the size computation, we usually care about the closed bounds.

- Change the getUpperBoundForIndex to favor constant bounds when required. The sliceBounds will return a tighter but non-constant bounds, which can't be used for padding. The constantRequired option requires getUpperBoundForIndex to get the constant bounds when possible.

Reviewed By: hanchung

Differential Revision: https://reviews.llvm.org/D124821
2022-05-09 08:45:37 -07:00
Ashay Rane e287d647c6 [mlir] Add translation from tensor.reshape to memref.reshape
This patch augments the `tensor-bufferize` pass by adding a conversion
rule to translate ReshapeOp from the `tensor` dialect to the `memref`
dialect, in addition to adding a unit test to validate the translation.

Reviewed By: springerm

Differential Revision: https://reviews.llvm.org/D125031
2022-05-09 17:45:07 +02:00
Benjamin Kramer a48adc5658 [mlir][math] Promote (b)f16 to f32 when lowering to libm calls
libm doesn't have overloads for the small types, so promote them to a
bigger type and use the f32 function.

Differential Revision: https://reviews.llvm.org/D125093
2022-05-09 11:59:55 +02:00
Christopher Bate 9879807393 [mlir][NvGpu] Fix nvgpu.mma.sync lowering to NVVM for f32, tf32 types
Adds missing logic in the lowering from NvGPU to NVVM to support fp32
(in an accumulator operand) and tf32 (in multiplicand operand) types.
Fixes logic in one of the helper functions for converting the result
of a mma.sync operation with multiple 8x256bit output tiles, which is
the case for f32 outputs.

Differential Revision: https://reviews.llvm.org/D124533
2022-05-08 21:49:42 -06:00
Sam McCall e571e1a6c3 Reland "[FuzzMutate] Split out FuzzerCLI library that doesn't depend on IR."
This reverts commit a1bb952e83.

I'd somehow missed updating llvm-yaml-parser-fuzzer, now fixed.
2022-05-07 13:49:54 +02:00
Aaron Ballman a1bb952e83 Revert "[FuzzMutate] Split out FuzzerCLI library that doesn't depend on IR."
This reverts commit 1c5e85b3da.

It broke a lot of bots with a link error:
https://lab.llvm.org/buildbot/#/builders/171/builds/14222
https://lab.llvm.org/buildbot/#/builders/188/builds/13748
https://lab.llvm.org/buildbot/#/builders/109/builds/38127
2022-05-07 07:29:57 -04:00
Sam McCall 1c5e85b3da [FuzzMutate] Split out FuzzerCLI library that doesn't depend on IR.
All llvm-project fuzzers use this library to parse command-line arguments.
Many of them don't deal with LLVM IR or modules in any way. Bundling those
functions in one library forces build dependencies that don't need to be there.

Among other things, this means check-clang-pseudo no longer depends on most of
LLVM.

Differential Revision: https://reviews.llvm.org/D125081
2022-05-07 12:11:51 +02:00
Mehdi Amini 25cd6fba98 Fix MLIR integration test after a8308020 (`func.` prefix is required bythe parser now) 2022-05-07 09:09:24 +00:00
River Riddle a8308020ac [mlir] Remove special case parsing/printing of `func` operations
This was leftover from when the standard dialect was destroyed, and
when FuncOp moved to the func dialect. Now that these transitions
have settled a bit we can drop these.

Most updates were handled using a simple regex: replace `^( *)func` with `$1func.func`

Differential Revision: https://reviews.llvm.org/D124146
2022-05-06 13:36:15 -07:00
Mehdi Amini 6a9c1029f8 Fix build with shared libs: add missing CMake dep to MLIR sparse pipeline 2022-05-06 20:20:03 +00:00
Mehdi Amini b37d158f71 Apply clang-tidy fixes for bugprone-copy-constructor-init in TestPassManager.cpp (NFC) 2022-05-06 20:19:19 +00:00
Mehdi Amini 298d2fa1c5 Apply clang-tidy fixes for readability-identifier-naming in SparseTensorUtils.cpp (NFC) 2022-05-06 20:19:19 +00:00
Mehdi Amini 90c2af57af Apply clang-tidy fixes for llvm-include-order in Merger.cpp (NFC) 2022-05-06 20:19:19 +00:00
Mehdi Amini 072e0aabbc Enable the use of ThreadPoolTaskGroup in MLIR threading helper to enable nested parallelism
The LLVM ThreadPool recently got the addition of the concept of
ThreadPoolTaskGroup: this is a way to "partition" the threadpool
into a group of tasks and enable nested parallelism through this
grouping at every level of nesting.
We make use of this feature in MLIR threading abstraction to fix a long
lasting TODO and enable nested parallelism.

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D124902
2022-05-06 19:40:22 +00:00
Mehdi Amini c5ea8d509c Apply clang-tidy fixes for llvm-else-after-return in Merger.cpp (NFC) 2022-05-06 19:38:03 +00:00
Mehdi Amini 061f253e13 Apply clang-tidy fixes for llvm-prefer-isa-or-dyn-cast-in-conditionals in OpenMPDialect.cpp (NFC) 2022-05-06 19:38:02 +00:00
Aart Bik 5b122a7310 [mlir][sparse] integration test for zero preserving math op
Also fixes omission in lowering math ops that require lib support

Reviewed By: bixia

Differential Revision: https://reviews.llvm.org/D125104
2022-05-06 10:42:33 -07:00
Nikita Popov 686bd6dd2b [MLIR] Fix build with make
https://reviews.llvm.org/D124075 causes MLIR to no longer build
when using make rather than ninja, due to a tablegen-generated
header being used before it is created.

It seems that this is related to the use of LLVM_ENABLE_OBJLIB when
using add_tablgen with a non-Ninja/Xcode generator. In that case an
intermediate objlib target is generated.

This patch fixes the issue by a) declaring dependencies in
add_tablegen for mlir-pdll and b) making sure those dependencies
are added to the objlib target.

Differential Revision: https://reviews.llvm.org/D125010
2022-05-06 14:52:35 +02:00
Matthias Springer 9785eb1b98 [mlir][bufferize] Disallow adding new bufferizable ops during bufferization
Ops that are created during the bufferization were not analyzed (when run with One-Shot Bufferize), and users should instead create memref ops directly.

Futhermore, this fixes an issue where an op was erased (and put on the `erasedOps` list), but subsequently a new tensor op was created at the same memory location. This op was then not bufferized. Disallowing the creation of new tensor ops simplifies the bufferization and fixes such issues.

Differential Revision: https://reviews.llvm.org/D125017
2022-05-06 18:41:49 +09:00
Matthias Springer 988748c077 [mlir][bufferize] Do not copy buffers with undefined contents
Buffers with undefined contents (e.g., the result of an init_tensor) are no longer copied.

Differential Revision: https://reviews.llvm.org/D125015
2022-05-06 17:31:01 +09:00
Matthias Springer a5d09c6372 [mlir][scf] Implement BufferizableOpInterface for scf::WhileOp
This follows the same implementation strategy as scf::ForOp and common functionality is extracted into helper functions.

This implementation works well in cases where each yielded value (from either body/condition region) is equivalent to the corresponding bbArg of the parent block. In that case, each OpResult of the loop may be aliasing with the corresponding OpOperand of the loop (and with no other OpOperand).

In the absence of said equivalence relationship, new buffer copies must be inserted, so that the aliasing OpOperand/OpResult contract of scf::WhileOp is honored. In essence, by yielding a newly allocated buffer, we can enforce the specified may-alias relationship. (Newly allocated buffers cannot alias with any OpOperands of the loop.)

Differential Revision: https://reviews.llvm.org/D124929
2022-05-06 17:24:33 +09:00
Mehdi Amini be310632d0 Apply clang-tidy fixes for llvm-else-after-return in OpenMPDialect.cpp (NFC) 2022-05-06 05:40:02 +00:00
Mehdi Amini 6e32535078 Apply clang-tidy fixes for bugprone-argument-comment in AffineOps.cpp (NFC) 2022-05-06 05:40:02 +00:00
Chengji Yao e5a4cf6743 [mlir] Fix printer when it is a DenseElementsAttr of i1
A large DenseElementsAttr of i1could trigger a bug in printer/parser roundtrip.

Ex. A DenseElementsAttr of i1 with 200 elements will print as Hex format of length 400 before the fix. However, when parsing the printed text, an error will be triggered. After fix, the printed length will be 50.

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D122925
2022-05-05 16:56:43 -07:00
Aart Bik 952fa3018e [mlir][sparse] add more zero-preserving unary ops to sparse compiler
Although we now have semi-rings to deal with arbitrary ops,
it is still good to convey zero-preserving semantics of
ops to the sparse compiler.

Reviewed By: bixia

Differential Revision: https://reviews.llvm.org/D125043
2022-05-05 15:35:19 -07:00
River Riddle 6609c1cc59 [mlir] Add a better error message when failing to parse an attribute
The fallback attribute parse path is parsing a Type attribute, but this results
in a really unintuitive error message: `expected non-function type`, which
doesn't really hint at tall that we were trying to parse an attribute. This
commit fixes this by trying to optionally parse a type, and on failure
emitting an error that we were expecting an attribute.

Differential Revision: https://reviews.llvm.org/D124870
2022-05-05 15:06:11 -07:00
River Riddle 8bb5b657fe [mlir:ExecutionEngine] Update use of getAddress now that lookup returns ExecutorAddr
This was changed in 16dcbb53dc
2022-05-05 14:24:32 -07:00
Stella Stamenova d4555698f8 [mlir] Fix the names of exported functions
The names of the functions that are supposed to be exported do not match the implementations. This is due in part to cac7aabbd8.

This change makes the implementations and declarations match and adds a couple missing declarations.

The new names follow the pattern of the existing `verify` functions where the prefix is maintained as `_mlir_ciface_` but the suffix follows the new naming convention.

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D124891
2022-05-05 13:46:15 -07:00
Christopher Bate 22c6e7b277 [mlir][nvvm] Fix support for tf32 data type in mma.sync
The NVVM dialect test coverage for all possible type/shape combinations
in the `nvvm.mma.sync` op is mostly complete. However, there were tests
missing for TF32 datatype support. This change adds tests for the one
relevant shape/type combination. This uncovered a small bug in the op
verifier, which this change also fixes.

Differential Revision: https://reviews.llvm.org/D124975
2022-05-05 11:02:03 -06:00
Matthias Springer e300682597 [mlir][scf][bufferize] Update verifyAnalysis error message
The previous error message was technically incorrect. We do not compare equivalence of YieldOp operands and ForOp operands.

Differential Revision: https://reviews.llvm.org/D124934
2022-05-05 16:56:50 +09:00
Matthias Springer 417e1c7d52 [mlir][scf][bufferize][NFC] Split ForOp bufferization into smaller functions
This is in preparation of WhileOp bufferization, which reuses these functions.

Differential Revision: https://reviews.llvm.org/D124933
2022-05-05 16:55:44 +09:00
Matthias Springer f178c386f5 [mlir][scf][bufferize][NFC] Simplify verifyAnalysis implementation
Differential Revision: https://reviews.llvm.org/D124928
2022-05-05 16:51:10 +09:00
Min-Yih Hsu 794c4218a6 [mlir][LLVMIR] Do not update instMap via assignments to entry references
Inside processInstruction, we assign the translated mlir::Value to a
reference previously taken from the corresponding entry in instMap.
However, instMap (a DenseMap) might resize after the entry reference was
taken, rendering the assignment useless since it's assigning to a
dangling reference. Here is a (pseudo) snippet that shows the concept:
```
// inst has type llvm::Instruction *
Value &v = instMap[inst];
...
// op is one of the operands of inst, has type llvm::Value *
processValue(op);
// instMap resizes inside processValue
...
translatedValue = b.createOp<Foo>(...);
// v is already a dangling reference at this point!
// The following assignment is bogus.
v = translatedValue;
```

Nevertheless, after we stop caching llvm::Constant into instMap, there
is only one case that can cause processValue to resize instMap: If the
operand is a llvm::ConstantExpr. In which case we will insert the
derived llvm::Instruction into instMap.
To trigger instMap to resize, which is a DenseMap, the threshold depends
on the ratio between # of map entries and # of (hash) buckets. More specifically,
it resizes if (# of map entries / # of buckets) >= 0.75.
In this case # of map entries is equal to # of LLVM instructions, and # of
buckets is the power-of-two upperbound of # of map entries. Thus, eventually
in the attaching test case (test/Target/LLVMIR/Import/incorrect-instmap-assignment.ll),
we picked 96 and 128 for the # of map entries and # of buckets, respectively.
(We can't pick numbers that are too small since DenseMap used inlined
storage for small number of entries). Therefore, the ConstantExpr in the
said test case (i.e. a GEP) is the 96-th llvm::Value cached into the
instMap, triggering the issue we're discussing here on its enclosing
instruction (i.e. a load).

This patch fixes this issue by calling `operator[]` everytime we need to
update an entry.

Differential Revision: https://reviews.llvm.org/D124627
2022-05-04 13:16:51 -07:00
Bixia Zheng 1cd13e6e98 [mlir][sparse][taco] Support more data types.
Support int8, int16, int32 and int32. Also fix source code format in mlir_pytaco_utils.py.

Add tests.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D124925
2022-05-04 10:05:20 -07:00
Alexander Belyaev e8f7d019fc [mlir] Add a flag to allow equivalent results.
Differential Revision: https://reviews.llvm.org/D124931
2022-05-04 17:48:18 +02:00
Matthias Springer b34ea97f55 [mlir][linalg][bufferize][NFC] Remove remaining Comprehensive Bufferize code
This commit removes the Linalg Comprehensive Bufferize pass.

Differential Revision: https://reviews.llvm.org/D124854
2022-05-04 17:19:44 +09:00
Matthias Springer 5f60c4825b [mlir][linalg][bufferize][NFC] Make init_tensor elimination a separate pre-processing pass
This commit decouples init_tensor elimination from the rest of the bufferization.

Differential Revision: https://reviews.llvm.org/D124853
2022-05-04 17:17:27 +09:00
Matthias Springer 37a1473524 [mlir][bufferize] Allow in-place bufferization for writes to init_tensors in loops
This commit relaxes the rules around ops that define a value but do not specify the tensor's contents. (The only such op at the moment is init_tensor.)

When such a tensor is written in a loop, it should not cause out-of-place bufferization.

Differential Revision: https://reviews.llvm.org/D124849
2022-05-04 16:43:43 +09:00
Marius Brehler 63aaf9a6e7 [mlir] Add missing CMake deps to mlir-pdll
Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D124851
2022-05-04 06:17:13 +00:00
Aart Bik 2617f2f708 [mlir][sparse] fix build issue with unused local under opt builds
Reviewed By: rdzhabarov

Differential Revision: https://reviews.llvm.org/D124883
2022-05-03 14:55:32 -07:00
Aart Bik 1abcdc677c [mlir][sparse] add missing types to from/to-MLIR conversion routines
This will enable our usual set of element types in external
environments, such as PyTACO support.

Reviewed By: bixia

Differential Revision: https://reviews.llvm.org/D124875
2022-05-03 14:36:37 -07:00
Stella Stamenova 5f14aee3bb [mlir] Fix Visual Studio warnings
There are only a couple of warnings when compiling with VS on Windows. This fixes the last remaining warnings so that we can enable LLVM_ENABLE_WERROR on the mlir windows bot.

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D124862
2022-05-03 14:12:15 -07:00
Jim Kitchen 2c33266084 [mlir][sparse] Add lowering for unary and binary ops
Adding lowering for Unary and Binary required several changes due to
their unique nature of containing custom code for different "regions"
of the sparse structure being operated on. Along with a Kind, a pointer
to the Operation is passed along to be merged once the lattice
structure is figured out.

The original operation is maintained, as it is required for subsequent
lattice decisions. However, sparse_tensor.binary has some branches
are considered as fully handled and therefore are marked with as
kBinaryBranch to distinguish them.

A unique aspect of the custom code is that sometimes the desired result
is no result at all -- i.e. a user wants overlapping sparse entries to
become empty in the output. The solution to this is to return an
uninitialized Value(), which is checked and handled elsewhere in the
code and results in nothing being written to the output tensor for that
case.

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D123057
2022-05-03 15:50:26 -05:00
Goran Flegar 672b908bca [mlir] Add sin & cos ops to complex dialect
Also adds conversions for those ops to math + arith.

Differential Revision: https://reviews.llvm.org/D124773
2022-05-03 19:36:12 +02:00
Min-Yih Hsu 857eb4a152 [mlir][LLVMIR] Add support for translating Switch instruction
Add support for translating llvm::SwitchInst.

Differential Revision: https://reviews.llvm.org/D124628
2022-05-03 09:41:40 -07:00
Hanhan Wang 919e459f1b [Linalg] Remove Optional from getStaticLoopRanges interface method.
It is very wrong if the ranges can't be infered. It's also checked in
verifyStructuredOpInterface, so we don't need the Optional return type.

Reviewed By: springerm

Differential Revision: https://reviews.llvm.org/D124596
2022-05-03 05:12:54 -07:00
Alex Zinenko 6c57b0debe [mlir] improve and test TransformState::Extension
Add the mechanism for TransformState extensions to update the mapping between
Transform IR values and Payload IR operations held by the state. The mechanism
is intentionally restrictive, similarly to how results of the transform op are
handled.

Introduce test ops that exercise a simple extension that maintains information
across the application of multiple transform ops.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D124778
2022-05-03 11:33:00 +02:00
Benjamin Kramer dccb73318a [mlir][MemRef] Return `0` for the canonical strided layout expr of a 0d memref
There can't be any strides, and the offset for the canonical expr is
always 0. Fixes #55229.

Differential Revision: https://reviews.llvm.org/D124795
2022-05-03 10:58:44 +02:00
Vitaly Buka 8611909572 [mlir] Create lbOperands before op.setLowerBound
To fix msan report like https://reviews.llvm.org/P8284

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D124575
2022-05-02 22:23:51 -07:00