This makes it easier to use as a utility function to query the
mappings, including the reverse.
This commit also drops some storage classes that aren't needed
for now.
Reviewed By: kuhar
Differential Revision: https://reviews.llvm.org/D131411
This commit moves MemRef memory space to SPIR-V storage class
conversion out of the main SPIR-V type converter. Now the mapping
should happen as a prelimiary step before performing the final
conversion to SPIR-V. Flows are expect to write their own memory
space mappings like the `MapMemRefStorageClassPass` to handle
memory space mappings according to their needs.
This is needed because SPIR-V is serving multiple client APIs,
including Vulkan and OpenCL. Different client APIs might want
to use different storage classes for buffers in a particular
memory space, e.g., `StorageBuffer` for Vulkan vs. `CrossWorkgroup`
for OpenCL when converting the default 0 memory space. Hardcoding
a specific mapping makes that hard. While it's possible to embed
selection logic further inside the main type converter, it will
make the main type converter even complicated. So it's better to
separate the concerns, as mapping the memory space is really
concretizing the meaning of those numeric memory spaces in the
particular context of SPIR-V lowering.
Reviewed By: kuhar
Differential Revision: https://reviews.llvm.org/D131410
* Avoid restricting the pass to to builtin module ops. The pass
should be able to run on any region ops.
* Avoid hardcoding func FuncOp when handling functions. Instead,
use the function op interface.
* Assigns the default mapping in the constructor. So for cases
where we are using the pass in a pipeline, we still have a
meaningful default.
Along the way, dropped uncessary unrealized conversion casts and
use full conversion. The pass should be able to convert all sorts
of ops; there is really no need to have such bridages.
Reviewed By: kuhar
Differential Revision: https://reviews.llvm.org/D131409
In addition to memref, accept ranked tensor as the base operand of vector.gather, similar to vector.trasnfer_read.
This will allow us to vectorize noncontiguous tensor.extract into vector.gather. Full discussion can be found here: https://github.com/iree-org/iree/issues/9198
Reviewed By: hanchung, dcaballe
Differential Revision: https://reviews.llvm.org/D130097
Previously we can only lower arith.index_cast with 1-D vectors to LLVM. This change added the support for n-D vectors.
Reviewed By: ftynse, hanchung
Differential Revision: https://reviews.llvm.org/D129907
Previously we are using IntegerAttr to back all SPIR-V enum
attributes. Therefore we all such attributes are showed like
IntegerAttr in IRs, which is barely readable and breaks
roundtripability of the IR. This commit changes to use
`EnumAttr` as the base directly so that we can have separate
attribute definitions and better IR printing.
Reviewed By: kuhar
Differential Revision: https://reviews.llvm.org/D131311
Resolve almost all clang tidy warnings in this file:
1. Clean up string constants.
2. Use consistent argument names across function declarations and definitions. Rename `state` - > `result`, which is consistent with the other dialects.
3. Remove misleading function parameter name comments (`argTypes`). This did not match the actual function argument (`bool enableNameShadowing`).
4. Simplify calls to `is_splat`.
Reviewed By: antiagainst
Differential Revision: https://reviews.llvm.org/D131297
now that `arith` dialect has maxf/minf use it instead of cmp/select.
Also refactor clamp helpers to make them simlper.
Reviewed By: rsuderman
Differential Revision: https://reviews.llvm.org/D131426
The test was using a missing prefix. Add the prefix and fix the naming.
Found by @csigg
Reviewed By: csigg
Differential Revision: https://reviews.llvm.org/D131428
Moved some parts from comments (not user facing) to the actual description
(user facing). Rephrased a bit as well.
Reviewed By: Peiming
Differential Revision: https://reviews.llvm.org/D131418
This further relaxes the requirement to allow aliased resources
to have different primitive types and some are scalars while the
other are vectors.
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D131207
Currently, dense tensors are initialized in Sparse Integration tests using
"buffer.tensor_alloc and scf.for" . This makes code harder to read and maintain.
This diff uses tensor.generate instead to initialize dense tensors.
Testing: Ran integration tests after building with -DLLVM_USE_SANITIZER=Address flag.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D131404
This commit updates all SPIR-V enum definitions to match the latest
specification (v1.6 revision 2). Along the way, fixed some issues
in `gen_spirv_dialect.py` and added a new script for refreshing
all op definitions for such cases.
Reviewed By: kuhar
Differential Revision: https://reviews.llvm.org/D131293
Adds an integer absolute value op to the math dialect.
When converting to LLVM, this op is lowered to the LLVM `abs` intrinsic.
When converting to SPIRV, this op is lowered to `spv.GL.SAbs`.
Depends on D131325
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D131327
IntegerPolyhedron::findIntegerLexmin currently does not return values of
the local ids, so when a test for sampling includes a set with locals, the
result of findIntegerLexmin should be checked using containsPointNoLocal,
not containsPoint.
Prior to this patch, the `add_tablegen()` macro in
llvm/cmake/modules/TableGen.cmake added the install rule only if
`project` matched `LLVM` or `MLIR`. This patch adds an optional
`DESTINATION` argument, which, if non-empty, decides whether (and where)
to install the tablegen tool, thus eliminating the need for
project-specific overrides. This patch also updates all other
invocations of the `add_tablegen()` macro.
Reviewed By: nikic
Differential Revision: https://reviews.llvm.org/D131282
This patch fixes:
llvm-project/mlir/include/mlir/IR/OpDefinition.h:1544:19: error: use
of bitwise '|' with boolean operands
[-Werror,-Wbitwise-instead-of-logical]
Now that C++17 is enabled in LLVM, a lot of the TODOs and patterns to emulate C++17 features can be eliminated.
The steps I have taken were essentially:
```
git grep C++17
git grep c++17
git grep "initializer_list<int>"
```
and address given comments and patterns.
Most of the changes boiled down to just using fold expressions rather than initializer_list.
While doing this I also discovered that Clang by default restricts the depth of fold expressions to 256 elements. I specifically hit this with `TestDialect` in `addOperations`. I opted to not replace it with fold expressions because of that but instead adding a comment documenting the issue.
If any other functions may be called with more than 256 elements in the future we might have to revert other parts as well.
I don't think this is a common occurence besides the `TestDialect` however. If need be, this could potentially be fixed via `mlir-tblgen` in the future.
Differential Revision: https://reviews.llvm.org/D131323
This has previously been done for `mlir-opt` and `mlir-reduce` and roughly the same approach has been done here.
The use case for having a separate library is that it is easier for downstream to make custom TableGen backends/executable that work on top of the utilities that are defined in `mlir/TableGen`.
The customization point here is the same one as for any upstream TableGen backends: One can add a new generator by simply creating a global instance of `mlir::GenRegistration`.
Differential Revision: https://reviews.llvm.org/D131112
Based on `spv.ISubBorrow` from D127909.
Also resolved some clang-tidy warnings.
Reviewed By: antiagainst, ThomasRaoux
Differential Revision: https://reviews.llvm.org/D131281
MemRef types now can carry an attribute to represent the memory
space. Still, upper layers in the compilation stack mostly use
nuemric values. They don't mean much (other than differentiating
separate memory domains) in MLIR's multi-level settings. Those
numeric memory space inside MemRef types need to be translated
into concrete SPIR-V storage classes during lowering to pin down
to concrete memory types.
Thus far we have been hardcoding an arbitrary mapping from memory
space to storage class for converting MemRef types. This works fine
for only targeting Vulkan; it falls apart if we want to target other
SPIR-V consumers like OpenCL, as different consumers might want
different storage classes for the buffer/variable of the same
lifetime. For example, StorageClass in Vulkan vs. CrossWorkgroup
in OpenCL.
So putting up a new pass to let the user to control how to map
MemRef memory spaces into SPIR-V storage classes. This provides
more flexibility and can address the awkwardness in the current
SPIR-V type converter. This pass should be the prelimiary step
towards lowering MemRef related types/ops into SPIR-V.
Reviewed By: mravishankar
Differential Revision: https://reviews.llvm.org/D130317
This patch adds constant folder for Atan2Op which only supports single and double precision floating-point.
Differential Revision: https://reviews.llvm.org/D131050
We were checking real-part twice, not real/imag-part.
The new test only passes after the bug fix.
Reviewed By: Peiming
Differential Revision: https://reviews.llvm.org/D131190
As a percaution, truncate memory addresses passed to kernels to 48 bits,
since bits 48-63 of the buffer descriptor are used for the stride field
and, on gfx10, to control swizzling.
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D131016
This reverts commit 07aaa35f74.
This breaks the Windows bot, and while the fix addressed the
`char`/`int8_t` case, it does not make sense for other cases like
`float`.
This can easily overflow and it is possible for these unsigned overflows to result in incorrect results.
For example, the two LCMs could be 641 and 6700417, which multiply to 2^32 + 1, which overflows to 1.
Unsigned overflows already occur in the existing tests.
Also, when switching to arbitrary-precision arithmetic, this results in a many
large integer multiplications resulting in a significant slowdown.
Reviewed By: Groverkss
Differential Revision: https://reviews.llvm.org/D131184
The `IR/AttributeTest.cpp` test fails to compile on Solaris:
/vol/llvm/src/llvm-project/local/mlir/unittests/IR/AttributeTest.cpp:223:36: error: no matching function for call to 'allocate'
AttrT::get(type, "resource", UnmanagedAsmResourceBlob::allocate(data));
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/vol/llvm/src/llvm-project/local/mlir/unittests/IR/AttributeTest.cpp:237:3: note: in instantiation of function template specialization 'checkNativeAccess<mlir::detail::DenseResourceElementsAttrBase<int8_t>, char>' requested here
checkNativeAccess<AttrT, T>(builder.getContext(), llvm::makeArrayRef(data),
^
/vol/llvm/src/llvm-project/local/mlir/unittests/IR/AttributeTest.cpp:258:3: note: in instantiation of function template specialization 'checkNativeIntAccess<mlir::detail::DenseResourceElementsAttrBase<int8_t>, char>' requested here
checkNativeIntAccess<DenseI8ResourceElementsAttr, int8_t>(builder, 8);
^
/vol/llvm/src/llvm-project/local/mlir/include/mlir/IR/AsmState.h:221:3: note: candidate template ignored: requirement '!std::is_same<char, char>::value' was not satisfied [with T = char]
allocate(ArrayRef<T> data, bool dataIsMutable = false) {
^
/vol/llvm/src/llvm-project/local/mlir/include/mlir/IR/AsmState.h:214:26: note: candidate function not viable: requires at least 2 arguments, but 1 was provided
static AsmResourceBlob allocate(ArrayRef<char> data, size_t align,
^
I suspect this happens because `char` is `signed` by default on Solaris.
Tested on `amd64-pc-solaris2.11` and `sparcv9-sun-solaris2.11`.
Differential Revision: https://reviews.llvm.org/D131148
This prepares patterns that sometimes are generated by the front-end
and would prohibit fusion of SDDMM flavored kernels.
Reviewed By: springerm
Differential Revision: https://reviews.llvm.org/D131126
The functions are effectively independent of the interface already, however, they take it as an argument for no reason.
The current state complicates reuse outside of MLIR.
Differential Revision: https://reviews.llvm.org/D131120
`SymbolTable::lookupSymbolIn` is an expensive operation and we do not want to do it twice
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D131145
Previously cf.br cf.cond_br and cf.switch always lowered to their LLVM
equivalents. These ops are all ops that take in some values of given
types and jump to other blocks with argument lists of the same types. If
the types are not the same, a verification failure will later occur. This led
to confusions, as everything works when func->llvm and cf->llvm lowering
both occur because func->llvm updates the blocks and argument lists
while cf->llvm updates the branching ops. Without func->llvm though,
there will potentially be a type mismatch.
This change now only lowers the CF ops if they will later pass
verification. This is possible because the parent op and its blocks will
be updated before the contained branching ops, so they can test their
new operand types against the types of the blocks they jump to.
Another plan was to have func->llvm only update the entry block
signature and to allow cf->llvm to update all other blocks, but this had
2 problems:
1. This would create a FuncOp lowering in cf->llvm lowering which is
awkward
2. This new pattern would only be applied if the containing FuncOp is
marked invalid. This is infeasible with the shared LLVM type
conversion/target infrastructure.
See previous discussions at
https://discourse.llvm.org/t/lowering-cf-to-llvm/63863 and
https://github.com/llvm/llvm-project/issues/55301
Differential Revision: https://reviews.llvm.org/D130971
This patch adds a DenseI1ArrayAttr to support arrays of i1. Importantly,
the implementation is as a simple `ArrayRef<bool>` instead of using bit
compression, which was problematic in DenseElementsAttr.
Reviewed By: rriddle
Differential Revision: https://reviews.llvm.org/D130957
This change separates computation of the actual parameters of the subset and
the materialization of subview/extract_slice. That way the users can still use
Linalg tiling logic even if they use different operations to materialize the
subsets.
Differential Revision: https://reviews.llvm.org/D131053
Using if (TARGET ${LLVM_NATIVE_ARCH}) only works if MLIR is built
together with LLVM, but not for standalone builds of MLIR. The
correct way to check this is
if (${LLVM_NATIVE_ARCH} IN_LIST LLVM_TARGETS_TO_BUILD), as the
LLVM build system exports LLVM_TARGETS_TO_BUILD.
To avoid repeating the same check many times, add a
MLIR_ENABLE_EXECUTION_ENGINE variable.
Differential Revision: https://reviews.llvm.org/D131071
It seems only the default implementation is ever used, so it doesn't seem
necessary to include this method in the interface.
Differential Revision: https://reviews.llvm.org/D130986
The reconciliation pass has been improved to introduce the support for chains of casts, thus not limiting anymore the reconciliation to just consider pairs of unrealized casts.
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D130711
As reported in Issue #56850, mlir/lib/Analysis/Presburger/Utils.cpp doesn't
compile on Solaris 11.4/SPARC with the bundled GCC 11, as seen when testing
LLVM 15.0.0 rc1:
/var/llvm/reltest/llvm-15.0.0-rc1/rc1/llvm-project/mlir/include/mlir/Analysis/Presburger/MPInt.h:260:47:
error: inlining failed in call to ‘always_inline’ ‘int64_t
mlir::presburger::int64FromMPInt(const mlir::presburger::MPInt&)’: indirect
function call with a yet undetermined callee
This patch hacks around this and allowed the build to finish.
Tested on `sparcv9-sun-solaris2.11`.
Differential Revision: https://reviews.llvm.org/D131060
This patch adds constant folder for AtanOp which only supports single and double precision floating-point.
Differential Revision: https://reviews.llvm.org/D130983
llvm::Optional is in the process of switching to the
std::optional-like interface with has_value/value as opposed to
hasValue/getValue.
This patch adds has_value and value to enable the same transition.
Differential Revision: https://reviews.llvm.org/D130819
In some cases the recursion will grow the `visited` hash table and
invalidate the cached iterator.
(caught with ASAN)
Differential Revision: https://reviews.llvm.org/D131027
In the ROCm runtime (and probably CUDA as well), all kernel arguments
are aligned. Therefore, enable using bare pointers for memref
arguments to kernels when these memrefs have static shape and a
trivial layout.
This is a substantial optimization to launching kernels that use
memrefs with known, static sizes, since it causes the kernel launch
packet to no longer include information already known to the kernel,
which can enable packing the kernel launch arguments into launch
packets instead of having to allocate an entire separate structure to
hold unneeded memref information.
Reviewed By: ftynse
Differential Revision: https://reviews.llvm.org/D130716
This rewriting was no longer functional after recent migration to one shot
bufferization. However, this revision makes it work again, with a CHECK test
to ensure fusion happens. Note that functionality is tested by several
integration tests.
Reviewed By: Peiming
Differential Revision: https://reviews.llvm.org/D130996
The generic allocation and deallocation instructions, which are optionally used during the MemRef -> LLVM conversion, should have a name that is specifically bound to their origin, that is the conversion pass itself.
Reviewed By: silvas
Differential Revision: https://reviews.llvm.org/D130588
This patch adds constant folder for TanhOp which only supports single and double precision floating-point.
Differential Revision: https://reviews.llvm.org/D130960
When the mlir-tblgen tool is set up, the `MLIR_TABLEGEN_EXE` variable
is set, which either points to the mlir-tblgen tool built in the current
cmake build, or points to one built in a nested cmake build (if cross
conpiling, or if building with e.g. `LLVM_OPTIMIZED_TABLEGEN`.
The `MLIR_TABLEGEN_EXE` variable is only set within the scope of the
mlir/CMakeLists.txt file, so it's unavailable in sibling level projects
such as flang.
Set the `MLIR_TABLEGEN_EXE` and the `MLIR_TABLEGEN_TARGET` variables
as global, so that flang can use them properly without guessing.
Differential Revision: https://reviews.llvm.org/D130350
This patch adds constant folder for TanOp which only supports single and double precision floating-point.
Differential Revision: https://reviews.llvm.org/D130873
Previously, DenseArrayAttr used VectorType for its shaped type.
VectorType is problematic for arrays because it doesn't support zero
dimensions, meaning that an empty array would have `vector<i32>` as its
type. ElementsAttr would think that an empty dense array is size 1, not
0. This patch switches over to TensorType, which does support zero
dimensions.
Fixes#56860
Reviewed By: mehdi_amini
Differential Revision: https://reviews.llvm.org/D130921
Adds optional attribute to support tensor cores on F32 datatype by lowering to `mma.sync` with TF32 operands. Since, TF32 is not a native datatype in LLVM we are adding `tf32Enabled` as an attribute to allow the IR to be aware of `MmaSyncOp` datatype. Additionally, this patch adds placeholders for nvgpu-to-nvgpu transformation targeting higher precision tf32x3.
For mma.sync on f32 input using tensor cores there are two possibilites:
(a) tf32 (1 `mma.sync` per warp-level matrix-multiply-accumulate)
(b) tf32x3 (3 `mma.sync` per warp-level matrix-multiply-accumulate)
Typically, tf32 tensor core acceleration comes at a cost of accuracy from missing precision bits. While f32 has 23 precision bits, tf32 has only 10 precision bits. tf32x3 aims to recover the precision bits by splitting each operand into two tf32 values and issue three `mma.sync` tensor core operations.
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D130294
This attribute is technical debt from the early stages of MLIR, before
ElementsAttr was an interface and when it was more difficult for
dialects to define their own types of attributes. At present it isn't
used at all in tree (aside from being convenient for eliding other
ElementsAttr), and has had little to no evolution in the past three years.
Differential Revision: https://reviews.llvm.org/D129917
This attributes is intended cover the current set of use cases that abuse
DenseElementsAttr, e.g. when the data is large. Using resources for large
data is one of the major reasons why they were added; e.g. they can be
deallocated mid-compilation, they support a wide variety of data origins
(e.g, heap allocated, mmap'd, etc.), they can support mutation, etc.
I considered at length not having a builtin variant of this, and instead
having multiple versions of this attribute for dialects that are interested,
but they all boiled down to the exact same attribute definition. Given the
generality of this attribute, it feels more aligned to keep it next to DenseArrayAttr
(given that DenseArrayAttr covers the "small" case, and DenseResourcesElementsAttr
covers the "large" case). The underlying infra used to build this attribute is
general, and having a builtin attribute doesn't preclude users from defining
their own when it makes sense (they can even share a blob manager with the
builtin dialect to avoid data duplication).
Differential Revision: https://reviews.llvm.org/D130022
The DialectResourceBlobManager class provides functionality for managing resource blobs
in a generic, dialect-agnostic fashion. In addition to this class, a dialect interface and custom
resource handle are provided to simplify referencing and interacting with the manager. These
classes intend to simplify the work required for dialects that want to manage resource blobs
during compilation, such as for large elements attrs. The old manager for the resource example
in the test dialect has been updated to use this, which provides and cleaner and more consistent API.
This commit also adds new HeapAsmResourceBlob and ImmortalAsmResourceBlob to simplify
creating resource blobs in common scenarios.
Differential Revision: https://reviews.llvm.org/D130021
Attempting to apply the range analysis to real code revealed that
trunci wasn't correctly handling the case where truncation would
create wider ranges - for example, if we truncate [255, 257] : i16 to
i8, the result can be 255, 0, or 1, which isn't a contiguous range of
values.
The previous implementation would naively map this to [255, 1], which
would cause issues with unsigned ranges and unification.
Reviewed By: Mogball
Differential Revision: https://reviews.llvm.org/D130501
This is the follow up on https://reviews.llvm.org/D130730 which goes through upstream code and removes creating constant values in favour of using the constant indices in GEP directly. This leads to less and more readable code and more compact IR as well.
Differential Revision: https://reviews.llvm.org/D130731
Scope of changes:
1) Added new function to generate loop versioning
2) Added support for if clause to applySimd function
2) Added tests which confirm that lowering is successful
If ifCond is specified, then collapsed loop is duplicated and if branch
is added. Duplicated loop is executed if simd ifCond is evaluated to false.
Reviewed By: Meinersbur
Differential Revision: https://reviews.llvm.org/D129368
Signed-off-by: Dominik Adamski <dominik.adamski@amd.com>
The "optimization" would replace the AffineMap for an empty shape with a 0 to represent its indexing (stride * dimension) logic. Meanwhile other pieces of core logic (such as getStridesAndOffset and makeStridedLinearLayoutMap) require strides for all dimensions to ensure no aliasing can occur which would occur if the shape was not empty. For now, this optimization is removed as different pieces of core types disagree on this, so the optimization should be caller supplied or should be consistent throughout the infrastructure.
Differential Revision: https://reviews.llvm.org/D130772
e179532284 removed the Type field from attributes and
arith::ConstantOp argument is now a TypedAttrInterface which isn't
supported by the python generator.
This patch temporarily restore the functionality for arith.constant but
won't generalize: we need to work on the generator instead.
Differential Revision: https://reviews.llvm.org/D130878
This patch adds constant folder for ExpM1Op which only supports single and double precision floating-point.
Differential Revision: https://reviews.llvm.org/D130567
This patch removes the `type` field from `Attribute` along with the
`Attribute::getType` accessor.
Going forward, this means that attributes in MLIR will no longer have
types as a first-class concept. This patch lays the groundwork to
incrementally remove or refactor code that relies on generic attributes
being typed. The immediate impact will be on attributes that rely on
`Attribute` containing a type, such as `IntegerAttr`,
`DenseElementsAttr`, and `ml_program::ExternAttr`, which will now need
to define a type parameter on their storage classes. This will save
memory as all other attribute kinds will no longer contain a type.
Moreover, it will not be possible to generically query the type of an
attribute directly. This patch provides an attribute interface
`TypedAttr` that implements only one method, `getType`, which can be
used to generically query the types of attributes that implement the
interface. This interface can be used to retain the concept of a "typed
attribute". The ODS-generated accessor for a `type` parameter
automatically implements this method.
Next steps will be to refactor the assembly formats of certain operations
that rely on `parseAttribute(type)` and `printAttributeWithoutType` to
remove special handling of type elision until `type` can be removed from
the dialect parsing hook entirely; and incrementally remove uses of
`TypedAttr`.
Reviewed By: lattner, rriddle, jpienaar
Differential Revision: https://reviews.llvm.org/D130092
When dead-code analysis is run at the scope of a function, call ops to
other functions at the same level were being marked as unreachable,
since the analysis optimistically assumes the call op to have no known
predecessors and that all predecessors are known, but the callee would
never get visited.
This patch fixes the bug by checking if a referenced function is above
the top-level op of the analysis, and is thus considered an external
callable.
Fixes#56830
Reviewed By: zero9178
Differential Revision: https://reviews.llvm.org/D130829
Aart Bik