maxjhandsome
/
llvm-project
forked from OSchip/llvm-project
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity
llvm-project/mlir/python/CMakeLists.txt

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

408 lines
12 KiB
CMake
Raw Normal View History

Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
include(AddMLIRPython)
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-29 04:04:17 +08:00
################################################################################
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
# Structural groupings.
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-29 04:04:17 +08:00
################################################################################
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
declare_mlir_python_sources(MLIRPythonSources)
declare_mlir_python_sources(MLIRPythonSources.Dialects
ADD_TO_PARENT MLIRPythonSources)
################################################################################
# Pure python sources and generated code
################################################################################
declare_mlir_python_sources(MLIRPythonSources.Core
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
ADD_TO_PARENT MLIRPythonSources
SOURCES
_mlir_libs/__init__.py
ir.py
passmanager.py
dialects/_ods_common.py
[mlir][python] Add pyi stub files to enable auto completion. There is no completely automated facility for generating stubs that are both accurate and comprehensive for native modules. After some experimentation, I found that MyPy's stubgen does the best at generating correct stubs with a few caveats that are relatively easy to fix: * Some types resolve to cross module symbols incorrectly. * staticmethod and classmethod signatures seem to always be completely generic and need to be manually provided. * It does not generate an __all__ which, from testing, causes namespace pollution to be visible to IDE code completion. As a first step, I did the following: * Ran `stubgen` for `_mlir.ir`, `_mlir.passmanager`, and `_mlirExecutionEngine`. * Manually looked for all instances where unnamed arguments were being emitted (i.e. as 'arg0', etc) and updated the C++ side to include names (and re-ran stubgen to get a good initial state). * Made/noted a few structural changes to each `pyi` file to make it minimally functional. * Added the `pyi` files to the CMake rules so they are installed and visible. To test, I added a `.env` file to the root of the project with `PYTHONPATH=...` set as per instructions. Then reload the developer window (in VsCode) and verify that completion works for various changes to test cases. There are still a number of overly generic signatures, but I want to check in this low-touch baseline before iterating on more ambiguous changes. This is already a big improvement. Differential Revision: https://reviews.llvm.org/D114679
2021-11-29 06:08:06 +08:00
# The main _mlir module has submodules: include stubs from each.
_mlir_libs/_mlir/__init__.pyi
_mlir_libs/_mlir/ir.pyi
_mlir_libs/_mlir/passmanager.pyi
[mlir] Introduce Python bindings for the quantization dialect So far, only the custom dialect types are exposed. The build and packaging is same as for Linalg and SparseTensor, and in need of refactoring that is beyond the scope of this patch. Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D116605
2022-01-05 00:43:44 +08:00
# TODO: this should be split out into a separate library.
_mlir_libs/_mlir/dialects/quant.pyi
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-29 04:04:17 +08:00
)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
declare_mlir_python_sources(MLIRPythonSources.ExecutionEngine
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
ADD_TO_PARENT MLIRPythonSources
SOURCES
execution_engine.py
[mlir][python] Add pyi stub files to enable auto completion. There is no completely automated facility for generating stubs that are both accurate and comprehensive for native modules. After some experimentation, I found that MyPy's stubgen does the best at generating correct stubs with a few caveats that are relatively easy to fix: * Some types resolve to cross module symbols incorrectly. * staticmethod and classmethod signatures seem to always be completely generic and need to be manually provided. * It does not generate an __all__ which, from testing, causes namespace pollution to be visible to IDE code completion. As a first step, I did the following: * Ran `stubgen` for `_mlir.ir`, `_mlir.passmanager`, and `_mlirExecutionEngine`. * Manually looked for all instances where unnamed arguments were being emitted (i.e. as 'arg0', etc) and updated the C++ side to include names (and re-ran stubgen to get a good initial state). * Made/noted a few structural changes to each `pyi` file to make it minimally functional. * Added the `pyi` files to the CMake rules so they are installed and visible. To test, I added a `.env` file to the root of the project with `PYTHONPATH=...` set as per instructions. Then reload the developer window (in VsCode) and verify that completion works for various changes to test cases. There are still a number of overly generic signatures, but I want to check in this low-touch baseline before iterating on more ambiguous changes. This is already a big improvement. Differential Revision: https://reviews.llvm.org/D114679
2021-11-29 06:08:06 +08:00
_mlir_libs/_mlirExecutionEngine.pyi
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
SOURCES_GLOB
runtime/*.py
)
declare_mlir_python_sources(MLIRPythonSources.Passes
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
ADD_TO_PARENT MLIRPythonSources
SOURCES_GLOB
all_passes_registration/*.py
conversions/*.py
transforms/*.py
)
[MLIR][python] Export CAPI headers. * Adds source targets (not included in the full set that downstreams use by default) to bundle mlir-c/ headers into the mlir/_mlir_libs/include directory. * Adds a minimal entry point to get include and library directories. * Used by npcomp to export a full CAPI (which is then used by the Torch extension to link npcomp). Reviewed By: mikeurbach Differential Revision: https://reviews.llvm.org/D107090
2021-07-30 03:06:22 +08:00
declare_mlir_python_sources(MLIRPythonCAPIHeaderSources
ROOT_DIR "${MLIR_SOURCE_DIR}/include"
SOURCES_GLOB "mlir-c/*.h"
DEST_PREFIX "_mlir_libs/include"
)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
################################################################################
# Dialect bindings
################################################################################
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/AsyncOps.td
SOURCES_GLOB dialects/async_dialect/*.py
DIALECT_NAME async_dialect)
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/BuiltinOps.td
SOURCES
dialects/builtin.py
dialects/_builtin_ops_ext.py
DIALECT_NAME builtin)
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/GPUOps.td
SOURCES_GLOB dialects/gpu/*.py
DIALECT_NAME gpu)
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/LinalgOps.td
SOURCES
dialects/_linalg_ops_ext.py
SOURCES_GLOB
dialects/linalg/*.py
DIALECT_NAME linalg
DEPENDS LinalgOdsGen)
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/MathOps.td
SOURCES dialects/math.py
DIALECT_NAME math)
[MLIR] Replace std ops with arith dialect ops Precursor: https://reviews.llvm.org/D110200 Removed redundant ops from the standard dialect that were moved to the `arith` or `math` dialects. Renamed all instances of operations in the codebase and in tests. Reviewed By: rriddle, jpienaar Differential Revision: https://reviews.llvm.org/D110797
2021-10-13 07:14:57 +08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/ArithmeticOps.td
SOURCES
dialects/arith.py
dialects/_arith_ops_ext.py
DIALECT_NAME arith)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/MemRefOps.td
[mlir] fix python bindings cmake
2021-10-13 23:29:19 +08:00
SOURCES
dialects/memref.py
dialects/_memref_ops_ext.py
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
DIALECT_NAME memref)
[mlir] Introduce Python bindings for the quantization dialect So far, only the custom dialect types are exposed. The build and packaging is same as for Linalg and SparseTensor, and in need of refactoring that is beyond the scope of this patch. Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D116605
2022-01-05 00:43:44 +08:00
declare_mlir_python_sources(
MLIRPythonSources.Dialects.quant
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
SOURCES
dialects/quant.py)
[mlir][python] provide bindings for the SCF dialect This is an important core dialect that has not been exposed previously. Set up the default bindings generation and provide a nicer wrapper for the `for` loop with access to the loop configuration and body. Depends On D110758 Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D110759
2021-09-30 03:42:53 +08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/SCFOps.td
SOURCES
dialects/scf.py
dialects/_scf_ops_ext.py
DIALECT_NAME scf)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/ShapeOps.td
SOURCES dialects/shape.py
DIALECT_NAME shape)
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
[mlir][python] provide bindings for ops from the sparse_tensor dialect Previously, the dialect was exposed for linking and pass management purposes, but we did not generate op classes for it. Generate them. Reviewed By: nicolasvasilache Differential Revision: https://reviews.llvm.org/D110819
2021-09-30 21:09:30 +08:00
TD_FILE dialects/SparseTensorOps.td
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
SOURCES dialects/sparse_tensor.py
DIALECT_NAME sparse_tensor)
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/StandardOps.td
[mlir][python] Provide more convenient wrappers for std.ConstantOp Constructing a ConstantOp using the default-generated API is verbose and requires to specify the constant type twice: for the result type of the operation and for the type of the attribute. It also requires to explicitly construct the attribute. Provide custom constructors that take the type once and accept a raw value instead of the attribute. This requires dynamic dispatch based on type in the constructor. Also provide the corresponding accessors to raw values. In addition, provide a "refinement" class ConstantIndexOp similar to what exists in C++. Unlike other "op view" Python classes, operations cannot be automatically downcasted to this class since it does not correspond to a specific operation name. It only exists to simplify construction of the operation. Depends On D110946 Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D110947
2021-10-04 17:38:53 +08:00
SOURCES
dialects/std.py
dialects/_std_ops_ext.py
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
DIALECT_NAME std)
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/TensorOps.td
SOURCES dialects/tensor.py
DIALECT_NAME tensor)
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/TosaOps.td
SOURCES dialects/tosa.py
DIALECT_NAME tosa)
declare_mlir_dialect_python_bindings(
ADD_TO_PARENT MLIRPythonSources.Dialects
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
TD_FILE dialects/VectorOps.td
SOURCES dialects/vector.py
DIALECT_NAME vector)
################################################################################
# Python extensions.
# The sources for these are all in lib/Bindings/Python, but since they have to
# be rebuilt for each package and integrate with the source setup here, we
# just reference them here instead of having ordered, cross package target
# dependencies.
################################################################################
set(PYTHON_SOURCE_DIR "${MLIR_SOURCE_DIR}/lib/Bindings/Python")
declare_mlir_python_extension(MLIRPythonExtension.Core
MODULE_NAME _mlir
ADD_TO_PARENT MLIRPythonSources.Core
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
ROOT_DIR "${PYTHON_SOURCE_DIR}"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
DialectLinalg.cpp # TODO: Break this out.
DialectSparseTensor.cpp # TODO: Break this out.
[mlir] Introduce Python bindings for the quantization dialect So far, only the custom dialect types are exposed. The build and packaging is same as for Linalg and SparseTensor, and in need of refactoring that is beyond the scope of this patch. Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D116605
2022-01-05 00:43:44 +08:00
DialectQuant.cpp # TODO: Break this out.
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
MainModule.cpp
IRAffine.cpp
IRAttributes.cpp
IRCore.cpp
IRInterfaces.cpp
IRModule.cpp
IRTypes.cpp
PybindUtils.cpp
Pass.cpp
# Headers must be included explicitly so they are installed.
Dialects.h
Globals.h
IRModule.h
Pass.h
PybindUtils.h
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIDebug
MLIRCAPIIR
[mlir] support interfaces in Python bindings Introduce the initial support for operation interfaces in C API and Python bindings. Interfaces are a key component of MLIR's extensibility and should be available in bindings to make use of full potential of MLIR. This initial implementation exposes InferTypeOpInterface all the way to the Python bindings since it can be later used to simplify the operation construction methods by inferring their return types instead of requiring the user to do so. The general infrastructure for binding interfaces is defined and InferTypeOpInterface can be used as an example for binding other interfaces. Reviewed By: gysit Differential Revision: https://reviews.llvm.org/D111656
2021-10-14 23:18:28 +08:00
MLIRCAPIInterfaces
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
MLIRCAPIRegistration # TODO: See about dis-aggregating
# Dialects
MLIRCAPILinalg # TODO: Remove when above is removed.
MLIRCAPISparseTensor # TODO: Remove when above is removed.
MLIRCAPIStandard
[mlir] Introduce Python bindings for the quantization dialect So far, only the custom dialect types are exposed. The build and packaging is same as for Linalg and SparseTensor, and in need of refactoring that is beyond the scope of this patch. Reviewed By: stellaraccident Differential Revision: https://reviews.llvm.org/D116605
2022-01-05 00:43:44 +08:00
MLIRCAPIQuant # TODO: Remove when above is removed.
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
)
declare_mlir_python_extension(MLIRPythonExtension.AllPassesRegistration
MODULE_NAME _mlirAllPassesRegistration
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
ROOT_DIR "${PYTHON_SOURCE_DIR}"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
AllPassesRegistration.cpp
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIConversion
MLIRCAPITransforms
)
declare_mlir_python_extension(MLIRPythonExtension.AsyncDialectPasses
MODULE_NAME _mlirAsyncPasses
ADD_TO_PARENT MLIRPythonSources.Dialects.async_dialect
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
ROOT_DIR "${PYTHON_SOURCE_DIR}"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
AsyncPasses.cpp
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIAsync
)
declare_mlir_python_extension(MLIRPythonExtension.Conversions
MODULE_NAME _mlirConversions
ADD_TO_PARENT MLIRPythonSources.Passes
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
ROOT_DIR "${PYTHON_SOURCE_DIR}"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
Conversions/Conversions.cpp
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIConversion
)
Break apart the MLIR ExecutionEngine from core python module. * For python projects that don't need JIT/ExecutionEngine, cuts the number of files to compile roughly in half (with similar reduction in end binary size). Differential Revision: https://reviews.llvm.org/D106992
2021-07-29 04:00:02 +08:00
declare_mlir_python_extension(MLIRPythonExtension.ExecutionEngine
MODULE_NAME _mlirExecutionEngine
ADD_TO_PARENT MLIRPythonSources.ExecutionEngine
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
ROOT_DIR "${PYTHON_SOURCE_DIR}"
Break apart the MLIR ExecutionEngine from core python module. * For python projects that don't need JIT/ExecutionEngine, cuts the number of files to compile roughly in half (with similar reduction in end binary size). Differential Revision: https://reviews.llvm.org/D106992
2021-07-29 04:00:02 +08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
ExecutionEngineModule.cpp
Break apart the MLIR ExecutionEngine from core python module. * For python projects that don't need JIT/ExecutionEngine, cuts the number of files to compile roughly in half (with similar reduction in end binary size). Differential Revision: https://reviews.llvm.org/D106992
2021-07-29 04:00:02 +08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
[MLIR] NFC. Rename MLIR CAPI ExecutionEngine target for consistency Rename MLIR CAPI ExecutionEngine target for consistency: MLIRCEXECUTIONENGINE -> MLIRCAPIExecutionEngine in line with other targets. Differential Revision: https://reviews.llvm.org/D114596
2021-11-25 23:08:32 +08:00
MLIRCAPIExecutionEngine
Break apart the MLIR ExecutionEngine from core python module. * For python projects that don't need JIT/ExecutionEngine, cuts the number of files to compile roughly in half (with similar reduction in end binary size). Differential Revision: https://reviews.llvm.org/D106992
2021-07-29 04:00:02 +08:00
)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
declare_mlir_python_extension(MLIRPythonExtension.GPUDialectPasses
MODULE_NAME _mlirGPUPasses
ADD_TO_PARENT MLIRPythonSources.Dialects.gpu
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
ROOT_DIR "${PYTHON_SOURCE_DIR}"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
GPUPasses.cpp
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIGPU
)
declare_mlir_python_extension(MLIRPythonExtension.LinalgPasses
MODULE_NAME _mlirLinalgPasses
ADD_TO_PARENT MLIRPythonSources.Dialects.linalg
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
ROOT_DIR "${PYTHON_SOURCE_DIR}"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
LinalgPasses.cpp
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPILinalg
)
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-29 04:04:17 +08:00
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
declare_mlir_python_extension(MLIRPythonExtension.SparseTensorDialectPasses
MODULE_NAME _mlirSparseTensorPasses
ADD_TO_PARENT MLIRPythonSources.Dialects.sparse_tensor
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
ROOT_DIR "${PYTHON_SOURCE_DIR}"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
SparseTensorPasses.cpp
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPISparseTensor
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-29 04:04:17 +08:00
)
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
declare_mlir_python_extension(MLIRPythonExtension.Transforms
MODULE_NAME _mlirTransforms
ADD_TO_PARENT MLIRPythonSources.Passes
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
ROOT_DIR "${PYTHON_SOURCE_DIR}"
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
Transforms/Transforms.cpp
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPITransforms
)
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-29 04:04:17 +08:00
[mlir][python] Segment MLIR Python test dialect to avoid testonly dependency. With https://reviews.llvm.org/rG14c9207063bb00823a5126131e50c93f6e288bd3, the build is broken with -DMLIR_INCLUDE_TESTS=OFF. This patch fixes the build and we may want to do a better fix to the layering in a followup. Differential Revision: https://reviews.llvm.org/D112560
2021-10-27 01:14:50 +08:00
# TODO: Figure out how to put this in the test tree.
# This should not be included in the main Python extension. However,
[mlir] support interfaces in Python bindings Introduce the initial support for operation interfaces in C API and Python bindings. Interfaces are a key component of MLIR's extensibility and should be available in bindings to make use of full potential of MLIR. This initial implementation exposes InferTypeOpInterface all the way to the Python bindings since it can be later used to simplify the operation construction methods by inferring their return types instead of requiring the user to do so. The general infrastructure for binding interfaces is defined and InferTypeOpInterface can be used as an example for binding other interfaces. Reviewed By: gysit Differential Revision: https://reviews.llvm.org/D111656
2021-10-14 23:18:28 +08:00
# putting it into MLIRPythonTestSources along with the dialect declaration
# above confuses Python module loader when running under lit.
[mlir][python] Segment MLIR Python test dialect to avoid testonly dependency. With https://reviews.llvm.org/rG14c9207063bb00823a5126131e50c93f6e288bd3, the build is broken with -DMLIR_INCLUDE_TESTS=OFF. This patch fixes the build and we may want to do a better fix to the layering in a followup. Differential Revision: https://reviews.llvm.org/D112560
2021-10-27 01:14:50 +08:00
set(_ADDL_TEST_SOURCES)
if(MLIR_INCLUDE_TESTS)
set(_ADDL_TEST_SOURCES MLIRPythonTestSources)
declare_mlir_python_sources(MLIRPythonTestSources)
declare_mlir_python_sources(MLIRPythonTestSources.Dialects
ADD_TO_PARENT MLIRPythonTestSources)
# TODO: this uses a tablegen file from the test directory and should be
# decoupled from here.
declare_mlir_python_sources(
MLIRPythonTestSources.Dialects.PythonTest
ROOT_DIR "${CMAKE_CURRENT_SOURCE_DIR}/mlir"
ADD_TO_PARENT MLIRPythonTestSources.Dialects
SOURCES dialects/python_test.py)
set(LLVM_TARGET_DEFINITIONS
"${MLIR_MAIN_SRC_DIR}/test/python/python_test_ops.td")
mlir_tablegen(
"dialects/_python_test_ops_gen.py"
-gen-python-op-bindings
-bind-dialect=python_test)
add_public_tablegen_target(PythonTestDialectPyIncGen)
declare_mlir_python_sources(
MLIRPythonTestSources.Dialects.PythonTest.ops_gen
ROOT_DIR "${CMAKE_CURRENT_BINARY_DIR}"
ADD_TO_PARENT MLIRPythonTestSources.Dialects.PythonTest
SOURCES "dialects/_python_test_ops_gen.py")
declare_mlir_python_extension(MLIRPythonTestSources.PythonTestExtension
MODULE_NAME _mlirPythonTest
ADD_TO_PARENT MLIRPythonTestSources.Dialects
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
ROOT_DIR "${MLIR_SOURCE_DIR}/test/python/lib"
[mlir][python] Segment MLIR Python test dialect to avoid testonly dependency. With https://reviews.llvm.org/rG14c9207063bb00823a5126131e50c93f6e288bd3, the build is broken with -DMLIR_INCLUDE_TESTS=OFF. This patch fixes the build and we may want to do a better fix to the layering in a followup. Differential Revision: https://reviews.llvm.org/D112560
2021-10-27 01:14:50 +08:00
SOURCES
Re-apply "[mlir] Allow out-of-tree python building from installed MLIR." Re-applies D111513: * Adds a full-fledged Python example dialect and tests to the Standalone example (need to do a bit of tweaking in the top level CMake and lit tests to adapt better to if not building with Python enabled). * Rips out remnants of custom extension building in favor of pybind11_add_module which does the right thing. * Makes python and extension sources installable (outputs to src/python/${name} in the install tree): Both Python and C++ extension sources get installed as downstreams need all of this in order to build a derived version of the API. * Exports sources targets (with our properties that make everything work) by converting them to INTERFACE libraries (which have export support), as recommended for the forseeable future by CMake devs. Renames custom properties to start with lower-case letter, as also recommended/required (groan). * Adds a ROOT_DIR argument to declare_mlir_python_extension since now all C++ sources for an extension must be under the same directory (to line up at install time). * Downstreams will need to adapt by: * Remove absolute paths from any SOURCES for declare_mlir_python_extension (I believe all downstreams are just using ${CMAKE_CURRENT_SOURCE_DIR} here, which can just be ommitted). May need to set ROOT_DIR if not relative to the current source directory. * To allow further downstreams to install/build, will need to make sure that all C++ extension headers are also listed under SOURCES for declare_mlir_python_extension. This reverts commit 1a6c26d1f52999edbfbf6a978ae3f0e6759ea755. Reviewed By: stephenneuendorffer Differential Revision: https://reviews.llvm.org/D113732
2021-11-15 06:44:25 +08:00
PythonTestModule.cpp
[mlir][python] Segment MLIR Python test dialect to avoid testonly dependency. With https://reviews.llvm.org/rG14c9207063bb00823a5126131e50c93f6e288bd3, the build is broken with -DMLIR_INCLUDE_TESTS=OFF. This patch fixes the build and we may want to do a better fix to the layering in a followup. Differential Revision: https://reviews.llvm.org/D112560
2021-10-27 01:14:50 +08:00
PRIVATE_LINK_LIBS
LLVMSupport
EMBED_CAPI_LINK_LIBS
MLIRCAPIPythonTestDialect
)
endif()
[mlir] support interfaces in Python bindings Introduce the initial support for operation interfaces in C API and Python bindings. Interfaces are a key component of MLIR's extensibility and should be available in bindings to make use of full potential of MLIR. This initial implementation exposes InferTypeOpInterface all the way to the Python bindings since it can be later used to simplify the operation construction methods by inferring their return types instead of requiring the user to do so. The general infrastructure for binding interfaces is defined and InferTypeOpInterface can be used as an example for binding other interfaces. Reviewed By: gysit Differential Revision: https://reviews.llvm.org/D111656
2021-10-14 23:18:28 +08:00
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-29 04:04:17 +08:00
################################################################################
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
# Common CAPI dependency DSO.
# All python extensions must link through one DSO which exports the CAPI, and
# this must have a globally unique name amongst all embeddors of the python
# library since it will effectively have global scope.
#
# The presence of this aggregate library is part of the long term plan, but its
# use needs to be made more flexible.
#
# TODO: Upgrade to the aggregate utility in https://reviews.llvm.org/D106419
# once ready.
Move MLIR python sources to mlir/python. * NFC but has some fixes for CMake glitches discovered along the way (things not cleaning properly, co-mingled depends). * Includes previously unsubmitted fix in D98681 and a TODO to fix it more appropriately in a smaller followup. Differential Revision: https://reviews.llvm.org/D101493
2021-04-29 04:04:17 +08:00
################################################################################
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
add_mlir_python_common_capi_library(MLIRPythonCAPI
INSTALL_COMPONENT MLIRPythonModules
INSTALL_DESTINATION python_packages/mlir_core/mlir/_mlir_libs
OUTPUT_DIRECTORY "${MLIR_BINARY_DIR}/python_packages/mlir_core/mlir/_mlir_libs"
RELATIVE_INSTALL_ROOT "../../../.."
DECLARED_SOURCES
MLIRPythonSources
MLIRPythonExtension.AllPassesRegistration
[mlir][python] Segment MLIR Python test dialect to avoid testonly dependency. With https://reviews.llvm.org/rG14c9207063bb00823a5126131e50c93f6e288bd3, the build is broken with -DMLIR_INCLUDE_TESTS=OFF. This patch fixes the build and we may want to do a better fix to the layering in a followup. Differential Revision: https://reviews.llvm.org/D112560
2021-10-27 01:14:50 +08:00
${_ADDL_TEST_SOURCES}
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
)
################################################################################
# The fully assembled package of modules.
# This must come last.
################################################################################
add_mlir_python_modules(MLIRPythonModules
ROOT_PREFIX "${MLIR_BINARY_DIR}/python_packages/mlir_core/mlir"
INSTALL_PREFIX "python_packages/mlir_core/mlir"
DECLARED_SOURCES
MLIRPythonSources
MLIRPythonExtension.AllPassesRegistration
[MLIR][python] Export CAPI headers. * Adds source targets (not included in the full set that downstreams use by default) to bundle mlir-c/ headers into the mlir/_mlir_libs/include directory. * Adds a minimal entry point to get include and library directories. * Used by npcomp to export a full CAPI (which is then used by the Torch extension to link npcomp). Reviewed By: mikeurbach Differential Revision: https://reviews.llvm.org/D107090
2021-07-30 03:06:22 +08:00
MLIRPythonCAPIHeaderSources
[mlir][python] Segment MLIR Python test dialect to avoid testonly dependency. With https://reviews.llvm.org/rG14c9207063bb00823a5126131e50c93f6e288bd3, the build is broken with -DMLIR_INCLUDE_TESTS=OFF. This patch fixes the build and we may want to do a better fix to the layering in a followup. Differential Revision: https://reviews.llvm.org/D112560
2021-10-27 01:14:50 +08:00
${_ADDL_TEST_SOURCES}
Re-engineer MLIR python build support. * Implements all of the discussed features: - Links against common CAPI libraries that are self contained. - Stops using the 'python/' directory at the root for everything, opening the namespace up for multiple projects to embed the MLIR python API. - Separates declaration of sources (py and C++) needed to build the extension from building, allowing external projects to build custom assemblies from core parts of the API. - Makes the core python API relocatable (i.e. it could be embedded as something like 'npcomp.ir', 'npcomp.dialects', etc). Still a bit more to do to make it truly isolated but the main structural reset is done. - When building statically, installed python packages are completely self contained, suitable for direct setup and upload to PyPi, et al. - Lets external projects assemble their own CAPI common runtime library that all extensions use. No more possibilities for TypeID issues. - Begins modularizing the API so that external projects that just include a piece pay only for what they use. * I also rolled in a re-organization of the native libraries that matches how I was packaging these out of tree and is a better layering (i.e. all libraries go into a nested _mlir_libs package). There is some further cleanup that I resisted since it would have required source changes that I'd rather do in a followup once everything stabilizes. * Note that I made a somewhat odd choice in choosing to recompile all extensions for each project they are included into (as opposed to compiling once and just linking). While not leveraged yet, this will let us set definitions controlling the namespacing of the extensions so that they can be made to not conflict across projects (with preprocessor definitions). * This will be a relatively substantial breaking change for downstreams. I will handle the npcomp migration and will coordinate with the circt folks before landing. We should stage this and make sure it isn't causing problems before landing. * Fixed a couple of absolute imports that were causing issues. Differential Revision: https://reviews.llvm.org/D106520
2021-07-23 03:57:41 +08:00
COMMON_CAPI_LINK_LIBS
MLIRPythonCAPI
)