Currently, clang-offload-bundler has -inputs and -outputs options that accept
values with comma as the delimiter. This causes issues with file paths
containing commas, which are valid file paths on Linux.
This add two new options -input and -output, which accept one single file,
and allow multiple instances. This allows arbitrary file paths. The old
-inputs and -outputs options will be kept for backward compatibility, but
are not allowed to be used with -input and -output options for simplicity.
In the future, -inputs and -outputs options will be phasing out.
RFC: https://discourse.llvm.org/t/rfc-adding-input-and-output-options-to-clang-offload-bundler/60049
Patch by: Siu Chi Chan
Reviewed by: Yaxun Liu
Differential Revision: https://reviews.llvm.org/D120662
`hip-openmp-compatible` flag treats hip and hipv4 offload kinds
as compatible with openmp offload kind while extracting code objects
from a heterogenous archive library. Vice versa is also considered
compatible if hip code was compiled with -fgpu-rdc.
This flag only relaxes compatibility criteria on `OffloadKind`,
rest of the components like `Triple` and `GPUArhc` still needs to
be compatible.
Reviewed By: yaxunl
Differential Revision: https://reviews.llvm.org/D120697
HIP programs compiled with -c -fgpu-rdc generate clang-offload-bundler
bundles which contain bitcode for different GPU's.
Such files can be archived to an archive file which can be linked with
HIP programs with -fgpu-rdc.
This patch adds suppor of linking archive of bundled bitcode.
When an archive of bundled bitcode is passed to clang by -l, for each
GPU specified through --offload-arch, clang extracts bitcode from
the archive and creates a new archive for that GPU and pass it
to lld.
Reviewed by: Artem Belevich
Differential Revision: https://reviews.llvm.org/D120070
Fixes: SWDEV-321741, SWDEV-315773
GPUs are not supported on AIX, so this patch sets these tests as unsupported.
Reviewed By: stevewan
Differential Revision: https://reviews.llvm.org/D114381
Test using debug-only=CodeObjectComaptibility was failing in
non-assert builds, so it has been moved to a different file which
requires assert.
Reviewed By: RKSimon
Differential Revision: https://reviews.llvm.org/D109592
Earlier BundleEntryID used to be <OffloadKind>-<Triple>-<GPUArch>.
This used to work because the clang-offload-bundler didn't need
GPUArch explicitly for any bundling/unbundling action. With
unbundleArchive it needs GPUArch to ensure compatibility between
device specific code objects. D93525 enforced triples to have
separators for all 4 components irrespective of number of
components, like "amdgcn-amd-amdhsa--". It was required to
to correctly parse a possible 4th environment component or a GPU.
But, this condition is breaking backward compatibility with
archive libraries compiled with compilers older than D93525.
This patch allows triples to have any number of components with
and without extra separator for empty environment field. Thus,
both the following bundle entry IDs are same:
openmp-amdgcn-amd-amdhsa--gfx906
openmp-amdgcn-amd-amdhsa-gfx906
Reviewed By: yaxunl, grokos
Differential Revision: https://reviews.llvm.org/D106809
This patch adds unbundling support of an archive file. It takes an
archive file along with a set of offload targets as input.
Output is a device specific archive for each given offload target.
Input archive contains bundled code objects bundled using
clang-offload-bundler. Each generated device specific archive contains
a set of device code object files which are named as
<Parent Bundle Name>-<CodeObject-GPUArch>.
Entries in input archive can be of any binary type which is
supported by clang-offload-bundler, like *.bc. Output archives will
contain files in same type.
Example Usuage:
clang-offload-bundler --unbundle --inputs=lib-generic.a -type=a
-targets=openmp-amdgcn-amdhsa--gfx906,openmp-amdgcn-amdhsa--gfx908
-outputs=devicelib-gfx906.a,deviceLib-gfx908.a
Reviewed By: jdoerfert, yaxunl
Differential Revision: https://reviews.llvm.org/D93525
Added --gpu-bundle-output to control bundling/unbundling output of HIP device compilation.
By default preprocessor expansion, llvm bitcode and assembly are unbundled, code objects are
bundled.
Reviewed by: Artem Belevich, Jan Svoboda
Differential Revision: https://reviews.llvm.org/D101630
llvm-objcopy has been changed to support adding a section and updating section flags
in one run (D90438), so we can now change clang-offload-bundler to run llvm-objcopy
tool only once when creating fat object.
Reviewed By: ABataev
Differential Revision: https://reviews.llvm.org/D102670
On z/OS, the following error message is not matched correctly in lit tests.
```
EDC5129I No such file or directory.
```
This patch uses a lit config substitution to check for platform specific error messages.
Reviewed By: muiez, jhenderson
Differential Revision: https://reviews.llvm.org/D95246
clang-offload-bundler is not only used by clang driver
to bundle/unbundle files for offloading toolchains,
but also used by out of tree tools to unbundle
fat binaries generated by clang. It is important
to be able to list the bundle IDs in a bundled
file so that the bundles can be extracted.
This patch adds an option -list to list bundle
ID's in a bundled file. Each bundle ID is separated
by new line. If the file is not a bundled file
nothing is output and returns 0.
Differential Revision: https://reviews.llvm.org/D92954
There are out-of-tree tools using clang-offload-bundler to extract
bundles from bundled files. When a bundle is not in the bundled
file, clang-offload-bundler is expected to emit an error message
and return non-zero value. However currently clang-offload-bundler
silently generates empty file for the missing bundles.
Since OpenMP/HIP toolchains expect the current behavior, an option
-allow-missing-bundles is added to let clang-offload-bundler
create empty file when a bundle is missing when unbundling.
The unbundling job action is updated to use this option by
default.
clang-offload-bundler itself will emit error when a bundle
is missing when unbundling by default.
Changes are also made to check duplicate targets in -targets
option and emit error.
Differential Revision: https://reviews.llvm.org/D93068
To facilitate faster loading of device binaries and share them among processes,
HIP runtime favors their alignment being 4096 bytes. HIP runtime can load
unaligned device binaries, however, aligning them at 4096 bytes results in
faster loading and less shared memory usage.
This patch adds an option -bundle-align to clang-offload-bundler which allows
bundles to be aligned at specified alignment. By default it is 1, which is NFC
compared to existing format.
This patch then aligns embedded fat binary and device binary inside fat binary
at 4096 bytes.
It has been verified this change does not cause significant overall file size increase
for typical HIP applications (less than 1%).
Differential Revision: https://reviews.llvm.org/D88734
Summary:
Fat object size has significantly increased after D65819 which changed bundler tool to add host object as a normal bundle to the fat output which almost doubled its size. That patch was fixing the following issues
1. Problems associated with the partial linking - global constructors were not called for partially linking objects which clearly resulted in incorrect behavior.
2. Eliminating "junk" target object sections from the linked binary on the host side.
The first problem is no longer relevant because we do not use partial linking for creating fat objects anymore. Target objects sections are now inserted into the resulting fat object with a help of llvm-objcopy tool.
The second issue, "junk" sections in the linked host binary, has been fixed in D73408 by adding "exclude" flag to the fat object's sections which contain target objects. This flag tells linker to drop section from the inputs when linking executable or shared library, therefore these sections will not be propagated in the linked binary.
Since both problems have been solved, we can revert D65819 changes to reduce fat object size and this patch essentially is doing that.
Reviewers: ABataev, alexshap, jdoerfert
Reviewed By: ABataev
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D73642
Summary: This flag tells link editor to exclude section from linker inputs when linking executable or shared library.
Reviewers: ABataev, alexshap, jdoerfert
Reviewed By: ABataev
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D73408
- Changed FileHandler read/write methods to return llvm::Error
- Using unified way of reporting errors
- Removed trailing '.' from the error messages
Differential Revision: https://reviews.llvm.org/D67031
- Changed FileHandler read/write methods to return llvm::Error
- Using unified way of reporting errors
- Removed trailing '.' from the error messages
Differential Revision: https://reviews.llvm.org/D67031
This reverts r371497 (git commit 3d7e9ab7b9)
Reorder `not` with `env` in these two tests so they pass:
Driver/rewrite-map-in-diagnostics.c
Index/crash-recovery-modules.m.
This will not be necessary after D66531 lands.
llvm-svn: 371552
I see in the history for some of these tests REQUIRES:shell was used as
a way to disable tests on Windows because they are flaky there. I tried
not to re-enable such tests, but it's possible that I missed some and
this will re-enable flaky tests on Windows. If so, we should disable
them with UNSUPPORTED:system-windows and add a comment that they are
flaky there. So far as I can tell, the lit internal shell is capable of
running all of these tests, and we shouldn't use REQUIRES:shell as a
proxy for Windows.
llvm-svn: 371478
Bundler currently requires host triple to be provided no matter if you are performing bundling or unbundling, but for unbundling operation such requirement is too restrictive. You may for example want to examine device part of the object for a particular offload target, but you have to extract host part as well even though you do not need it. Host triple isn't really needed for unbundling, so this patch removes that requirement.
Differential Revision: https://reviews.llvm.org/D66601
llvm-svn: 370143
clang-offload-bundler tool may hang under certain conditions when it extracts a subset of all available device bundles from the fat binary that is handled by the BinaryFileHandler. This patch fixes this problem.
Differential Revision: https://reviews.llvm.org/D66598
llvm-svn: 370115
clang-offload-bundler currently uses partial linking for creating fat object files, but such technique cannot be used on Windows due to the absence of partial linking support in the linker. This patch changes implementation to use llvm-objcopy for merging device and host objects instead of doing partial linking. This is one step forward towards enabling OpenMP offload on Windows.
Differential Revision: https://reviews.llvm.org/D66485
llvm-svn: 369955
Summary:
Previously, object files were bundled using partial linking. It resulted
in the following structure of the bundled objects:
```
<host_code>
clang-offload-bundle
__CLANG_OFFLOAD_BUNDLE__<target>
<target_code>
```
But when we tried to unbundle object files, it worked correctly only for
the target objects. The host object remains bundled. It produced a lot of
junk sections in the host object files and in some cases may caused
incorrect linking.
Patch improves bundling of the object files. After this patch the
bundled object looks like this:
```
<host_code>
clang-offload-bundle
__CLANG_OFFLOAD_BUNDLE__<target>
<target_code>
__CLANG_OFFLOAD_BUNDLE__<host>
<host_code>
```
With this structure we are able to unbundle the host object files too so
that after unbundling they are the same as were before.
The host section is bundled twice. The bundled section is used to
unbundle the original host section.
Reviewers: yaxunl, tra, jlebar, hfinkel, jdoerfert
Subscribers: caomhin, kkwli0, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D65819
llvm-svn: 369019
Summary:
Make the test more portable and do not rely on the pre-bundled object
file.
Reviewers: Hahnfeld, hfinkel, jdoerfert
Subscribers: caomhin, kkwli0, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D66296
llvm-svn: 369015
Summary:
By default, `parseCommandLineOptions()` will accept either a
`-` or `--` prefix for long options -- options with names longer than
a single character.
While this change does not affect behavior, it will be helpful with a
subsequent change that requires long options use the `--` prefix.
Reviewers: rnk, thopre
Reviewed By: thopre
Subscribers: thopre, cfe-commits, hiraditya, llvm-commits
Tags: #llvm, #clang
Differential Revision: https://reviews.llvm.org/D61269
llvm-svn: 359909
Summary:
This patch adds the capability to bundle object files in sections of the host binary using a designated naming convention for these sections. This patch uses the functionality of the object reader already in the LLVM library to read bundled files, and invokes clang with the incremental linking options to create bundle files.
Bundling files involves creating an IR file with the contents of the bundle assigned as initializers of globals binded to the designated sections. This way the bundling implementation is agnostic of the host object format.
The features added by this patch were requested in the RFC discussion in http://lists.llvm.org/pipermail/cfe-dev/2016-February/047547.html.
Reviewers: echristo, tra, jlebar, hfinkel, ABataev, Hahnfeld
Subscribers: mkuron, whchung, cfe-commits, andreybokhanko, Hahnfeld, arpith-jacob, carlo.bertolli, mehdi_amini, caomhin
Differential Revision: https://reviews.llvm.org/D21851
llvm-svn: 279634
Summary:
One of the goals of programming models that support offloading (e.g. OpenMP) is to enable users to offload with little effort, by annotating the code with a few pragmas. I'd also like to save users the trouble of changing their existent applications' build system. So having the compiler always return a single file instead of one for the host and each target even if the user is doing separate compilation is desirable.
This diff proposes a tool named clang-offload-bundler (happy to change the name if required) that is used to bundle files associated with the same user source file but different targets, or to unbundle a file into separate files associated with different targets.
This tool supports the driver support for OpenMP under review in http://reviews.llvm.org/D9888. The tool is used there to enable separate compilation, so that the very first action on input files that are not source files is a "unbundling action" and the very last non-linking action is a "bundling action".
The format of the bundled files is currently very simple: text formats are concatenated with comments that have a magic string and target identifying triple in between, and binary formats have a header that contains the triple and the offset and size of the code for host and each target.
The goal is to improve this tool in the future to deal with archive files so that each individual file in the archive is properly dealt with. We see that archives are very commonly used in current applications to combine separate compilation results. So I'm convinced users would enjoy this feature.
This tool can be used like this:
`clang-offload-bundler -targets=triple1,triple2 -type=ii -inputs=a.triple1.ii,a.triple2.ii -outputs=a.ii`
or
`clang-offload-bundler -targets=triple1,triple2 -type=ii -outputs=a.triple1.ii,a.triple2.ii -inputs=a.ii -unbundle`
I implemented the tool under clang/tools. Please let me know if something like this should live somewhere else.
This patch is prerequisite for http://reviews.llvm.org/D9888.
Reviewers: hfinkel, rsmith, echristo, chandlerc, tra, jlebar, ABataev, Hahnfeld
Subscribers: whchung, caomhin, andreybokhanko, arpith-jacob, carlo.bertolli, mehdi_amini, guansong, Hahnfeld, cfe-commits
Differential Revision: https://reviews.llvm.org/D13909
llvm-svn: 279632
Fangrui Song