clang-offload-bundler is not only used by clang driver
to bundle/unbundle files for offloading toolchains,
but also used by out of tree tools to unbundle
fat binaries generated by clang. It is important
to be able to list the bundle IDs in a bundled
file so that the bundles can be extracted.
This patch adds an option -list to list bundle
ID's in a bundled file. Each bundle ID is separated
by new line. If the file is not a bundled file
nothing is output and returns 0.
Differential Revision: https://reviews.llvm.org/D92954
There are out-of-tree tools using clang-offload-bundler to extract
bundles from bundled files. When a bundle is not in the bundled
file, clang-offload-bundler is expected to emit an error message
and return non-zero value. However currently clang-offload-bundler
silently generates empty file for the missing bundles.
Since OpenMP/HIP toolchains expect the current behavior, an option
-allow-missing-bundles is added to let clang-offload-bundler
create empty file when a bundle is missing when unbundling.
The unbundling job action is updated to use this option by
default.
clang-offload-bundler itself will emit error when a bundle
is missing when unbundling by default.
Changes are also made to check duplicate targets in -targets
option and emit error.
Differential Revision: https://reviews.llvm.org/D93068
Use a different container that preserves existing elements on modification
for storing temporary file names. Current container can make StringRefs
returned earlier invalid on reallocation.
Reviewed By: ABataev
Differential Revision: https://reviews.llvm.org/D92010
The path produced by getMainExecutable() may not be the right one when the files are installed in
a symlinked tree and when the real location of llvm-objdump is in a different directory.
Given that clang-offload-bundler is invoked by clang, the driver already does the job figuring out
the right path (e.g. it pays attention to -no-canonical-prefixes).
Offload bundler should use it, instead of trying to figure out the path on its
own.
Differential Revision: https://reviews.llvm.org/D90436
To facilitate faster loading of device binaries and share them among processes,
HIP runtime favors their alignment being 4096 bytes. HIP runtime can load
unaligned device binaries, however, aligning them at 4096 bytes results in
faster loading and less shared memory usage.
This patch adds an option -bundle-align to clang-offload-bundler which allows
bundles to be aligned at specified alignment. By default it is 1, which is NFC
compared to existing format.
This patch then aligns embedded fat binary and device binary inside fat binary
at 4096 bytes.
It has been verified this change does not cause significant overall file size increase
for typical HIP applications (less than 1%).
Differential Revision: https://reviews.llvm.org/D88734
The `intrinsics_gen` target exists in the CMake exports since r309389
(see LLVMConfig.cmake.in), hence projects can depend on `intrinsics_gen`
even it they are built separately from LLVM.
Reviewed By: MaskRay, JDevlieghere
Differential Revision: https://reviews.llvm.org/D83454
Summary:
Fat object size has significantly increased after D65819 which changed bundler tool to add host object as a normal bundle to the fat output which almost doubled its size. That patch was fixing the following issues
1. Problems associated with the partial linking - global constructors were not called for partially linking objects which clearly resulted in incorrect behavior.
2. Eliminating "junk" target object sections from the linked binary on the host side.
The first problem is no longer relevant because we do not use partial linking for creating fat objects anymore. Target objects sections are now inserted into the resulting fat object with a help of llvm-objcopy tool.
The second issue, "junk" sections in the linked host binary, has been fixed in D73408 by adding "exclude" flag to the fat object's sections which contain target objects. This flag tells linker to drop section from the inputs when linking executable or shared library, therefore these sections will not be propagated in the linked binary.
Since both problems have been solved, we can revert D65819 changes to reduce fat object size and this patch essentially is doing that.
Reviewers: ABataev, alexshap, jdoerfert
Reviewed By: ABataev
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D73642
Summary: This flag tells link editor to exclude section from linker inputs when linking executable or shared library.
Reviewers: ABataev, alexshap, jdoerfert
Reviewed By: ABataev
Subscribers: cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D73408
This reverts commit 3f76260dc0.
Breaks at least these tests on Windows:
Clang :: Driver/clang-offload-bundler.c
Clang :: Driver/clang-offload-wrapper.c
- Changed FileHandler read/write methods to return llvm::Error
- Using unified way of reporting errors
- Removed trailing '.' from the error messages
Differential Revision: https://reviews.llvm.org/D67031
- Changed FileHandler read/write methods to return llvm::Error
- Using unified way of reporting errors
- Removed trailing '.' from the error messages
Differential Revision: https://reviews.llvm.org/D67031
Allow finding installed llvm-objcopy in PATH if it's not present
in the directory containing clang-offload-bundler. This is the case
if clang is being built stand-alone, and llvm-objcopy is already
installed while the c-o-b tool is still present in build directory.
This is consistent with how e.g. llvm-symbolizer is found in LLVM.
However, most of similar searches in LLVM and Clang are performed
without special-casing the program directory.
Fixes r369955.
Differential Revision: https://reviews.llvm.org/D68931
llvm-svn: 374754
Switch clang-check, clang-extdef-mapping and clang-offload-bundler
to use add_clang_tool() rather than add_clang_executable() with a custom
install rule. This makes them LLVM_DISTRIBUTION_COMPONENTS-friendly.
Differential Revision: https://reviews.llvm.org/D68429
llvm-svn: 373785
Bundler leaks memory if it is called with -type=o but given input isn't an object file (though it has to have a known binary type like IR, archive, etc...). Memory leak is happening when binary object returned by the createBinary(...) call cannot be casted to an ObjectFile type. In this case returned BinaryOrErr object releases ownership of the binary, but no one is taking it (see line 626).
Differential Revision: https://reviews.llvm.org/D67416
llvm-svn: 371633
Bundler currently requires host triple to be provided no matter if you are performing bundling or unbundling, but for unbundling operation such requirement is too restrictive. You may for example want to examine device part of the object for a particular offload target, but you have to extract host part as well even though you do not need it. Host triple isn't really needed for unbundling, so this patch removes that requirement.
Differential Revision: https://reviews.llvm.org/D66601
llvm-svn: 370143
clang-offload-bundler tool may hang under certain conditions when it extracts a subset of all available device bundles from the fat binary that is handled by the BinaryFileHandler. This patch fixes this problem.
Differential Revision: https://reviews.llvm.org/D66598
llvm-svn: 370115
clang-offload-bundler currently uses partial linking for creating fat object files, but such technique cannot be used on Windows due to the absence of partial linking support in the linker. This patch changes implementation to use llvm-objcopy for merging device and host objects instead of doing partial linking. This is one step forward towards enabling OpenMP offload on Windows.
Differential Revision: https://reviews.llvm.org/D66485
llvm-svn: 369955
Summary:
Previously, object files were bundled using partial linking. It resulted
in the following structure of the bundled objects:
```
<host_code>
clang-offload-bundle
__CLANG_OFFLOAD_BUNDLE__<target>
<target_code>
```
But when we tried to unbundle object files, it worked correctly only for
the target objects. The host object remains bundled. It produced a lot of
junk sections in the host object files and in some cases may caused
incorrect linking.
Patch improves bundling of the object files. After this patch the
bundled object looks like this:
```
<host_code>
clang-offload-bundle
__CLANG_OFFLOAD_BUNDLE__<target>
<target_code>
__CLANG_OFFLOAD_BUNDLE__<host>
<host_code>
```
With this structure we are able to unbundle the host object files too so
that after unbundling they are the same as were before.
The host section is bundled twice. The bundled section is used to
unbundle the original host section.
Reviewers: yaxunl, tra, jlebar, hfinkel, jdoerfert
Subscribers: caomhin, kkwli0, cfe-commits
Tags: #clang
Differential Revision: https://reviews.llvm.org/D65819
llvm-svn: 369019
r360876 didn't fix 2 call sites in clang.
Expected<ArrayRef<uint8_t>> may be better but use Expected<StringRef> for now.
Follow-up of D61781.
llvm-svn: 360892
HIP uses clang-offload-bundler to create fat binary. The bundle for host is empty.
Currently clang-offload-bundler checks if the bundle size is 0 when unbundling.
If so it will exit without unbundling the remaining bundles. This causes
clang-offload-bundler not being able to unbundle fat binaries generated for HIP.
This patch allows bundles size to be 0 when clang-offload-bundler unbundles
input files.
Differential Revision: https://reviews.llvm.org/D58057
llvm-svn: 355419
to reflect the new license.
We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.
Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.
llvm-svn: 351636
This simplifies some code which had StringRefs to begin with, and
makes other code more complicated which had const char* to begin
with.
In the end, I think this makes for a more idiomatic and platform
agnostic API. Not all platforms launch process with null terminated
c-string arrays for the environment pointer and argv, but the api
was designed that way because it allowed easy pass-through for
posix-based platforms. There's a little additional overhead now
since on posix based platforms we'll be takign StringRefs which
were constructed from null terminated strings and then copying
them to null terminate them again, but from a readability and
usability standpoint of the API user, I think this API signature
is strictly better.
llvm-svn: 334518
When bundle/unbundle intermediate files for HIP, there may be multiple
sub archs, therefore BoundArch needs to be included in the target
and output file names for clang-offload-bundler.
Differential Revision: https://reviews.llvm.org/D46473
llvm-svn: 332121
This is similar to the LLVM change https://reviews.llvm.org/D46290.
We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.
Patch produced by
for i in $(git grep -l '\@brief'); do perl -pi -e 's/\@brief //g' $i & done
for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done
Differential Revision: https://reviews.llvm.org/D46320
llvm-svn: 331834
We currently use target_link_libraries without an explicit scope
specifier (INTERFACE, PRIVATE or PUBLIC) when linking executables.
Dependencies added in this way apply to both the target and its
dependencies, i.e. they become part of the executable's link interface
and are transitive.
Transitive dependencies generally don't make sense for executables,
since you wouldn't normally be linking against an executable. This also
causes issues for generating install export files when using
LLVM_DISTRIBUTION_COMPONENTS. For example, clang has a lot of LLVM
library dependencies, which are currently added as interface
dependencies. If clang is in the distribution components but the LLVM
libraries it depends on aren't (which is a perfectly legitimate use case
if the LLVM libraries are being built static and there are therefore no
run-time dependencies on them), CMake will complain about the LLVM
libraries not being in export set when attempting to generate the
install export file for clang. This is reasonable behavior on CMake's
part, and the right thing is for LLVM's build system to explicitly use
PRIVATE dependencies for executables.
Unfortunately, CMake doesn't allow you to mix and match the keyword and
non-keyword target_link_libraries signatures for a single target; i.e.,
if a single call to target_link_libraries for a particular target uses
one of the INTERFACE, PRIVATE, or PUBLIC keywords, all other calls must
also be updated to use those keywords. This means we must do this change
in a single shot. I also fully expect to have missed some instances; I
tested by enabling all the projects in the monorepo (except dragonegg),
and configuring both with and without shared libraries, on both Darwin
and Linux, but I'm planning to rely on the buildbots for other
configurations (since it should be pretty easy to fix those).
Even after this change, we still have a lot of target_link_libraries
calls that don't specify a scope keyword, mostly for shared libraries.
I'm thinking about addressing those in a follow-up, but that's a
separate change IMO.
Differential Revision: https://reviews.llvm.org/D40823
llvm-svn: 319840
Summary:
Other llvm tools display their registered targets when showing version
information, but for some reason clang has never done this.
To support this, D33899 adds the llvm parts, which make it possible to
print version information to arbitrary raw_ostreams. This change adds
a call to printRegisteredTargetsForVersion in clang's PrintVersion, and
adds a raw_ostream parameter to two other PrintVersion functions.
Reviewers: beanz, chandlerc, dberris, mehdi_amini, zturner
Reviewed By: mehdi_amini
Subscribers: cfe-commits
Differential Revision: https://reviews.llvm.org/D33900
llvm-svn: 304836
ClangOffloadBundler.cpp has the following include chain:
llvm/Bitcode/BitcodeWriter.h
llvm/IR/ModuleSummaryIndex.h
llvm/IR/Module.h
llvm/IR/Function.h
llvm/IR/Argument.h
llvm/IR/Attributes.h
llvm/IR/Attributes.gen
This means clang-offload-bundler needs to depend on intrinsics_gen.
llvm-svn: 287406
The change in D26502 splits ReaderWriter.h, which contains the APIs
into both the BitReader and BitWriter libraries, into BitcodeReader.h
and BitcodeWriter.h.
Change clang uses to the appropriate split header(s).
llvm-svn: 286567
Summary:
1. Pair removed from StringMap was not destroyed
2. ObjectFile had no owner
Reviewers: sfantao
Subscribers: llvm-commits
Differential Revision: https://reviews.llvm.org/D23865
llvm-svn: 279722
Summary:
This patch adds the capability to bundle object files in sections of the host binary using a designated naming convention for these sections. This patch uses the functionality of the object reader already in the LLVM library to read bundled files, and invokes clang with the incremental linking options to create bundle files.
Bundling files involves creating an IR file with the contents of the bundle assigned as initializers of globals binded to the designated sections. This way the bundling implementation is agnostic of the host object format.
The features added by this patch were requested in the RFC discussion in http://lists.llvm.org/pipermail/cfe-dev/2016-February/047547.html.
Reviewers: echristo, tra, jlebar, hfinkel, ABataev, Hahnfeld
Subscribers: mkuron, whchung, cfe-commits, andreybokhanko, Hahnfeld, arpith-jacob, carlo.bertolli, mehdi_amini, caomhin
Differential Revision: https://reviews.llvm.org/D21851
llvm-svn: 279634
Summary:
One of the goals of programming models that support offloading (e.g. OpenMP) is to enable users to offload with little effort, by annotating the code with a few pragmas. I'd also like to save users the trouble of changing their existent applications' build system. So having the compiler always return a single file instead of one for the host and each target even if the user is doing separate compilation is desirable.
This diff proposes a tool named clang-offload-bundler (happy to change the name if required) that is used to bundle files associated with the same user source file but different targets, or to unbundle a file into separate files associated with different targets.
This tool supports the driver support for OpenMP under review in http://reviews.llvm.org/D9888. The tool is used there to enable separate compilation, so that the very first action on input files that are not source files is a "unbundling action" and the very last non-linking action is a "bundling action".
The format of the bundled files is currently very simple: text formats are concatenated with comments that have a magic string and target identifying triple in between, and binary formats have a header that contains the triple and the offset and size of the code for host and each target.
The goal is to improve this tool in the future to deal with archive files so that each individual file in the archive is properly dealt with. We see that archives are very commonly used in current applications to combine separate compilation results. So I'm convinced users would enjoy this feature.
This tool can be used like this:
`clang-offload-bundler -targets=triple1,triple2 -type=ii -inputs=a.triple1.ii,a.triple2.ii -outputs=a.ii`
or
`clang-offload-bundler -targets=triple1,triple2 -type=ii -outputs=a.triple1.ii,a.triple2.ii -inputs=a.ii -unbundle`
I implemented the tool under clang/tools. Please let me know if something like this should live somewhere else.
This patch is prerequisite for http://reviews.llvm.org/D9888.
Reviewers: hfinkel, rsmith, echristo, chandlerc, tra, jlebar, ABataev, Hahnfeld
Subscribers: whchung, caomhin, andreybokhanko, arpith-jacob, carlo.bertolli, mehdi_amini, guansong, Hahnfeld, cfe-commits
Differential Revision: https://reviews.llvm.org/D13909
llvm-svn: 279632
Yaxun (Sam) Liu