Commit Graph

63 Commits

Author SHA1 Message Date
Saiyedul Islam 7a02abf06f
[clang-offload-bundler] HIP and OpenMP comaptibility for linking heterogeneous archive library
`hip-openmp-compatible` flag treats hip and hipv4 offload kinds
as compatible with openmp offload kind while extracting code objects
from a heterogenous archive library. Vice versa is also considered
compatible if hip code was compiled with -fgpu-rdc.

This flag only relaxes compatibility criteria on `OffloadKind`,
rest of the components like `Triple` and `GPUArhc` still needs to
be compatible.

Reviewed By: yaxunl

Differential Revision: https://reviews.llvm.org/D120697
2022-03-02 07:55:06 +00:00
Sylvestre Ledru f2c2e924e7 Fix a typo (occured => occurred)
Reported:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1005195
2022-02-08 21:35:26 +01:00
Kazu Hirata d677a7cb05 [clang] Remove redundant member initialization (NFC)
Identified with readability-redundant-member-init.
2022-01-02 10:20:23 -08:00
Sergey Dmitriev 4b553297ef [clang-offload-bundler] Reuse original file extension for device archive member
This patch changes clang-offload-bundler to use the original file extension for
the device archive member when unbundling archives instead of printing a warning
and defaulting to ".o".

Differential Revision: https://reviews.llvm.org/D114776
2021-11-30 20:38:59 -08:00
Saiyedul Islam 35ebe4cc24 [Clang][OpenMP] Add partial support for Static Device Libraries
An archive containing device code object files can be passed to
clang command line for linking. For each given offload target
it creates a device specific archives which is either passed to llvm-link
if the target is amdgpu, or to clang-nvlink-wrapper if the target is
nvptx. -L/-l flags are used to specify these fat archives on the command
line. E.g.
  clang++ -fopenmp -fopenmp-targets=nvptx64 main.cpp -L. -lmylib

It currently doesn't support linking an archive directly, like:
  clang++ -fopenmp -fopenmp-targets=nvptx64 main.cpp libmylib.a

Linking with x86 offload also does not work.

Reviewed By: ye-luo

Differential Revision: https://reviews.llvm.org/D105191
2021-10-08 09:37:51 +00:00
Saiyedul Islam 94e2b0258a Revert "[Clang][OpenMP] Add partial support for Static Device Libraries"
This reverts commit 4c41170895.
2021-10-07 14:13:24 +00:00
Saiyedul Islam 4c41170895 [Clang][OpenMP] Add partial support for Static Device Libraries
An archive containing device code object files can be passed to
clang command line for linking. For each given offload target
it creates a device specific archives which is either passed to llvm-link
if the target is amdgpu, or to clang-nvlink-wrapper if the target is
nvptx. -L/-l flags are used to specify these fat archives on the command
line. E.g.
  clang++ -fopenmp -fopenmp-targets=nvptx64 main.cpp -L. -lmylib

It currently doesn't support linking an archive directly, like:
  clang++ -fopenmp -fopenmp-targets=nvptx64 main.cpp libmylib.a

Linking with x86 offload also does not work.

Reviewed By: ye-luo

Differential Revision: https://reviews.llvm.org/D105191
2021-10-07 04:45:19 +00:00
Saiyedul Islam 98380762c3 [clang-offload-bundler] Make Bundle Entry ID backward compatible
Earlier BundleEntryID used to be <OffloadKind>-<Triple>-<GPUArch>.
This used to work because the clang-offload-bundler didn't need
GPUArch explicitly for any bundling/unbundling action. With
unbundleArchive it needs GPUArch to ensure compatibility between
device specific code objects. D93525 enforced triples to have
separators for all 4 components irrespective of number of
components, like "amdgcn-amd-amdhsa--". It was required to
to correctly parse a possible 4th environment component or a GPU.
But, this condition is breaking backward compatibility with
archive libraries compiled with compilers older than D93525.

This patch allows triples to have any number of components with
and without extra separator for empty environment field. Thus,
both the following bundle entry IDs are same:
openmp-amdgcn-amd-amdhsa--gfx906
openmp-amdgcn-amd-amdhsa-gfx906

Reviewed By: yaxunl, grokos

Differential Revision: https://reviews.llvm.org/D106809
2021-09-08 16:06:12 +05:30
Simon Pilgrim c94f4a9c5d ClangOffloadBundler - getCompatibleOffloadTargets - Fix unknown parameter Wdocumentation warnings. NFC. 2021-08-20 14:31:11 +01:00
David Blaikie 1def2579e1 PR51018: Remove explicit conversions from SmallString to StringRef to future-proof against C++23
C++23 will make these conversions ambiguous - so fix them to make the
codebase forward-compatible with C++23 (& a follow-up change I've made
will make this ambiguous/invalid even in <C++23 so we don't regress
this & it generally improves the code anyway)
2021-07-08 13:37:57 -07:00
Simon Pilgrim 769e782793 Fix MSVC "truncation from 'int' to 'bool'" warning. NFCI. 2021-07-08 18:08:24 +01:00
Saiyedul Islam f7ce532d62 [clang-offload-bundler] Add unbundling of archives containing bundled object files into device specific archives
This patch adds unbundling support of an archive file. It takes an
archive file along with a set of offload targets as input.
Output is a device specific archive for each given offload target.
Input archive contains bundled code objects bundled using
clang-offload-bundler. Each generated device specific archive contains
a set of device code object files which are named as
<Parent Bundle Name>-<CodeObject-GPUArch>.

Entries in input archive can be of any binary type which is
supported by clang-offload-bundler, like *.bc. Output archives will
contain files in same type.

Example Usuage:
  clang-offload-bundler --unbundle --inputs=lib-generic.a -type=a
      -targets=openmp-amdgcn-amdhsa--gfx906,openmp-amdgcn-amdhsa--gfx908
      -outputs=devicelib-gfx906.a,deviceLib-gfx908.a

Reviewed By: jdoerfert, yaxunl

Differential Revision: https://reviews.llvm.org/D93525
2021-06-30 17:55:50 +05:30
Yaxun (Sam) Liu 5fc2673fbc [HIP] Add --gpu-bundle-output
Added --gpu-bundle-output to control bundling/unbundling output of HIP device compilation.

By default preprocessor expansion, llvm bitcode and assembly are unbundled, code objects are
bundled.

Reviewed by: Artem Belevich, Jan Svoboda

Differential Revision: https://reviews.llvm.org/D101630
2021-06-09 23:31:43 -04:00
Sergey Dmitriev f8444a8e94 [clang-offload-bundler] Delimit input/output file names by '--' for llvm-objcopy
That fixes a problem of using bundler with file names starting with dash.

Reviewed By: ABataev

Differential Revision: https://reviews.llvm.org/D102752
2021-05-19 20:25:05 -07:00
Sergey Dmitriev 8998a8aa97 [clang-offload-bundler] Add sections and set section flags using one llvm-objcopy invocation
llvm-objcopy has been changed to support adding a section and updating section flags
in one run (D90438), so we can now change clang-offload-bundler to run llvm-objcopy
tool only once when creating fat object.

Reviewed By: ABataev

Differential Revision: https://reviews.llvm.org/D102670
2021-05-18 08:44:41 -07:00
Yaxun (Sam) Liu 4fd05e0ad7 [HIP] Change to code object v4
Change to code object v4 by default to match ROCm 4.1.

Reviewed by: Artem Belevich

Differential Revision: https://reviews.llvm.org/D99235
2021-04-06 20:22:58 -04:00
Yaxun (Sam) Liu 90bf3ecef4 [clang-offload-bundler] Add option -list
clang-offload-bundler is not only used by clang driver
to bundle/unbundle files for offloading toolchains,
but also used by out of tree tools to unbundle
fat binaries generated by clang. It is important
to be able to list the bundle IDs in a bundled
file so that the bundles can be extracted.

This patch adds an option -list to list bundle
ID's in a bundled file. Each bundle ID is separated
by new line. If the file is not a bundled file
nothing is output and returns 0.

Differential Revision: https://reviews.llvm.org/D92954
2021-01-06 16:23:01 -05:00
Yaxun (Sam) Liu b9fb063e63 [clang-offload-bundler] Add option -allow-missing-bundles
There are out-of-tree tools using clang-offload-bundler to extract
bundles from bundled files. When a bundle is not in the bundled
file, clang-offload-bundler is expected to emit an error message
and return non-zero value. However currently clang-offload-bundler
silently generates empty file for the missing bundles.

Since OpenMP/HIP toolchains expect the current behavior, an option
-allow-missing-bundles is added to let clang-offload-bundler
create empty file when a bundle is missing when unbundling.
The unbundling job action is updated to use this option by
default.

clang-offload-bundler itself will emit error when a bundle
is missing when unbundling by default.

Changes are also made to check duplicate targets in -targets
option and emit error.

Differential Revision: https://reviews.llvm.org/D93068
2020-12-16 14:52:39 -05:00
Sergey Dmitriev 1b0ca81a6c [clang-offload-bundler] use std::forward_list for storing temp file names [NFC]
Use a different container that preserves existing elements on modification
for storing temporary file names. Current container can make StringRefs
returned earlier invalid on reallocation.

Reviewed By: ABataev

Differential Revision: https://reviews.llvm.org/D92010
2020-11-24 08:07:31 -08:00
Artem Belevich cdbf6bfdc7 [HIP] Use argv[0] as the default choice for the Executable name.
The path produced by getMainExecutable() may not be the right one when the files are installed in
a symlinked tree and when the real location of llvm-objdump is in a different directory.

Given that clang-offload-bundler is invoked by clang, the driver already does the job figuring out
the right path (e.g. it pays attention to -no-canonical-prefixes).
Offload bundler should use it, instead of trying to figure out the path on its
own.

Differential Revision: https://reviews.llvm.org/D90436
2020-11-03 10:31:39 -08:00
Yaxun (Sam) Liu dc6a0b0ec7 [HIP] Align device binary
To facilitate faster loading of device binaries and share them among processes,
HIP runtime favors their alignment being 4096 bytes. HIP runtime can load
unaligned device binaries, however, aligning them at 4096 bytes results in
faster loading and less shared memory usage.

This patch adds an option -bundle-align to clang-offload-bundler which allows
bundles to be aligned at specified alignment. By default it is 1, which is NFC
compared to existing format.

This patch then aligns embedded fat binary and device binary inside fat binary
at 4096 bytes.

It has been verified this change does not cause significant overall file size increase
for typical HIP applications (less than 1%).

Differential Revision: https://reviews.llvm.org/D88734
2020-10-02 18:10:44 -04:00
Michele Scandale 53880b8cb9 [CMake] Make `intrinsics_gen` dependency unconditional.
The `intrinsics_gen` target exists in the CMake exports since r309389
(see LLVMConfig.cmake.in), hence projects can depend on `intrinsics_gen`
even it they are built separately from LLVM.

Reviewed By: MaskRay, JDevlieghere

Differential Revision: https://reviews.llvm.org/D83454
2020-07-17 16:43:17 -07:00
Sergey Dmitriev 5be2ca2921 [Clang][Bundler][NFC] Replace SmallString<...> with StringRef
Reviewers: ABataev

Reviewed By: ABataev

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D73738
2020-01-30 12:50:04 -08:00
Sergey Dmitriev c53cb2bdc7 [Clang][Bundler] Reduce fat object size
Summary:
Fat object size has significantly increased after D65819 which changed bundler tool to add host object as a normal bundle to the fat output which almost doubled its size. That patch was fixing the following issues

1. Problems associated with the partial linking - global constructors were not called for partially linking objects which clearly resulted in incorrect behavior.
2. Eliminating "junk" target object sections from the linked binary on the host side.

The first problem is no longer relevant because we do not use partial linking for creating fat objects anymore. Target objects sections are now inserted into the resulting fat object with a help of llvm-objcopy tool.

The second issue, "junk" sections in the linked host binary, has been fixed in D73408 by adding "exclude" flag to the fat object's sections which contain target objects. This flag tells linker to drop section from the inputs when linking executable or shared library, therefore these sections will not be propagated in the linked binary.

Since both problems have been solved, we can revert D65819 changes to reduce fat object size and this patch essentially is doing that.

Reviewers: ABataev, alexshap, jdoerfert

Reviewed By: ABataev

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D73642
2020-01-30 08:21:39 -08:00
Sergey Dmitriev 6e82d0dfd8 [Clang][Bundler] Add 'exclude' flag to target objects sections
Summary: This flag tells link editor to exclude section from linker inputs when linking executable or shared library.

Reviewers: ABataev, alexshap, jdoerfert

Reviewed By: ABataev

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D73408
2020-01-29 09:00:45 -08:00
Nico Weber 6f773205cd Revert "Use InitLLVM to setup a pretty stack printer"
This reverts commit 3f76260dc0.
Breaks at least these tests on Windows:
    Clang :: Driver/clang-offload-bundler.c
    Clang :: Driver/clang-offload-wrapper.c
2019-11-25 21:06:56 -05:00
Rui Ueyama 3f76260dc0 Use InitLLVM to setup a pretty stack printer
InitLLVM does not only save a few lines from main() but also makes the
commands do the right thing for multibyte character pathnames on
Windows (i.e. canonicalize argv's to UTF-8) because of the code we
have in this file:

https://github.com/llvm/llvm-project/blob/master/llvm/lib/Support/InitLLVM.cpp#L32

For many LLVM commands, we already have calls of InitLLVM, but there
are still remainings.

Differential Revision: https://reviews.llvm.org/D70702
2019-11-26 10:56:10 +09:00
Sergey Dmitriev edb1a1de1b Reland "[Clang][Bundler] Error reporting improvements"
- Changed FileHandler read/write methods to return llvm::Error
- Using unified way of reporting errors
- Removed trailing '.' from the error messages

Differential Revision: https://reviews.llvm.org/D67031
2019-10-25 19:00:06 -07:00
Sergey Dmitriev 93b29d3882 Revert "[Clang][Bundler] Error reporting improvements"
This reverts commit dd501045cd.
2019-10-25 17:57:55 -07:00
Sergey Dmitriev dd501045cd [Clang][Bundler] Error reporting improvements
- Changed FileHandler read/write methods to return llvm::Error
- Using unified way of reporting errors
- Removed trailing '.' from the error messages

Differential Revision: https://reviews.llvm.org/D67031
2019-10-25 16:29:57 -07:00
Michal Gorny f89e758d88 [clang] [clang-offload-bundler] Fix finding installed llvm-objcopy
Allow finding installed llvm-objcopy in PATH if it's not present
in the directory containing clang-offload-bundler.  This is the case
if clang is being built stand-alone, and llvm-objcopy is already
installed while the c-o-b tool is still present in build directory.

This is consistent with how e.g. llvm-symbolizer is found in LLVM.
However, most of similar searches in LLVM and Clang are performed
without special-casing the program directory.

Fixes r369955.

Differential Revision: https://reviews.llvm.org/D68931

llvm-svn: 374754
2019-10-14 05:33:23 +00:00
Michael Liao 2ae54aba03 [clang-offload-bundler] Support `.cui` and `.d`.
Reviewers: tra, yaxunl

Subscribers: cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D68663

llvm-svn: 374167
2019-10-09 13:53:37 +00:00
Michal Gorny c178809eb6 [clang] [cmake] Use add_clang_tool() to install all tools
Switch clang-check, clang-extdef-mapping and clang-offload-bundler
to use add_clang_tool() rather than add_clang_executable() with a custom
install rule.  This makes them LLVM_DISTRIBUTION_COMPONENTS-friendly.

Differential Revision: https://reviews.llvm.org/D68429

llvm-svn: 373785
2019-10-04 20:28:59 +00:00
Sergey Dmitriev e39b5710c9 [Clang][Bundler] Replace std::vector by SmallVector [NFC]
Differential Revision: https://reviews.llvm.org/D67413

llvm-svn: 371637
2019-09-11 16:28:47 +00:00
Sergey Dmitriev 0c1257f517 [Clang][Bundler] Fix for a potential memory leak [NFC]
Bundler leaks memory if it is called with -type=o but given input isn't an object file (though it has to have a known binary type like IR, archive, etc...). Memory leak is happening when binary object returned by the createBinary(...) call cannot be casted to an ObjectFile type. In this case returned BinaryOrErr object releases ownership of the binary, but no one is taking it (see line 626).

Differential Revision: https://reviews.llvm.org/D67416

llvm-svn: 371633
2019-09-11 16:03:21 +00:00
Sergey Dmitriev 60a99f4964 [Clang][Bundler] Do not require host triple for extracting device bundles
Bundler currently requires host triple to be provided no matter if you are performing bundling or unbundling, but for unbundling operation such requirement is too restrictive. You may for example want to examine device part of the object for a particular offload target, but you have to extract host part as well even though you do not need it. Host triple isn't really needed for unbundling, so this patch removes that requirement.

Differential Revision: https://reviews.llvm.org/D66601

llvm-svn: 370143
2019-08-28 01:26:13 +00:00
Sergey Dmitriev 4368971b05 [Clang][Bundler] Fix for a hang when unbundling fat binary
clang-offload-bundler tool may hang under certain conditions when it extracts a subset of all available device bundles from the fat binary that is handled by the BinaryFileHandler. This patch fixes this problem.

Differential Revision: https://reviews.llvm.org/D66598

llvm-svn: 370115
2019-08-27 21:47:52 +00:00
Sergey Dmitriev ed153ef044 [Clang][Bundler] Use llvm-objcopy for creating fat object files
clang-offload-bundler currently uses partial linking for creating fat object files, but such technique cannot be used on Windows due to the absence of partial linking support in the linker. This patch changes implementation to use llvm-objcopy for merging device and host objects instead of doing partial linking. This is one step forward towards enabling OpenMP offload on Windows.

Differential Revision: https://reviews.llvm.org/D66485

llvm-svn: 369955
2019-08-26 19:48:43 +00:00
Alexey Bataev 4fb80d56db [Driver][Bundler] Improve bundling of object files.
Summary:
Previously, object files were bundled using partial linking. It resulted
in the following structure of the bundled objects:
```
<host_code>
clang-offload-bundle
__CLANG_OFFLOAD_BUNDLE__<target>
<target_code>
```
But when we tried to unbundle object files, it worked correctly only for
the target objects. The host object remains bundled. It produced a lot of
junk sections in the host object files and in some cases may caused
incorrect linking.

Patch improves bundling of the object files. After this patch the
bundled object looks like this:

```
<host_code>
clang-offload-bundle
__CLANG_OFFLOAD_BUNDLE__<target>
<target_code>
__CLANG_OFFLOAD_BUNDLE__<host>
<host_code>
```

With this structure we are able to unbundle the host object files too so
that after unbundling they are the same as were before.
The host section is bundled twice. The bundled section is used to
unbundle the original host section.

Reviewers: yaxunl, tra, jlebar, hfinkel, jdoerfert

Subscribers: caomhin, kkwli0, cfe-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D65819

llvm-svn: 369019
2019-08-15 17:15:35 +00:00
George Rimar a11d302fa0 [clang] - An update after LLVM change.
SectionRef::getName() was changed to return Expected<> (D66089)

llvm-svn: 368825
2019-08-14 11:10:01 +00:00
Fangrui Song d9b948b6eb Rename F_{None,Text,Append} to OF_{None,Text,Append}. NFC
F_{None,Text,Append} are kept for compatibility since r334221.

llvm-svn: 367800
2019-08-05 05:43:48 +00:00
Tom Stellard 2e97d2aa1b cmake: Add CLANG_LINK_CLANG_DYLIB option
Summary:
Setting CLANG_LINK_CLANG_DYLIB=ON causes clang tools to link against
libclang_shared.so instead of the individual component libraries.

Reviewers: mgorny, beanz, smeenai, phosek, sylvestre.ledru

Subscribers: arphaman, cfe-commits, llvm-commits

Tags: #clang

Differential Revision: https://reviews.llvm.org/D63503

llvm-svn: 365092
2019-07-03 22:45:55 +00:00
Fangrui Song e183340c29 Recommit [Object] Change object::SectionRef::getContents() to return Expected<StringRef>
r360876 didn't fix 2 call sites in clang.

Expected<ArrayRef<uint8_t>> may be better but use Expected<StringRef> for now.

Follow-up of D61781.

llvm-svn: 360892
2019-05-16 13:24:04 +00:00
Yaxun Liu 071949c3af Allow bundle size to be 0 in clang-offload-bundler
HIP uses clang-offload-bundler to create fat binary. The bundle for host is empty.
Currently clang-offload-bundler checks if the bundle size is 0 when unbundling.
If so it will exit without unbundling the remaining bundles. This causes
clang-offload-bundler not being able to unbundle fat binaries generated for HIP.

This patch allows bundles size to be 0 when clang-offload-bundler unbundles
input files.

Differential Revision: https://reviews.llvm.org/D58057

llvm-svn: 355419
2019-03-05 17:52:32 +00:00
Chandler Carruth 2946cd7010 Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636
2019-01-19 08:50:56 +00:00
Zachary Turner 08426e1f9f Refactor ExecuteAndWait to take StringRefs.
This simplifies some code which had StringRefs to begin with, and
makes other code more complicated which had const char* to begin
with.

In the end, I think this makes for a more idiomatic and platform
agnostic API.  Not all platforms launch process with null terminated
c-string arrays for the environment pointer and argv, but the api
was designed that way because it allowed easy pass-through for
posix-based platforms.  There's a little additional overhead now
since on posix based platforms we'll be takign StringRefs which
were constructed from null terminated strings and then copying
them to null terminate them again, but from a readability and
usability standpoint of the API user, I think this API signature
is strictly better.

llvm-svn: 334518
2018-06-12 17:43:52 +00:00
Yaxun Liu 609f752929 [HIP] Let clang-offload-bundler support HIP
When bundle/unbundle intermediate files for HIP, there may be multiple
sub archs, therefore BoundArch needs to be included in the target
and output file names for clang-offload-bundler.

Differential Revision: https://reviews.llvm.org/D46473

llvm-svn: 332121
2018-05-11 19:02:18 +00:00
Adrian Prantl 9fc8faf9e6 Remove \brief commands from doxygen comments.
This is similar to the LLVM change https://reviews.llvm.org/D46290.

We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.

Patch produced by

for i in $(git grep -l '\@brief'); do perl -pi -e 's/\@brief //g' $i & done
for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done

Differential Revision: https://reviews.llvm.org/D46320

llvm-svn: 331834
2018-05-09 01:00:01 +00:00
Rafael Espindola 76c8f82e08 Update for llvm change. NFC.
llvm-svn: 325156
2018-02-14 19:11:37 +00:00
Hiroshi Inoue 1019f8a98e [NFC] fix trivial typos in comments
"to to" -> "to"

llvm-svn: 323627
2018-01-29 05:15:18 +00:00