Commit Graph

2920 Commits

Author SHA1 Message Date
Lang Hames f9b5f6937e [JITLink][ELF/x86-64] Range check 32-bit relocs.
Also switch to using little_<b> / ulittle_<b> types to write results for
consistency with MachO.
2021-01-22 15:59:19 +11:00
Lang Hames 95b63c7b13 [ORC] Move LookupRequest from OrcShared to Orc.
It depends on Orc types (SymbolLookupSet), so can't be part of OrcShared.
2021-01-19 20:23:47 +11:00
Lang Hames 24672ddea3 [ORC] Move OrcError.h to include/llvm/ExecutionEngine/Orc/Shared.
OrcShared is the correct home for this header since Orc was split in
1d0676b54c. (It should have been moved in that commit, but was overlooked).
2021-01-19 16:18:00 +11:00
Lang Hames a817f46d50 [JITLink][ELF] Skip DWARF sections in ELF objects.
This matches current JITLink/MachO behavior and avoids processing currently
unsupported relocations.
2021-01-18 12:42:48 +11:00
Kazu Hirata 352fcfc697 [llvm] Use llvm::sort (NFC) 2021-01-17 10:39:45 -08:00
Stefan Gränitz cf905274c6 [Orc] Allow LLJITBuilder's CreateObjectLinkingLayer to return errors
It can be useful for an ObjectLinkingLayerCreator to allow callee errors to get propagated to the builder. Specifically, this is the case when the ObjectLayer uses the EHFrameRegistrationPlugin, because it requires a TPCEHFrameRegistrar and instantiation for it may fail (e.g. if the required registration symbols are missing in the target process).

Reviewed By: lhames

Differential Revision: https://reviews.llvm.org/D94690
2021-01-15 12:53:41 +01:00
Stefan Gränitz a5eb9df1e3 [Orc][NFC] Turn LLJIT member ObjTransformLayer into unique_ptr
All other layers in LLJIT are stored as unique_ptr's already. At this point, it is not strictly necessary for ObjTransformLayer, but it makes a follow-up change more straightforward.

Reviewed By: lhames

Differential Revision: https://reviews.llvm.org/D94689
2021-01-15 12:53:24 +01:00
Kazu Hirata 5c1c39e8d8 [llvm] Use *Set::contains (NFC) 2021-01-13 19:14:41 -08:00
Kazu Hirata 12fc9ca3a4 [llvm] Remove redundant string initialization (NFC)
Identified with readability-redundant-string-init.
2021-01-12 21:43:46 -08:00
Lang Hames cd8a80de96 [Orc] Add a unit test for asynchronous definition generation. 2021-01-13 14:23:36 +11:00
Kazu Hirata 89e8eb946d [llvm] Use llvm::find_if (NFC) 2021-01-11 18:48:06 -08:00
Lang Hames ef50c07b1f [JITLink] Add a new PostAllocationPasses list.
Passes in the new PostAllocationPasses list will run immediately after memory
allocation and address assignment for defined symbols, and before
JITLinkContext::notifyResolved is called. These passes can set up state
associated with the addresses of defined symbols before any query for these
addresses completes.
2021-01-12 11:57:07 +11:00
Lang Hames 7b11f564dc [JITLink] Rename PostAllocationPasses to PreFixupPasses.
PreFixupPasses better reflects when these passes will run.

A future patch will (re)introduce a PostAllocationPasses list that will run
after allocation, but before JITLinkContext::notifyResolved is called to notify
the rest of the JIT about the resolved symbol addresses.
2021-01-11 18:33:50 +11:00
Kazu Hirata 1d0bc05551 [llvm] Use llvm::append_range (NFC) 2021-01-06 18:27:33 -08:00
Kazu Hirata eb198f4c3c [llvm] Use llvm::any_of (NFC) 2021-01-04 11:42:47 -08:00
Lang Hames a554cd6ae5 [RuntimeDyld] Fix dangling reference in RuntimeDyldELF.
Patch by Moritz Sichert. Thanks Moritz!

Differential Revision: https://reviews.llvm.org/D89373
2021-01-03 10:20:36 +11:00
Brandon Bergren 8f004471c2 [PowerPC] Add the LLVM triple for powerpcle [1/5]
Add a triple for powerpcle-*-*.

This is a little-endian encoding of the 32-bit PowerPC ABI, useful in certain niche situations:

1) A loader such as the FreeBSD loader which will be loading a little endian kernel. This is required for PowerPC64LE to load properly in pseries VMs.
Such a loader is implemented as a freestanding ELF32 LSB binary.

2) Userspace emulation of a 32-bit LE architecture such as x86 on 64-bit hosts such as PowerPC64LE with tools like box86 requires having a 32-bit LE toolchain and library set, as they operate by translating only the main binary and switching to native code when making library calls.

3) The Void Linux for PowerPC project is experimenting with running an entire powerpcle userland.

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D93918
2021-01-02 12:17:22 -06:00
Lang Hames 5efc71e119 [ORC] Move Orc RPC code into Shared, rename some RPC types.
Moves all headers from Orc/RPC to Orc/Shared, and from the llvm::orc::rpc
namespace into llvm::orc::shared. Also renames RPCTypeName to
SerializationTypeName and Function to RPCFunction.

In addition to being a more reasonable home for this code, this will make it
easier for the upcoming Orc runtime to re-use the Serialization system for
creating and parsing wrapper-function binary blobs.
2020-12-30 12:48:20 +11:00
Georgii Rymar 438bc157a4 [libObject] - Add more ELF types to LLVM_ELF_IMPORT_TYPES_ELFT define (ELFTypes.h).
This allows to get rid of lots for typedefs/usings from many places.

Differential revision: https://reviews.llvm.org/D93801
2020-12-25 11:39:05 +03:00
Kazu Hirata 9939cf5a56 [ExecutionEngine, Linker] Use erase_if (NFC) 2020-12-23 21:44:39 -08:00
Kazu Hirata ce94e7d867 [MCA, ExecutionEngine, Object] Use llvm::is_contained (NFC) 2020-12-18 09:09:04 -08:00
Lang Hames ec6b71df70 [JITLink][ORC] Enable creation / linking of raw jitlink::LinkGraphs.
Separates link graph creation from linking. This allows raw LinkGraphs to be
created and passed to a link. ObjectLinkingLayer is updated to support emission
of raw LinkGraphs in addition to object buffers.

Raw LinkGraphs can be created by in-memory compilers to bypass object encoding /
decoding (though this prevents caching, as LinkGraphs have do not have an
on-disk representation), and by utility code to add programatically generated
data structures to the JIT target process.
2020-12-16 14:01:50 +11:00
Lang Hames 04795ab836 Re-apply 8904ee8ac7 with missing header included this time. 2020-12-14 13:39:33 +11:00
Nico Weber 5b112bcc0d Revert "[JITLink] Add JITLinkDylib type, thread through JITLinkMemoryManager APIs."
This reverts commit 8904ee8ac7.
Didn't `git add` llvm/ExecutionEngine/JITLink/JITLinkDylib.h and hence doesn't
build anywhere.
2020-12-13 21:30:38 -05:00
Lang Hames 8904ee8ac7 [JITLink] Add JITLinkDylib type, thread through JITLinkMemoryManager APIs.
JITLinkDylib represents a target dylib for a JITLink link. By representing this
explicitly we can:
  - Enable JITLinkMemoryManagers to manage allocations on a per-dylib basis
    (e.g by maintaining a seperate allocation pool for each JITLinkDylib).
  - Enable new features and diagnostics that require information about the
    target dylib (not implemented in this patch).
2020-12-14 12:29:16 +11:00
Lang Hames 0207de0bfe [ORC] Prefer preincrement on iterator. 2020-12-14 12:00:21 +11:00
Lang Hames 5bb28fa0f5 [JITLink][ELF] Reformat/add debug logging in ELF_x86_64.cpp.
Moves symbol name to the end of the output and makes other columns fixed width
so that they line up.
2020-12-10 18:46:44 +11:00
Alexey Bader be9b4bbdfc [MCJIT] Add cmake variables to customize ittapi git location and revision.
To support llorg builds this patch provides the following changes:

1)  Added cmake variable ITTAPI_GIT_REPOSITORY to control the location of ITTAPI repository.
     Default value of ITTAPI_GIT_REPOSITORY is github location: https://github.com/intel/ittapi.git
     Also, the separate cmake variable ITTAPI_GIT_TAG was added for repo tag.
2)  Added cmake variable ITTAPI_SOURCE_DIR to control the place where the repo will be cloned.
     Default value of ITTAPI_SOURCE_DIR is build area: PROJECT_BINARY_DIR

Reviewed By: etyurin, bader

Patch by ekovanov.

Differential Revision: https://reviews.llvm.org/D91935
2020-12-09 21:04:24 +03:00
Raphael Isemann a2c157eb3e [cmake] Make ExecutionEngine/Orc/Shared depend on intrinsics_gen to fix modules build
The LLVM_ENABLE_MODULES builds currently randomly fail due depending on the
headers generated by the intrinsics_gen target, but the current dependency only model
the non-modules dependencies:

```
While building module 'LLVM_ExecutionEngine' imported from llvm-project/llvm/lib/ExecutionEngine/Orc/Shared/TargetProcessControlTypes.cpp:13:
While building module 'LLVM_intrinsic_gen' imported from llvm-project/llvm/include/llvm/ExecutionEngine/Orc/ThreadSafeModule.h:17:
In file included from <module-includes>:1:
In file included from llvm-project/llvm/include/llvm/IR/Argument.h:18:
llvm/include/llvm/IR/Attributes.h:75:14: fatal error: 'llvm/IR/Attributes.inc' file not found
    #include "llvm/IR/Attributes.inc"
             ^~~~~~~~~~~~~~~~~~~~~~~~
```

Depending on whether intrinsics_gen runs before compiling Orc/Shared files we either fail or include an outdated Attributes.inc
in module builds. The Clang modules require these additional dependencies as including/importing one module requires all
includes headers by that module to be parsable.

Differential Revision: https://reviews.llvm.org/D92873
2020-12-08 20:41:35 +01:00
Stefan Gränitz c54d827fdb [Orc] Two small fixes in TPCDynamicLibrarySearchGenerator
There is one result per lookup symbol, so we have to advance the result iterator no matter whether it's NULL or not.
MissingSymbols variable is unused.

Reviewed By: lhames

Differential Revision: https://reviews.llvm.org/D91707
2020-12-08 10:58:20 +01:00
Stefan Gränitz 83b52b5ba2 [JITLink][ELF] Route objects to their matching linker backends based on header info
Distinguish objects by target properties address size, endian and machine architecture. So far we only
support x86-64 (ELFCLASS64, ELFDATA2LSB, EM_X86_64).

Reviewed By: lhames

Differential Revision: https://reviews.llvm.org/D90860
2020-12-08 10:56:01 +01:00
Chris Sears 9737c128f1 [llvmbuildectomy] removed vestigial LLVMBuild.txt files
LLVMBuild has been removed from the build system. However, three LLVMBuild.txt
files remain in the tree. This patch simply removes them.

llvm/lib/ExecutionEngine/Orc/TargetProcess/LLVMBuild.txt
llvm/tools/llvm-jitlink/llvm-jitlink-executor/LLVMBuild.txt
llvm/tools/llvm-profgen/LLVMBuild.txt

Differential Revision: https://reviews.llvm.org/D92693
2020-12-05 22:00:22 +01:00
Georgii Rymar ffbce65f95 [lib/Object, tools] - Make ELFObjectFile::getELFFile return reference.
We always have an object, so we don't have to return a pointer.

Differential revision: https://reviews.llvm.org/D92560
2020-12-04 16:02:29 +03:00
Sylvestre Ledru fa39b9eab1 TargetProcessControl.cpp - Remove warning: extra ‘;’ 2020-11-27 18:19:58 +01:00
serge-sans-paille 733f7b5084 Revert "[build] normalize components dependencies"
This reverts commit c6ef6e1690.

Basically, publicly linked libraries have a different semantic than components,
which link libraries privately.

Differential Revision: https://reviews.llvm.org/D91461
2020-11-18 19:23:11 +01:00
serge-sans-paille c6ef6e1690 [build] normalize components dependencies
Use LINK_COMPONENTS instead of explicit target_link_libraries for components.
This avoids redundancy and potential inconsistencies.

Differential Revision: https://reviews.llvm.org/D91461
2020-11-17 10:42:34 +01:00
Lang Hames 22e44358d3 [ORC] Include config.h in RegisterEHFrames.cpp.
RegisterEHFrames.cpp needs access to the HAVE_REGISTER_FRAME /
HAVE_DEREGISTER_FRAME defines.

rdar://71458921
2020-11-17 14:18:04 +11:00
Lang Hames f62e5f4569 [MCJIT] Profile the code generated by MCJIT engine using Intel VTune profiler
Patch by Elena Kovanova. Thanks Elena!

Problem:

LLVM already has a feature to profile the JIT-compiled code with VTune. This is
done using Intel JIT Profiling API (https://github.com/intel/ittapi). Function
information is captured by VTune as soon as the function is JIT-compiled. We
tried to use the same approach to report the function information generated by
the MCJIT engine – read parsing the debug information for in-memory ELF module
and report it using JIT API. As the results, we figured out that it did not work
properly for the following cases: inline functions, the functions located in
multiple source files, the functions having several bodies (address ranges).

Solution:

To overcome limitations described above, we have introduced new APIs as a part
of Intel ITT APIs to report the entire in-memory ELF module to be further
processed as regular ELF binaries with debug information.

This patch

1. Switches LLVM to open source version of Intel ITT/JIT APIs
(https://github.com/intel/ittapi) to keep it always up to date.

2. Adds support of profiling the code generated by MCJIT engine using Intel
VTune profiler

Another separate patch will get rid of obsolete Intel ITT APIs stuff, having
LLVM already switched to https://github.com/intel/ittapi.

Differential Revision: https://reviews.llvm.org/D86435
2020-11-16 19:28:14 +11:00
serge-sans-paille b8ec05a533 llvmbuildectomy - remove conflict file 2020-11-13 15:56:05 +01:00
serge-sans-paille 9218ff50f9 llvmbuildectomy - replace llvm-build by plain cmake
No longer rely on an external tool to build the llvm component layout.

Instead, leverage the existing `add_llvm_componentlibrary` cmake function and
introduce `add_llvm_component_group` to accurately describe component behavior.

These function store extra properties in the created targets. These properties
are processed once all components are defined to resolve library dependencies
and produce the header expected by llvm-config.

Differential Revision: https://reviews.llvm.org/D90848
2020-11-13 10:35:24 +01:00
Lang Hames 98f70e94e0 [ORC] Add dependence of OrcJIT on OrcTargetProcess.
The SelfTargetProcessControl class depends on OrcTargetProcess.
2020-11-13 18:09:41 +11:00
Lang Hames bdf26d8d19 [ORC] Remove designated initializer. 2020-11-13 17:12:33 +11:00
Lang Hames 1d0676b54c [ORC] Break up OrcJIT library, add Orc-RPC based remote TargetProcessControl
implementation.

This patch aims to improve support for out-of-process JITing using OrcV2. It
introduces two new class templates, OrcRPCTargetProcessControlBase and
OrcRPCTPCServer, which together implement the TargetProcessControl API by
forwarding operations to an execution process via an Orc-RPC Endpoint. These
utilities are used to implement out-of-process JITing from llvm-jitlink to
a new llvm-jitlink-executor tool.

This patch also breaks the OrcJIT library into three parts:
  -- OrcTargetProcess: Contains code needed by the JIT execution process.
  -- OrcShared: Contains code needed by the JIT execution and compiler
     processes
  -- OrcJIT: Everything else.

This break-up allows JIT executor processes to link against OrcTargetProcess
and OrcShared only, without having to link in all of OrcJIT. Clients executing
JIT'd code in-process should start linking against OrcTargetProcess as well as
OrcJIT.

In the near future these changes will enable:
  -- Removal of the OrcRemoteTargetClient/OrcRemoteTargetServer class templates
     which provided similar functionality in OrcV1.
  -- Restoration of Chapter 5 of the Building-A-JIT tutorial series, which will
     serve as a simple usage example for these APIs.
  -- Implementation of lazy, cross-target compilation in lli's -jit-kind=orc-lazy
     mode.
2020-11-13 17:05:13 +11:00
David Tenty ae032e2714 [CMake][ExecutionEngine] add HAVE_(DE)REGISTER_FRAME as a config.h macros
The macro HAVE_EHTABLE_SUPPORT is used by parts of ExecutionEngine to tell __register_frame/__deregister_frame is available to register the
FDE for a generated (JIT) code. It's currently set by a slowly growing set of macro tests in the respective headers, which is updated now and then when it fails to link on some platform or another due to the symbols being missing (see for example https://bugs.llvm.org/show_bug.cgi?id=5715).

This change converts the macro in two HAVE_(DE)REGISTER_FRAME config.h macros (like most of the other HAVE_* macros) and set's them based on whether CMake can actually find a definition for these symbols to link to at configuration time.

Reviewed By: hubert.reinterpretcast

Differential Revision: https://reviews.llvm.org/D87114
2020-11-10 13:09:44 -05:00
Valentin Churavy 85f4be0912 [RTDYLD] support absolute relocations where needed
These appear in some sections, such as DWARF tables, since
RuntimeDyldELF explicitly maps to this as a sentinel value:
29d1fba7b5/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldELF.cpp (L1199)

That could then be a source of problems if it tried to examine these
sections (for example, with either setProcessAllSections(true) or ORCv2 on i686).

Replaces https://reviews.llvm.org/D89241

Reviewed By: lhames, vchuravy

Differential Revision: https://reviews.llvm.org/D90722
2020-11-06 14:08:59 -05:00
Stefan Gränitz b397795f1a [JITLink][ELF] Implement R_X86_64_PLT32 relocations
Basic implementation for call and jmp branches with 32 bit offset. Branches to local targets produce
Branch32 edges that are resolved like a regular PCRel32 relocations. Branches to external (undefined)
targets produce Branch32ToStub edges and go through a PLT entry by default. If the target happens to
get resolved within the 32 bit range from the callsite, the edge is relaxed during post-allocation
optimization. There is a test for each of these cases.

Reviewed By: lhames

Differential Revision: https://reviews.llvm.org/D90331
2020-11-03 12:05:54 +00:00
Simon Pilgrim d64ea0f189 Remove superfluous whitespace around if(). NFC. 2020-10-25 14:38:16 +00:00
Fangrui Song f04d92af94 [X86] Produce R_X86_64_GOTPCRELX for test/binop instructions (MOV32rm/TEST32rm/...) when -Wa,-mrelax-relocations=yes is enabled
We have been producing R_X86_64_REX_GOTPCRELX (MOV64rm/TEST64rm/...) and
R_X86_64_GOTPCRELX for CALL64m/JMP64m without the REX prefix since 2016 (to be
consistent with GNU as), but not for MOV32rm/TEST32rm/...
2020-10-24 15:14:17 -07:00
Stefan Gränitz 66abe650ff Reapply "[jitlink][ELF] Add zero-fill blocks for symbols in section SHN_COMMON"
Root cause of the test failure was fixed with:
[JITLink][ELF] PCRel32GOTLoad edge offset can be smaller three

This reverts commit 10b1a61baf.
2020-10-24 16:58:06 +02:00
Stefan Gränitz b6ef40891c [JITLink][ELF] PCRel32GOTLoad edge offset can be smaller three
Offset is 2 for MOVL instruction in test ELF_x86-64_common. This should fix the test failures.

Differential Revision: https://reviews.llvm.org/D89795
2020-10-24 16:57:48 +02:00
Stefan Gränitz 10b1a61baf Revert "[jitlink][ELF] Add zero-fill blocks for symbols in section SHN_COMMON"
This reverts commit e9955b0843. Cannot reproduce the buildbot failures yet. Reverting in the meantime.
2020-10-24 15:43:06 +02:00
Stefan Gränitz e9955b0843 [jitlink][ELF] Add zero-fill blocks for symbols in section SHN_COMMON
Symbols with special section index SHN_COMMON (0xfff2) haven't been handled so far and caused an invalid section error.

This is a more or less straightforward use of the code commented out at the end of the function. I checked with the ELF spec, that the symbol value gives the alignment.

Reviewed By: lhames

Differential Revision: https://reviews.llvm.org/D89795
2020-10-24 14:54:38 +02:00
Stefan Gränitz 138b9f1928 [JITLink][ELF] PCRel32GOTLoad relocations are resolved like regular PCRel32 ones
The difference is that the former are indirect and go to the GOT while the latter go to the target directly. This info can be used to relax indirect ones that don't need the GOT (because the target is in range). We check for this optimization beforehand. For formal correctness and to avoid confusion, we should only change the relocation kind if we actually apply the relaxation.
2020-10-24 14:54:38 +02:00
Geoffrey Martin-Noble bfb04aeb85 Unconditionally #include <future>
This unbreaks building with `LLVM_ENABLE_THREADS=0`. Since
https://github.com/llvm/llvm-project/commit/069919c9ba33 usage of
`std::promise` is not guarded by `LLVM_ENABLE_THREADS`, so this header
must be unconditionally included.

Reviewed By: lhames

Differential Revision: https://reviews.llvm.org/D89758
2020-10-23 19:17:37 +00:00
Lang Hames 996a8b42aa Re-apply "[JITLink][ELF] Add support for ELF::R_X86_64_REX_GOTPCRELX relocation"
This re-applies e2fceec2fd with fixes. Apparently we already *do* support
relaxation for ELF, so we need to make sure the test case allocates a slab at
a fixed address, and that the R_X86_64_REX_GOTPCRELX test references an external
that is guaranteed to be out of range.
2020-10-23 09:48:05 -07:00
Lang Hames 0f910387e6 Revert "[JITLink][ELF] Add support for ELF::R_X86_64_REX_GOTPCRELX relocation."
This reverts commit e2fceec2fd.

This commit broke one of the bots. Reverting while I investigate.
2020-10-22 23:21:29 -07:00
Lang Hames e2fceec2fd [JITLink][ELF] Add support for ELF::R_X86_64_REX_GOTPCRELX relocation.
No support for relaxation yet -- this will always use the GOT entry.
2020-10-22 22:43:16 -07:00
Lang Hames 9898d9d885 [ORC] Fix a missing include. 2020-10-19 12:13:55 -07:00
Lang Hames c89447b659 [ORC] Fix unused variable warning. 2020-10-19 09:06:33 -07:00
Lang Hames f35707047e [ORC] Break up C-API header Orc.h, and add JITEventListener support.
This patch breaks Orc.h up into Orc.h, LLJIT.h and OrcEE.h.

Orc.h contain core Orc utilities.
LLJIT.h contains LLJIT specific types and functions.
OrcEE.h contains types and functions that depend on ExecutionEngine.

The intent is that these headers should match future library divisions: Clients
who only use Orc.h should only need to link againt the Orc core libraries,
clients using LLJIT.h will also need to link against LLVM core, and clients
using OrcEE.h will also have to link against ExecutionEngine.

In addition to breaking up the Orc.h header this patch introduces functions to:
(1) Set the object linking layer creation function on LLJITBuilder.
(2) Create an RTDyldObjectLinkingLayer instance (particularly for use in (1)).
(3) Register JITEventListeners with an RTDyldObjectLinkingLayer.

Together (1), (2) and (3) can be used to force use of RTDyldObjectLinkingLayer
as the underlying JIT linker for LLJIT, rather than the platform default, and
to register event listeners with the RTDyldObjectLinkingLayer.
2020-10-19 01:59:04 -07:00
Lang Hames 00369849e1 [ORC] Add function to get pool entry string.
Patch by Andres Freund. Thanks Andres!
2020-10-19 01:59:04 -07:00
Lang Hames 24afffe63a [ORC] Add C API support for defining absolute symbols.
Also tweaks the definition of TryToGenerate to make it dovetail more neatly
with the new function.
2020-10-19 01:59:04 -07:00
Lang Hames b6ca0c7dd5 [ORC] Add support for custom generators to the C bindings.
C API clients can now define a custom definition generator by providing a
callback function (to implement DefinitionGenerator::tryToGenerate) and context
object. All arguments for the DefinitionGenerator::tryToGenerate method have
been given C API counterparts, and the API allows for optionally asynchronous
generation.
2020-10-19 01:59:04 -07:00
Lang Hames 91d1f417fd [ORC] Add basic ResourceTracker support to the OrcV2 C Bindings.
Based on a patch by Andres Freund. Thanks Andres!
2020-10-19 01:59:04 -07:00
Lang Hames 49c065ae70 [ORC] Rename LLVMOrcJITDylibDefinitionGeneratorRef.
The DefinitionGenerator class has been moved out of JITDylib. This updates
the C API type and function names to reflect that.
2020-10-19 01:59:04 -07:00
Lang Hames 40f3fb52f7 [ORC] Fix C API function name.
Patch by Andres Freund. Thanks Andres!
2020-10-19 01:59:03 -07:00
Lang Hames 35e48d7b91 [ORC] Add C API functions to obtain and clear the symbol string pool.
Symbol string pool entries are ref counted, but not automatically cleared.
This can cause the size of the pool to grow without bound if it's not
periodically cleared. These functions allow that to be done via the C API.
2020-10-19 01:59:03 -07:00
Lang Hames 14cb9b4e21 [ORC] Add a C API function to set the ExecutionSession error reporter. 2020-10-19 01:59:03 -07:00
Lang Hames c88d9eae8a [ORC] Fix a memory leak in the OrcV2 C API (and some comment typos).
The LLVMOrcLLJITAddLLVMIRModule function was leaking its
LLVMOrcThreadSafeModuleRef argument. Wrapping the argument in a unique_ptr
fixes this.
2020-10-19 01:59:03 -07:00
Lang Hames 069919c9ba [ORC] Update Symbol Lookup / DefinitionGenerator system.
This patch moves definition generation out from the session lock, instead
running it under a per-dylib generator lock. It also makes the
DefinitionGenerator::tryToGenerate method optionally asynchronous: Generators
are handed an opaque LookupState object which can be captured to stop/restart
the lookup process.

The new scheme provides the following benefits and guarantees:

(1) Queries that do not need to attempt definition generation (because all
    requested symbols matched against existing definitions in the JITDylib)
    can proceed without being blocked by any running definition generators.

(2) Definition generators can capture the LookupState to continue their work
    asynchronously. This allows generators to run for an arbitrary amount of
    time without blocking a thread. Definition generators that do not need to
    run asynchronously can return without capturing the LookupState to eliminate
    unnecessary recursion and improve lookup performance.

(3) Definition generators still do not need to worry about concurrency or
    re-entrance: Since they are still run under a (per-dylib) lock, generators
    will never be re-entered concurrently, or given overlapping symbol sets to
    generate.

Finally, the new system distinguishes between symbols that are candidates for
generation (generation candidates) and symbols that failed to match for a query
(due to symbol visibility). This fixes a bug where an unresolved symbol could
trigger generation of a duplicate definition for an existing hidden symbol.
2020-10-19 01:59:03 -07:00
Lang Hames 5d2e359ce6 [ORC] Move DefinitionGenerator out of JITDylib.
This will make it easier to implement asynchronous definition generators.
2020-10-19 01:59:03 -07:00
Lang Hames 680845ec0d [ORC] Move MaterializationResponsibility methods to ExecutionSession.
MaterializationResponsibility, JITDylib, and ExecutionSession collectively
manage the OrcV2 core JIT state. Responsibility for maintaining and
updating this state has previously been spread among these classes, resulting
in implementations that are each non-trivial, but all tightly coupled. This has
in turn made reading the code and reasoning about state update and locking
rules difficult.

The core state model can be simplified by thinking of
MaterializationResponsibility and JITDylib as facets of ExecutionSession. This
commit is the first in a series intended to refactor Core.cpp to reflect this
model. Operations on MaterializationResponsibility and JITDylib will forward to
implementation methods inside ExecutionSession. Raw state will remain with the
original classes, but in most cases will only be modified by the
ExecutionSession.
2020-10-19 01:59:03 -07:00
Lang Hames ad92f16ccc [ORC][examples] Update Kaleidoscope and BuildingAJIT tutorial series to OrcV2.
This patch updates the Kaleidoscope and BuildingAJIT tutorial series (chapter
1-4) to OrcV2. Chapter 5 of the BuildingAJIT series is removed -- it will be
re-instated once we have in-tree support for out-of-process JITing.

This patch only updates the tutorial code, not the text. Patches welcome for
that, otherwise I will try to update it in a few weeks.
2020-10-18 21:03:04 -07:00
Lang Hames 0aec49c853 [ORC] Add support for resource tracking/removal (removable code).
This patch introduces new APIs to support resource tracking and removal in Orc.
It is intended as a thread-safe generalization of the removeModule concept from
OrcV1.

Clients can now create ResourceTracker objects (using
JITDylib::createResourceTracker) to track resources for each MaterializationUnit
(code, data, aliases, absolute symbols, etc.) added to the JIT. Every
MaterializationUnit will be associated with a ResourceTracker, and
ResourceTrackers can be re-used for multiple MaterializationUnits. Each JITDylib
has a default ResourceTracker that will be used for MaterializationUnits added
to that JITDylib if no ResourceTracker is explicitly specified.

Two operations can be performed on ResourceTrackers: transferTo and remove. The
transferTo operation transfers tracking of the resources to a different
ResourceTracker object, allowing ResourceTrackers to be merged to reduce
administrative overhead (the source tracker is invalidated in the process). The
remove operation removes all resources associated with a ResourceTracker,
including any symbols defined by MaterializationUnits associated with the
tracker, and also invalidates the tracker. These operations are thread safe, and
should work regardless of the the state of the MaterializationUnits. In the case
of resource transfer any existing resources associated with the source tracker
will be transferred to the destination tracker, and all future resources for
those units will be automatically associated with the destination tracker. In
the case of resource removal all already-allocated resources will be
deallocated, any if any program representations associated with the tracker have
not been compiled yet they will be destroyed. If any program representations are
currently being compiled then they will be prevented from completing: their
MaterializationResponsibility will return errors on any attempt to update the
JIT state.

Clients (usually Layer writers) wishing to track resources can implement the
ResourceManager API to receive notifications when ResourceTrackers are
transferred or removed. The MaterializationResponsibility::withResourceKeyDo
method can be used to create associations between the key for a ResourceTracker
and an allocated resource in a thread-safe way.

RTDyldObjectLinkingLayer and ObjectLinkingLayer are updated to use the
ResourceManager API to enable tracking and removal of memory allocated by the
JIT linker.

The new JITDylib::clear method can be used to trigger removal of every
ResourceTracker associated with the JITDylib (note that this will only
remove resources for the JITDylib, it does not run static destructors).

This patch includes unit tests showing basic usage. A follow-up patch will
update the Kaleidoscope and BuildingAJIT tutorial series to OrcV2 and will
use this API to release code associated with anonymous expressions.
2020-10-18 21:02:54 -07:00
Lang Hames 6154c4115c [ORC] Remove OrcV1 APIs.
This removes all legacy layers, legacy utilities, the old Orc C bindings,
OrcMCJITReplacement, and OrcMCJITReplacement regression tests.

ExecutionEngine and MCJIT are not affected by this change.
2020-10-18 21:02:44 -07:00
Lang Hames b45b5166f8 [RuntimeDyld][COFF] Report fatal error on error, rather than emiting diagnostic.
Report a fatal error if an IMAGE_REL_AMD64_ADDR32NB cannot be applied due to an
out-of-range target. Previously we emitted a diagnostic to llvm::errs and
continued.

Patch by Dale Martin. Thanks Dale!
2020-10-06 15:16:29 -07:00
Lang Hames 01baeda7ca [JITLink][ELF] Handle BSS sections, improve some error messages.
This patch enables basic BSS section handling, and improves a couple of error
messages in the ELF section parsing code.

Patch by Christian Schafmeister. Thanks Christian!

Differential Revision: https://reviews.llvm.org/D88867
2020-10-05 21:35:35 -07:00
Lang Hames 9a0d1b6673 [ORC] Add operations to create and lookup JITDylibs to OrcV2 C bindings. 2020-09-16 13:49:30 -07:00
Georgii Rymar 4845531fa8 [lib/Object] - Refine interface of ELFFile<ELFT>. NFCI.
`ELFFile<ELFT>` has many methods that take pointers,
though they assume that arguments are never null and
hence could take references instead.

This patch performs such clean-up.

Differential revision: https://reviews.llvm.org/D87385
2020-09-15 11:38:31 +03:00
Lang Hames 783ba64a89 [JITLink] Improve formatting for Edge, Block and Symbol debugging output. 2020-09-13 15:44:07 -07:00
Lang Hames 7dcd0042e8 Re-apply "[ORC] Make MaterializationResponsibility immovable..." with fixes.
Re-applies c74900ca67 with fixes for the ThinLtoJIT example.
2020-09-11 14:09:05 -07:00
Florian Hahn c0825fa5fc Revert "[ORC] Make MaterializationResponsibility immovable, pass by unique_ptr."
This reverts commit c74900ca67.

This appears to be breaking some builds on macOS and has been causing
build failures on Green Dragon (see below). I am reverting this for now,
to unblock testing on Green Dragon.

http://green.lab.llvm.org/green/job/clang-stage1-cmake-RA-incremental/18144/console

[65/187] /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++  -DBUILD_EXAMPLES -DGTEST_HAS_RTTI=0 -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Iexamples/ThinLtoJIT -I/Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-incremental/llvm-project/llvm/examples/ThinLtoJIT -Iinclude -I/Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-incremental/llvm-project/llvm/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -O3  -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -mmacosx-version-min=10.9    -fno-exceptions -fno-rtti -UNDEBUG -std=c++14 -MD -MT examples/ThinLtoJIT/CMakeFiles/ThinLtoJIT.dir/ThinLtoDiscoveryThread.cpp.o -MF examples/ThinLtoJIT/CMakeFiles/ThinLtoJIT.dir/ThinLtoDiscoveryThread.cpp.o.d -o examples/ThinLtoJIT/CMakeFiles/ThinLtoJIT.dir/ThinLtoDiscoveryThread.cpp.o -c /Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-incremental/llvm-project/llvm/examples/ThinLtoJIT/ThinLtoDiscoveryThread.cpp
FAILED: examples/ThinLtoJIT/CMakeFiles/ThinLtoJIT.dir/ThinLtoDiscoveryThread.cpp.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++  -DBUILD_EXAMPLES -DGTEST_HAS_RTTI=0 -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Iexamples/ThinLtoJIT -I/Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-incremental/llvm-project/llvm/examples/ThinLtoJIT -Iinclude -I/Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-incremental/llvm-project/llvm/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -O3  -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -mmacosx-version-min=10.9    -fno-exceptions -fno-rtti -UNDEBUG -std=c++14 -MD -MT examples/ThinLtoJIT/CMakeFiles/ThinLtoJIT.dir/ThinLtoDiscoveryThread.cpp.o -MF examples/ThinLtoJIT/CMakeFiles/ThinLtoJIT.dir/ThinLtoDiscoveryThread.cpp.o.d -o examples/ThinLtoJIT/CMakeFiles/ThinLtoJIT.dir/ThinLtoDiscoveryThread.cpp.o -c /Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-incremental/llvm-project/llvm/examples/ThinLtoJIT/ThinLtoDiscoveryThread.cpp
In file included from /Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-incremental/llvm-project/llvm/examples/ThinLtoJIT/ThinLtoDiscoveryThread.cpp:7:
/Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-incremental/llvm-project/llvm/examples/ThinLtoJIT/ThinLtoInstrumentationLayer.h:37:68: error: non-virtual member function marked 'override' hides virtual member function
  void emit(MaterializationResponsibility R, ThreadSafeModule TSM) override;
                                                                   ^
/Users/buildslave/jenkins/workspace/clang-stage1-cmake-RA-incremental/llvm-project/llvm/include/llvm/ExecutionEngine/Orc/Layer.h:103:16: note: hidden overloaded virtual function 'llvm::orc::IRLayer::emit' declared here: type mismatch at 1st parameter ('std::unique_ptr<MaterializationResponsibility>' vs 'llvm::orc::MaterializationResponsibility')
  virtual void emit(std::unique_ptr<MaterializationResponsibility> R,
               ^
1 error generated.
2020-09-11 09:35:20 +01:00
Lang Hames c74900ca67 [ORC] Make MaterializationResponsibility immovable, pass by unique_ptr.
Making MaterializationResponsibility instances immovable allows their
associated VModuleKeys to be updated by the ExecutionSession while the
responsibility is still in-flight. This will be used in the upcoming
removable code feature to enable safe merging of resource keys even if
there are active compiles using the keys being merged.
2020-09-10 13:21:46 -07:00
Lang Hames 3b64052a25 [ORC] Fix some bugs in TPCDynamicLibrarySearchGenerator, use in llvm-jitlink.
TPCDynamicLibrarySearchGenerator was generating errors on missing
symbols, but that doesn't fit the DefinitionGenerator contract: A symbol
that isn't generated by a particular generator should not cause an
error.

This commit fixes the error by using SymbolLookupFlags::WeaklyReferencedSymbol
for all elements of the lookup, and switches llvm-jitlink to use
TPCDynamicLibrarySearchGenerator.
2020-09-04 13:23:52 -07:00
Lang Hames bd09e0dd74 [ORC] Remove stray debugging output. 2020-09-01 20:53:49 -07:00
Lang Hames c40ce0da71 [ORC] Add an early out for MachOPlatform's init-scraper plugin setup.
If there's no initializer symbol in the current MaterializationResponsibility
then bail out without installing JITLink passes: they're going to be no-ops
anyway.
2020-09-01 20:12:23 -07:00
Lang Hames 7ff335a25f [ORC] Fix MachOPlatform's synthetic symbol dependence registration.
A think-o in the existing code meant that dependencies were never registered.
This failure could lead to crashes rather than orderly error propagation if
initialization dependencies failed to materialize.

No test case: The bug was discovered in an out-of-tree code and requires
pathalogically misconfigured JIT to generate the original error that lead to
the crash.
2020-09-01 20:12:23 -07:00
Lang Hames b79e19e6d6 [ORC] Remove an unused variable.
The unused Main variable was accidentally left in an earlier commit.
2020-08-31 15:35:55 -07:00
Sourabh Singh Tomar db464a2753 [NFCI] Silent a build warning due to an extra semi-colon 2020-08-31 17:49:31 +05:30
Lang Hames e1d5f7d003 [ORC] Add getDFSLinkOrder / getReverseDFSLinkOrder methods to JITDylib.
DFS and Reverse-DFS linkage orders are used to order execution of
deinitializers and initializers respectively.

This patch replaces uses of special purpose DFS order functions in
MachOPlatform and LLJIT with uses of the new methods.
2020-08-29 15:17:06 -07:00
Lang Hames 605df8112c [ORC][JITLink] Switch to unique ownership for EHFrameRegistrars.
This will make stateful registrars (e.g. a future TargetProcessControl based
registrar) easier to deal with.
2020-08-26 16:59:45 -07:00
Kai Nacke c2ae7934c8 [SystemZ/ZOS]__(de)register_frame are not available on z/OS.
The functions `__register_frame`/`__deregister_frame` are not
available on z/OS, so add a guard to not use them.

Reviewed By: lhames, abhina.sreeskantharajan

Differential Revision: https://reviews.llvm.org/D84787
2020-08-17 09:00:09 -04:00
Lang Hames a49b05bb61 [JITLink][MachO] Use correct symbol scope when N_PEXT is set and N_EXT unset.
MachOLinkGraphBuilder has been treating these as hidden, but they should be
treated as local.

Symbols with N_PEXT set and N_EXT unset are produced when hidden symbols are
run through 'ld -r' without passing -keep_private_externs. They will show up
under 'nm -m' as "was private extern", hence the name of the test cases.

Testcase commited as relocatable object to ensure that the test suite doesn't
depend on having 'ld -r' available.
2020-08-15 15:53:33 -07:00
Stefan Gränitz 9a47bcae7c [ORC][NFC] Refactor loop to determine name of init symbol in IRMaterializationUnit
This loop caused me a little headache once, because I didn't see the assigned variable is a member. The refactored version appears more readable to me.

Differential Revision: https://reviews.llvm.org/D85922
2020-08-14 11:34:44 +02:00
Stefan Gränitz 5bcd32b744 [ORC][NFC] Fix typo in comment 2020-08-13 21:14:20 +02:00
Stefan Gränitz f12db8cf75 [ORC] cloneToNewContext() can work with a const-ref to ThreadSafeModule 2020-08-13 21:01:21 +02:00
Lang Hames adaadbfeac [JITLink][MachO] Return an error when MachO TLV relocations are encountered.
MachO TLV relocations aren't supported yet. Error out rather than falling
through to llvm_unreachable.
2020-08-13 11:19:35 -07:00
Lang Hames 6fd30f0669 [llvm-jitlink] Update llvm-jitlink to use TargetProcessControl. 2020-08-10 17:19:48 -07:00
Lang Hames ba8683f292 [JITLink][MachO][AArch64] More PAGEOFF12 relocation fixes.
Correctly sign extend the addend, and fix implicit shift operand decoding
(it incorrectly returned 0 for some cases), and check that the initial
encoded immediate is 0.
2020-08-05 21:09:45 -07:00
Lang Hames 47cfffe893 [JITLink][AArch64] Handle addends on PAGE21 / PAGEOFF12 relocations. 2020-08-05 08:50:46 -07:00
Lang Hames d561d1bf96 [JITLink][AArch64] Improve debug output for addend relocations. 2020-08-05 08:50:46 -07:00
David Turner ba0e71432a Do not map read-only data memory sections with EXECUTE flags.
The code in SectionMemoryManager.cpp unnecessarily maps
read-only data sections with the READ+EXECUTE flags. This is
undesirable from a security stand-point.

Moreover, on the Fuchsia platform, which is now very strict
about mapping pages with the EXECUTE permission, this simply
fails, because the section's pages were initially allocated
with only the READ+WRITE flags.

A more detailed description of the issue can be found in this
public SwiftShader bug:

  https://issuetracker.google.com/issues/154586551

This patch just restrict the mapping to the READ flag for ROData
sections. Code sections are still mapped with READ+EXECUTE as
expected.

Reviewed By: lhames

Differential Revision: https://reviews.llvm.org/D78574
2020-08-05 10:51:48 +02:00
Lang Hames 777824b49d [llvm-jitlink] Add support for static archives and MachO universal archives.
Archives can now be specified as input files the same way that object
files are. Archives will always be linked after all objects (regardless
of the relative order of the inputs) but before any dynamic libraries or
process symbols.

This patch also relaxes matching for slice triples in
StaticLibraryDefinitionGenerator in order to support this feature:
Vendors need not match if the source vendor is unknown.
2020-08-03 12:58:00 -07:00
Lang Hames 0f5b70769d [llvm-jitlink] Add -phony-externals option to suppress unresolved externals.
The -phony-externals option adds a generator which explicitly defines any
otherwise unresolved externals as null. This transforms link-time
unresolved-symbol errors into potential runtime null pointer accesses
(if an unresolved external is actually accessed during execution).

This option can be useful in -harness mode to avoid having to mock a
large number of symbols that are not reachable at runtime (e.g. unused
methods referenced by a class vtable).
2020-08-01 18:33:44 -07:00
Hans Wennborg 6a3b07a4bf RuntimeDyldELF: report_fatal_error instead of asserting for unimplemented relocations (PR46816)
This fixes the ExecutionEngine/MCJIT/stubs-sm-pic.ll test in no-asserts
builds which is set to XFAIL on some platforms like 32-bit x86. More
importantly, we probably don't want to silently error in these cases.

Differential revision: https://reviews.llvm.org/D84390
2020-07-31 20:06:47 +02:00
Lang Hames 8ce8cee1e1 [llvm-jitlink] Add -harness option to llvm-jitlink.
The -harness option enables new testing use-cases for llvm-jitlink. It takes a
list of objects to treat as a test harness for any regular objects passed to
llvm-jitlink.

If any files are passed using the -harness option then the following
transformations are applied to all other files:

  (1) Symbols definitions that are referenced by the harness files are promoted
      to default scope. (This enables access to statics from test harness).

  (2) Symbols definitions that clash with definitions in the harness files are
      deleted. (This enables interposition by test harness).

  (3) All other definitions in regular files are demoted to local scope.
      (This causes untested code to be dead stripped, reducing memory cost and
      eliminating spurious unresolved symbol errors from untested code).

These transformations allow the harness files to reference and interpose
symbols in the regular object files, which can be used to support execution
tests (including fuzz tests) of functions in relocatable objects produced by a
build.
2020-07-30 15:26:19 -07:00
Lang Hames 9f1dcdca71 [JITLink] Allow JITLinkContext::notifyResolved to return an Error.
This allows clients to detect invalid transformations applied by JITLink passes
(e.g. inserting or removing symbols in unexpected ways) and terminate linking
with an error.

This change is used to simplify the error propagation logic in
ObjectLinkingLayer.
2020-07-30 15:26:18 -07:00
Lang Hames 47a40eda17 [ORC] Remove a redundant call to getTargetMemory. 2020-07-26 17:34:31 -07:00
Lang Hames a01c4ee71c [ORC] Rename TargetProcessControl DynamicLibraryHandle and loadLibrary.
The new names, DylibHandle and loadDylib, are more concise and make
clear that these utilities are for loading dynamic libraries, not static
ones.
2020-07-25 15:21:43 -07:00
Lang Hames 11d5316afd [ORC] Don't require PageSize or Triple during TargetProcessControl construction
Subclasses will commonly gather that information from a remote during
construction, in which case they won't have meaningful values to pass to
TargetProcessControl's constructor.
2020-07-25 15:21:43 -07:00
Lang Hames 69091eb1c4 [ORC] Enable use of TargetProcessControl::getMemMgr with ObjectLinkingLayer.
This patch makes ownership of the JITLinkMemoryManager by ObjectLinkingLayer
optional: the layer can still own the memory manager but no longer has to.

Evevntually we want to move to a state where ObjectLinkingLayer never owns its
memory manager. For now allowing optional ownership makes it easier to develop
classes that can dynamically use either RTDyldObjectLinkingLayer, which owns
its memory managers, or ObjectLinkingLayer (e.g. LLJIT).
2020-07-23 16:18:57 -07:00
Simon Pilgrim f758d72eb8 Speculation.h - remove unnecessary includes. NFC. 2020-07-23 11:58:23 +01:00
Lang Hames 13ad00be98 [ORC] Add a TargetProcessControl-based dynamic library search generator.
TPCDynamicLibrarySearchGenerator uses a TargetProcessControl instance to
load libraries and search for symbol addresses in a target process. It
can be used in place of a DynamicLibrarySearchGenerator to enable
target-process agnostic lookup.
2020-07-22 16:19:24 -07:00
Lang Hames 574713c307 [ExecutionEngine] Initialize near block hint in SectionMemoryManager.
When allocating a new memory block in SectionMemoryManager, initialize
the Near hint for the other memory groups if they have not been set
already.

Patch by Dana Koch. Thanks Dana!
2020-07-20 14:40:54 -07:00
Lang Hames 0d944e00ea [ORC] Refactor TrampolinePool to reduce virtual function calls.
Virtual function calls are now only made when the pool needs to be
grown to accommodate o new request.
2020-07-19 22:38:41 -07:00
Lang Hames cdd10ca280 [JITLink][MachO] Tidy up debugging output for relocation parsing.
Identify relocations by (section name, offset) pairs, rather than plain
vmaddrs. This makes it easier to cross-reference debugging output for
relocations with output from standard object inspection tools (otool,
readelf, objdump, etc.).
2020-07-19 19:45:50 -07:00
Lang Hames f7a571537a [JITLink][MachO] Fix handling of non-extern UNSIGNED pair of SUBTRACTOR relocs.
When processing a MachO SUBTRACTOR/UNSIGNED pair, if the UNSIGNED target
is non-extern then check the r_symbolnum field of the relocation to find
the targeted section and use the section's address to find 'ToSymbol'.

Previously 'ToSymbol' was found by loading the initial value stored at
the fixup location and treating this as an address to search for. This
is incorrect, however: the initial value includes the addend and will
point to the wrong block if the addend is less than zero or greater than
the block size.

rdar://65756694
2020-07-19 10:22:55 -07:00
Logan Smith 3ee7fe4cfd [llvm][NFC] Add missing 'override's 2020-07-17 17:35:59 -07:00
David Tenty 8dea7f3202 [z/OS][AIX] Move lambda definition to fix build problem
This is a follow on change to eed19bd8 and contains a fix for a build
failure that occurs on both z/OS and AIX as a result of this commit:

https://reviews.llvm.org/rG670915094462d831e3733e5b01a76471b8cf6dd8.
2020-07-17 10:08:01 -04:00
Benjamin Kramer 9a0689e072 Make helpers static. NFC. 2020-07-17 13:49:11 +02:00
Lang Hames 10056238ac [ORC] Switch from initializer lists to named arguments to work around MSVC.
MSVC doesn't like some of the initializer list uses in 0e940d55f8.
Switch to named arguments to work around this.
2020-07-16 15:58:31 -07:00
Lang Hames b0bc77380d [ORC] Add more explicit casts to fix a narrowing conversion errors. 2020-07-16 15:37:18 -07:00
Lang Hames 121302ac62 [ORC] Add explicit cast to fix a narrowing conversion error. 2020-07-16 15:33:02 -07:00
Lang Hames 0e940d55f8 [ORC] Add TargetProcessControl and TPCIndirectionUtils APIs.
TargetProcessControl is a new API for communicating with JIT target processes.
It supports memory allocation and access, and inspection of some process
properties, e.g. the target proces triple and page size.

Centralizing these APIs allows utilities written against TargetProcessControl
to remain independent of the communication procotol with the target process
(which may be direct memory access/allocation for in-process JITing, or may
involve some form of IPC or RPC).

An initial set of TargetProcessControl-based utilities for lazy compilation is
provided by the TPCIndirectionUtils class.

An initial implementation of TargetProcessControl for in-process JITing
is provided by the SelfTargetProcessControl class.

An example program showing how the APIs can be used is provided in
llvm/examples/OrcV2Examples/LLJITWithTargetProcessControl.
2020-07-16 15:09:13 -07:00
Jared Wyles ef454c5444 [jitlink] Adding support for PCRel32GOTLoad in ELF x86 for the jitlinker
Summary: This adds the basic support for GOT in elf x86.
Was able to just get away using the macho code by generalising the edges.
There will be a follow up patch to turn that into a generic utility for both of the x86 and Mach-O code.

This patch also lands support for relocations relative to symbol.

Reviewers: lhames

Subscribers: hiraditya, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D83748
2020-07-16 07:15:08 +10:00
Abhina Sreeskantharajan eed19bd844 [NFC] [AIX] [z/OS] Fix build failure on AIX and z/OS
Summary: This PR contains a build failure fix that occurs on both AIX and z/OS as a result of this commit https://reviews.llvm.org/rG670915094462d831e3733e5b01a76471b8cf6dd8.

Reviewers: uweigand, Kai, hubert.reinterpretcast, daltenty, lhames

Reviewed By: Kai, hubert.reinterpretcast, daltenty

Subscribers: SeanP, hiraditya, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D83889
2020-07-15 16:07:34 -04:00
Richard Smith 71292379d7 Make LazyCallThroughManager destructor virtual rather than arranging for
all clients to delete it via the correct dynamic type.
2020-07-14 18:41:07 -07:00
Richard Smith 099fd37484 Fix undefined behavior due to deleting an object with a non-virtual
destructor via a pointer of the wrong static type.

This caused crashes during deallocation in C++14 builds when using a
deallocator whose sized delete requires the size argument to be correct.

Also make the LazyCallThroughManager destructor protected to catch this
sort of bug in the future.
2020-07-14 17:17:20 -07:00
Lang Hames 3d931e85f1 [ORC] Don't take ownership of the trampoline pool in LazyReexportsManager.
LazyReexportsManager instances use the trampoline pool, but they don't need to
own it. Keeping TrampolinePool ownership separate allows re-use of the
trampoline pool by other clients.
2020-07-14 10:56:45 -07:00
Lang Hames 31b5b12129 [ORC] Remove a spurious reinterpret_cast. 2020-07-13 12:39:24 -07:00
Lang Hames fb7ef0bb0b [ORC] Generalize emit re-entry, stub, etc. APIs for working addr != link addr.
This patch generalizes the APIs for writing re-entry blocks, trampolines and
stubs to allow their final linked address to differ from the address of
their initial working memory. This will allow these routines to be used with
JITLinkMemoryManagers, which will in turn allow for unification of code paths
for in-process and cross-process lazy JITing.
2020-07-13 10:15:10 -07:00
Lang Hames 6709150944 [ORC] Modify LazyCallThroughManager to support asynchronous resolution.
Asynchronous resolution is a better fit for handling reentry over
IPC/RPC where we want to avoid blocking a communication handler/thread.
2020-07-08 21:13:55 -07:00
Christopher Tetreault 9b500e564a [SVE] Remove calls to VectorType::getNumElements from ExecutionEngine
Reviewers: efriedma, lhames, sdesmalen, fpetrogalli

Reviewed By: lhames, sdesmalen

Subscribers: tschuett, hiraditya, rkruppe, psnobl, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D82211
2020-06-30 11:05:38 -07:00
Guillaume Chatelet 368a5e3a66 [Alignment][NFC] migrate DataLayout::getPreferredAlignment
This patch is part of a series to introduce an Alignment type.
See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133851.html
See this patch for the introduction of the type: https://reviews.llvm.org/D64790

Differential Revision: https://reviews.llvm.org/D82752
2020-06-29 11:24:36 +00:00
Lang Hames bf783a6aa8 [JITLink] Display host -> target address mapping in debugging output.
This can be helpful for sanity checking JITLink memory manager behavior.
2020-06-19 10:05:02 -07:00
Lang Hames 5682f192bd [RuntimeDyld] Add dependence on Core.
Commit 498dd745f5 introduced a dependence on Core. This patch updates
LLVMbuild.txt to reflect this.
2020-06-15 11:14:27 -07:00
Lang Hames 498dd745f5 [ORC] Honor linker private global prefix on symbol names.
If a symbol name begins with the linker private global prefix (as
described by the DataLayout) then it should be treated as non-exported,
regardless of its LLVM IR visibility value.
2020-06-15 10:28:36 -07:00
Leonard Chan 6adc664b9d [llvm][ELF][AArch64] Handle R_AARCH64_PLT32 relocation
This patch allows for usage of the @PLT modifier in AArch64 assembly which
lowers to an R_AARCH64_PLT32 relocation. See D81184 for handling this
relocation in lld.

Differential Revision: https://reviews.llvm.org/D81446
2020-06-10 11:34:16 -07:00
Lang Hames e429cffd4f [ORC] Always use ObjectLinkingLayer/JITLink for MachO on x86-64 and arm64.
JITLink supports all code and relocation models, so there's no reason to
conditionalize using JITLink on the code or relocation model settings.

Clients wanting to use RTDyldObjectLinkingLayer/RuntimeDyld will now
need to use a custom object linking layer creator.
2020-06-05 21:02:29 -07:00
Lang Hames d27cbf03cf [JITLink] Skip debug sections in MachO objects.
Debug sections will not be linked into the final executable and may contain
ambiguous relocations*. Skipping them avoids both some unnecessary processing
cost and the hassle of dealing with the problematic relocations.

* E.g. __debug_ranges contains non-extern relocations to the end of functions
hat begin with named symbols. Under the usual rules for interpreting non-extern
relocations these will be incorrectly associated with the following block, or
no block at all (if there is a gap between one block and the next).
2020-06-03 11:08:14 -07:00
Jared Wyles 3f0841f6d0 [jitlink] R_X86_64_PC32 support for the elf x86 jitlinker
Summary:

Adding in our first relocation type, and all the required plumbing to support the rest in following patches

Differential Revision: https://reviews.llvm.org/D80613

Reviewer: lhames
2020-05-30 10:53:18 +10:00
Lang Hames ff92d3c672 [ORC] Add debugging output for LLJIT construction.
This can be handy for checking whether the LLJIT instance you're constructing
matches your expectations.
2020-05-28 20:31:50 -07:00
Lang Hames a6deaeec37 [JITLink] Improve llvm-jitlink regression testing support for ELF.
This patch adds a jitlink pass, 'registerELFGraphInfo', that records section
and symbol information about each LinkGraph in the llvm-jitlink session object.
This allows symbols and sections to be referred to by name in llvm-jitlink
regression tests. This will enable a testcase to be written for
https://reviews.llvm.org/D80613.
2020-05-28 20:31:50 -07:00
David Blaikie fca76b7945 Roll variables into an LLVM_DEBUG block to address -Wunused-but-set-variable 2020-05-26 12:05:08 -07:00
Simon Pilgrim c9797cf98b Fix msvc "switch statement contains 'default' but no 'case' labels" warning. NFC.
Stripped out the switch statement, but kept case labels as comments for future reference.
2020-05-22 10:49:10 +01:00
Lang Hames 2e40cf06df [JITLink] Initial implementation of ELF / x86-64 support for JITLink.
This initial implementation supports section and symbol parsing, but no
relocation support. It enables JITLink to link and execute ELF relocatable
objects that do not require relocations.

Patch by Jared Wyles. Thanks Jared!

Differential Revision: https://reviews.llvm.org/D79832
2020-05-21 21:44:00 -07:00
Lang Hames 41379f1ec4 [ORC] Share ownership of JITDylibs between ExecutionSession and
MaterializationResponsibility.

MaterializationResponsibility objects provide a connection between a
materialization process (compiler, jit linker, etc.) and the JIT state held in
the ExecutionSession and JITDylib objects. Switching to shared ownership
extends the lifetime of JITDylibs to ensure they remain accessible until all
materializers targeting them have completed. This will allow (in a follow-up
patch) JITDylibs to be removed from the ExecutionSession and placed in a
pending-destruction state while they are kept alive to communicate errors
to/from any still-runnning materialization processes. The intent is to enable
JITDylibs to be safely removed even if they have running compiles targeting
them.
2020-05-10 16:37:17 -07:00
Lang Hames c66f89005f [ORC] Rename SearchOrder operations on JITDylib to LinkOrder.
Refering to the link order of a dylib better matches the terminology used in
static compilation. As upcoming patches will increase the number of places where
link order matters (for example when closing JITDylibs) it's better to get this
name change out of the way early.
2020-05-04 16:47:52 -07:00
Lang Hames 990eb94467 [JITLink] Fix missing ADDITIONAL_HEADER_DIRS for the LLVMJITLink component.
Also adds target_link_libraries.

Thanks to @jcmac for spotting this.
2020-05-04 12:58:09 -07:00
Lang Hames 7ac0c3b475 [JITLink] Fix a typo in a comment.
Thanks to @jcmac for pointing this out.
2020-05-04 12:58:09 -07:00