Also make the soft toolchain requirements hard. This allows
us to use C++17 features in LLVM now.
If we find patterns with C++17 that improve readability
it should be recommended in the coding standards.
Reviewed By: jhenderson, cor3ntin, MaskRay
Differential Revision: https://reviews.llvm.org/D130689
`__bolt_instr_data_dump()` does not lock the hash tables when iterating
over them, so the iteration can happen concurrently with a modification
done in another thread, when the table is in an inconsistent state. This
also has been observed in practice, when it caused a segmentation fault.
We fix this by locking hash tables during iteration. This is done by taking
the lock in `forEachElement()`.
The only other site of iteration, `resetCounters()`, has been correctly
locking the table even before this patch. This patch removes its `Lock`
because the lock is now taken in the inner `forEachElement()`.
Reviewed By: maksfb, yota9
Differential Revision: https://reviews.llvm.org/D129089
Compiler can generate calls to some functions implicitly, even under
constraints of freestanding environment. Make sure these functions are
available in our runtime objects.
Fixes test failures on some systems after https://reviews.llvm.org/D128960.
Reviewed By: yota9
Differential Revision: https://reviews.llvm.org/D129168
Changed acquire implemetaion to __atomic_test_and_set() and release
to __atomic_clear() so it eliminates inline asm usage and is arch
independent.
Elvina Yakubova,
Advanced Software Technology Lab, Huawei
Reviewers: yota9, maksfb, rafauler
Differential Revision: https://reviews.llvm.org/D129162
If BOLT instrumentation runtime uses XMM registers, it can interfere
with the user program causing crashes and unexpected behavior. This
happens as the instrumentation code preserves general purpose registers
only.
Build BOLT instrumentation runtime with "-mno-sse".
Reviewed By: Amir
Differential Revision: https://reviews.llvm.org/D128960
Summary:
Refactor remaining bolt sources to follow the braces rule for if/else/loop from
[LLVM Coding Standards](https://llvm.org/docs/CodingStandards.html).
(cherry picked from FBD33345885)
Summary:
In some of the system stack protection is enabled by default, which will
lead in extra symbols dependencies, which we want to avoid.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD32478426)
Summary: In some cases __open() is returning 0 for me. The open
syscall will return a negative number (-1) on error so 0 should be
valid.
This happens when one compiles a BOLT instrumented executable of
pyston (python implementation) and afterwards pip installs a package
which needs to be compiled and sets CFLAGS=-pipe. I could not reduce
it down to a small testcase but I guess it one can trigger when
manually closing std{in, out, err}. Everything seems to work
normally when disabling the assert for 0 in getBinaryPath() - I
decided to also modify the second case in readDescriptions() even
though I did not run into that one yet.
(cherry picked from FBD32409548)
Summary:
Increase the hard limit from 256 to 4096.
This fixes the 'Assertion failed: failed to open binary path' error I'm seeing.
(cherry picked from FBD31911946)
Summary:
This patch temporarily disables instrumentation and higufy build not for
x86 platforms to be able to build llvm-bolt tool on aarch64.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD31738306)
Summary:
Sync the file with storage device on data dump to stabilize
instrumentation testing
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD31738021)
Summary:
This commit introduces TryLock usage for SimpleHashTable getter to
avoid deadlock and relax syscalls usage which causes significant
overhead in runtime.
The old behavior left under -conservative-instrumentation option passed
to instrumentation library.
Also, this commit includes a corresponding test case: instrumentation of
executable which performs indirect calls from common code and signal
handler.
Note: in case if TryLock was failed to acquire the lock - this indirect
call will not be accounted in the resulting profile.
Vasily Leonenko,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30821949)
Summary:
To avoid RELATIVE relocations avoid using of GOT table
by using hidden visibility for all symbols in library.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30092712)
Summary:
The trampolines are no loger pointers to the functions. For
propper name resolving by bolt use extern "C" for all external symbols
in instr.cpp
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30092698)
Summary:
This commit introduces -instrumentation-binpath argument used
to point instuqmented binary in runtime in case if /proc/self/map_files
path is not accessible due to access restriction issues.
Vasily Leonenko
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30092681)
Summary:
This commit adds support for opening libs based on links
/proc/self/map_files. For this we're getting current virtual address
and searching the lib in the directory with such address range. After
that, we're getting full path to the binary by using readlink
function. Direct read from link in /proc/self/map_files entries is not
possible because of lack of permissions.
Elvina Yakubova,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30092422)
Summary:
This commit adds support for getting directory entries and
reading value of a symbolic link in instrumentation runtime library
Elvina Yakubova,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30092362)
Summary:
This commit implements new method for _start & _fini functions hooking
which allows to use relative jumps for future PIE & .so library support.
Instead of using absolute address of _start & _fini functions known on
linking stage - we'll use dynamically created trampoline functions and
use corresponding symbols in instrumentation runtime library.
As we would like to use instrumentation for dynamically loaded binaries
(with PIE & .so), thus we need to compile instrumentation library with
"-fPIC" flag to support relative address resolution for functions and
data.
For shared libraries we need to handle initialization of instrumentation
library case by using DT_INIT section entry point.
Also this commit adds detection if the binary is executable or shared
library based on existence of PT_INTERP header. In case of shared
library we save information about real library init function address
for further usage for instrumentation library init trampoline function
creation and also update DT_INIT to point instrumentation library init
function.
Functions called from init/fini functions should be called with forced
stack alignment to avoid issues with instructions which relies on it.
E.g. optimized string operations.
Vasily Leonenko,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30092316)
Summary:
When indirect call is instrmented it locks SimpleHashTable's mutex on get() call.
If while locked we we receive a signal and signal handler also will call
indirect function we will end up with deadlock.
PR facebookincubator/BOLT#167
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD28909921)
Summary:
This PR introduces 2 new instrumentation options:
1. instrumentation-no-counters-clear: Discussed at https://github.com/facebookincubator/BOLT/issues/121
2. instrumentation-wait-forks: Since the instrumentation counters are mapped as MAP_SHARED it will be nice to add ability to wait until all forks of the parent process will die using tracking of process group.
The last patch is just emitBinary code refactor.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
Pull Request resolved: https://github.com/facebookincubator/BOLT/pull/125
GitHub Author: Vladislav Khmelevskyi <Vladislav.Khmelevskyi@huawei.com>
(cherry picked from FBD26919011)
Summary:
This diff is a preparation for dumping the profile generated by BOLT's instrumenation on MachO.
1/ Function "bolt_instr_fini" is placed into the predefined section "__fini"
2/ In the instrumentation pass we create a symbol "bolt_instr_fini" and
replace the last global destructor with it.
This is a temporary solution, in the future we need to register bolt_instr_fini in addition to the existing destructors without dropping the last one.
(cherry picked from FBD25071864)
Summary:
Right now, the SAVE_ALL sequence executed upon entry of both
of our runtime libs (hugify and instrumentation) will cause the stack to
not be aligned at a 16B boundary because it saves 15 8-byte regs. Change
the code sequence to adjust for that. The compiler may generate code
that assumes the stack is aligned by using movaps instructions, which
will crash.
(cherry picked from FBD22744307)
Summary:
Reading through the LLVM coding standard again, realized a few places where I didn't follow the standard when coding. Addressing them:
1. prefer static functions over functions in unnamed namespace.
2. #include as little as possible in headers
3. Have vtable anchors.
(cherry picked from FBD22353046)
Summary:
This patch enables automated hugify for Bolt.
When running Bolt against a binary with -hugify specified, Bolt will inject a call to a runtime library function at the entry of the binary. The runtime library calls madvise to map the hot code region into a 2M huge page. We support both new kernel with THP support and old kernels. For kernels with THP support we simply make a madvise call, while for old kernels, we first copy the code out, remap the memory with huge page, and then copy the code back.
With this change, we no longer need to manually call into hugify_self and precompile it with --hot-text. Instead, we could simply combine --hugify option with existing optimizations, and at runtime it will automatically move hot code into 2M pages.
Some details around the changes made:
1. Add an command line option to support --hugify. --hugify will automatically turn on --hot-text to get the proper hot code symbols. However, running with both --hugify and --hot-text is not allowed, since --hot-text is used on binaries that has precompiled call to hugify_self, which contradicts with the purpose of --hugify.
2. Moved the common utility functions out of instr.cpp to common.h, which will also be used by hugify.cpp. Added a few new system calls definitions.
3. Added a new class that inherits RuntimeLibrary, and implemented the necessary emit and link logic for hugify.
4. Added a simple test for hugify.
(cherry picked from FBD21384529)
Summary:
Add full instrumentation support (branches, direct and
indirect calls). Add output statistics to show how many hot bytes
were split from cold ones in functions. Add -cold-threshold option
to allow splitting warm code (non-zero count). Add option in
bolt-diff to report missing functions in profile 2.
In instrumentation, fini hooks are fixed to run proper finalization
code after program finishes. Hooks for startup are added to setup
the runtime structures that needs initilization, such as indirect call
hash tables.
Add support for automatically dumping profile data every N seconds by
forking a watcher process during runtime.
(cherry picked from FBD17644396)
Summary:
Change our CMake config for the standalone runtime instrumentation
library to check for the elf.h header before using it, so the build
doesn't break on systems lacking it. Also fix a SmallPtrSet usage where
its elements are not really pointers, but uint64_t, breaking the build
in Apple's Clang.
(cherry picked from FBD17505759)
Summary:
Change our edge profiling technique when using instrumentation
to do not instrument every edge. Instead, build the spanning tree
for the CFG and omit instrumentation for edges in the spanning tree.
Infer the edge count for these edges when writing the profile during
run time. The inference works with a bottom-up traversal of the spanning
tree and establishes the value of the edge connecting to the parent based
on a simple flow equation involving output and input edges, where the
only unknown variable is the parent edge.
This requires some engineering in the runtime lib to support dynamic
allocation for building these graphs at runtime.
(cherry picked from FBD17062773)
Summary:
Avoid directly allocating string and description tables in
binary's static data region, since they are not needed during runtime
except when writing the profile at exit. Change the runtime library to
open the tables on disk and read only when necessary.
(cherry picked from FBD16626030)
Summary:
To allow the development of future instrumentation work, this
patch adds support in BOLT for linking arbitrary libraries into the
binary processed by BOLT. We use orc relocation handling mechanism for
that. With this support, this patch also moves code programatically
generated in X86 assembly language by X86MCPlusBuilder to C code written
in a new library called bolt_rt. Change CMake to support this library as
an external project in the same way as clang does with compiler_rt. This
library is installed in the lib/ folder relative to BOLT root
installation and by default instrumentation will look for the library
at that location to finish processing the binary with instrumentation.
(cherry picked from FBD16572013)
Tobias Hieta