Commit Graph

832 Commits

Author SHA1 Message Date
Vladislav Khmelevsky 76d346ca14 [BOLT][PR] Instrumentation: Introduce -no-counters-clear and -wait-forks options
Summary:
This PR introduces 2 new instrumentation options:
1. instrumentation-no-counters-clear: Discussed at https://github.com/facebookincubator/BOLT/issues/121
2. instrumentation-wait-forks: Since the instrumentation counters are mapped as MAP_SHARED it will be nice to add ability to wait until all forks of the parent process will die using tracking of process group.
The last patch is just emitBinary code refactor.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei

Pull Request resolved: https://github.com/facebookincubator/BOLT/pull/125
GitHub Author: Vladislav Khmelevskyi <Vladislav.Khmelevskyi@huawei.com>

(cherry picked from FBD26919011)
2021-03-09 16:18:11 -08:00
Maksim Panchenko 225a8d7f2c [BOLT] Ignore TBSS section at layout time
Summary:
TBSS section is a "virtual" section that does not take memory or file
space. Ignore it completely while adjusting section sizes.

(cherry picked from FBD26824484)
2021-03-04 16:31:12 -08:00
Vladislav Khmelevsky ec9751eef5 [BOLT][PR] readDynamicRelocations: Skip NONE relocations
Summary:
NONE relocations should not be processed during dynamic relocations read process
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei

Pull Request resolved: https://github.com/facebookincubator/BOLT/pull/118
GitHub Author: Vladislav Khmelevsky <Vladislav.Khmelevskyi@huawei.com>

(cherry picked from FBD26489881)
2021-02-17 15:36:58 -08:00
Alexander Yermolovich 06959eedcf Fix up test for Update DW_AT_stmt_list for .debug_types
Summary: As titled.

(cherry picked from FBD28112186)
2021-03-17 17:08:26 -07:00
Rafael Auler da752c9c5c Fix license for a few remaining files
Summary: As titled.

(cherry picked from FBD28112137)
2021-03-17 15:04:19 -07:00
Alexander Yermolovich 0ec91a25df Update DW_AT_stmt_list for .debug_types
Summary:
There is no real link between CU and TU, so relying on fact
that address are the same, and we are updating all of them.

(cherry picked from FBD28112114)
2021-02-17 15:30:10 -08:00
Rafael Auler 16521f1f79 [BOLT] Update license headers
Summary: Update license and fix headers for some files.

(cherry picked from FBD28112041)
2021-03-15 18:04:18 -07:00
Amir Ayupov 1c5d3a056c Rebase: Merge BOLT codebase in monorepo
Summary:
This commit is the first step in rebasing all of BOLT
history in the LLVM monorepo. It also solves trivial build issues
by updating BOLT codebase to use current LLVM. There is still work
left in rebasing some BOLT features and in making sure everything
is working as intended.

History has been rewritten to put BOLT in the /bolt folder, as
opposed to /tools/llvm-bolt.

(cherry picked from FBD33289252)
2020-12-01 16:29:39 -08:00
Alexander Shaposhnikov 0a8aaf56bb [BOLT] Add support for reading profile on Mach-O
Summary: Add support for reading profile on Mach-O.

(cherry picked from FBD25777049)
2021-01-29 16:37:07 -08:00
Alexander Shaposhnikov a0dd5b05dc [BOLT] Add support for dumping profile on MacOS
Summary: Add support for dumping profile on MacOS.

(cherry picked from FBD25751363)
2021-01-28 12:44:14 -08:00
Alexander Shaposhnikov 3b876cc3e7 [BOLT] Add support for dumping counters on MacOS
Summary: Add support for dumping counters on MacOS

(cherry picked from FBD25750516)
2021-01-28 12:32:03 -08:00
Alexander Shaposhnikov 6a84124e1d [BOLT] Add support for __literal16 section on MachO
Summary:
1. Add support for __literal16 section in the instrumentation runtime library for MacOS.
2. Fix emitting __counters section.

(cherry picked from FBD25746342)
2021-01-28 12:04:46 -08:00
Sergey Pupyrev fea6b4e469 an updated version of ExtTSP
Summary:
a few minor updates in block reordering:
- some refactoring to improve readability;
- optimized chain splitting strategy to improve quality of layout and performance of the algorithm.

(cherry picked from FBD25126220)
2021-01-27 18:29:16 -08:00
Alexander Shaposhnikov d6e60c5bec [BOLT] Enable intToStr for MacOS
Summary: Enable intToStr et al. in the runtime library for MacOS.

(cherry picked from FBD25745358)
2021-01-20 16:40:17 -08:00
Alexander Shaposhnikov faaefff618 [BOLT] Fix operator new signature
Summary:
Use size_t for the first parameter of operator new.
https://en.cppreference.com/w/cpp/memory/new/operator_new

(cherry picked from FBD25750921)
2021-01-20 12:56:41 -08:00
Amir Ayupov a86cd533b3 [BOLT] Fix missing newlines in debug prints
(cherry picked from FBD25966797)
2021-01-19 18:43:16 -08:00
Rafael Auler 0de92b8346 [PERF2BOLT] Relax segment matching requirements
Summary:
When looking at perf.data's available binaries and their
respective mmap'ed segments, match them with the input binary by
looking at both aligned and non-aligned addresses. If we suppose
the alignment is the mmap'ed page size, we may miss some cases and
perf2bolt will refuse to proceed because it failed to match the
input binary with a process recorded in perf.data.

(cherry picked from FBD25732673)
2021-01-11 06:24:46 -08:00
Rafael Auler e3898d5969 [BOLT] Add threshold options for lite mode
Summary:
Add options for trading processing speed for binary performance.

  -lite-threshold-pct=<uint>
    Threshold (in percent) for selecting functions to process in lite
    mode. Higher threshold means fewer functions to process.
    E.g threshold of 90 means only top 10 percent of functions with
    profile will be processed.

  -lite-threshold-count=<uint>
    Similar to '-lite-threshold-pct' but specify threshold using
    absolute function call count. I.e. limit processing to functions
    executed at least the specified number of times.

  -no-scan
    Do not scan cold functions for external references (may result in
    slower binary).

(cherry picked from FBD24739092)
2020-12-30 12:23:58 -08:00
Rafael Auler e0261a22ce [TEST] Remove dependency on debug output
Summary:
Test mistakenly used -debug output, which makes it fail on
no-asserts build.

(cherry picked from FBD25399449)
2020-12-09 12:25:58 -08:00
Rafael Auler d2f68039bc [BOLT] Fix shrinkwrapping bug when changing frame alignment
Summary:
This fixes a bug with shrink wrapping when trying to move
push-pops in a function where we are not allowed to modify the
stack layout for alignment reasons. In this bug, we failed to
propagate alignment requirement upwards in the call graph from
function A to B when: (1) there is a cycle in the call graph and
(2) the distance from A to B is greater than 1 in the call graph
and (3) there is a node in the path from A to B, not including
A or B, that does not access parameters in the stack.

(cherry picked from FBD25315977)
2020-12-03 20:09:32 -08:00
Alexander Shaposhnikov e067f2adf4 Inject instrumentation's global dtor on MachO
Summary:
This diff is a preparation for dumping the profile generated by BOLT's instrumenation on MachO.

1/  Function "bolt_instr_fini" is placed into the predefined section "__fini"

2/ In the instrumentation pass we create a symbol "bolt_instr_fini" and
replace the last global destructor with it.

This is a temporary solution, in the future we need to register bolt_instr_fini in addition to the existing destructors without dropping the last one.

(cherry picked from FBD25071864)
2020-11-19 18:18:28 -08:00
Alexander Shaposhnikov 1b258b8908 Refactor syscall wrappers for OSX
Summary: Refactor syscall wrappers for OSX.

(cherry picked from FBD25084642)
2020-11-19 14:56:45 -08:00
Amir Ayupov f9d00d418b [BOLT] Handle insertion of updated CFI at the first basic block
Summary:
Fix corner case of insertion of updated CFI with unset `PrevBB`.
Handle it in the same way as inserting past hot-cold split point.

(cherry picked from FBD24943911)
2020-11-17 18:40:19 -08:00
Alexander Shaposhnikov 1cf23e5ee8 Link the instrumentation runtime on OSX
Summary: Link the instrumentation runtime on OSX.

(cherry picked from FBD24390019)
2020-11-17 13:57:29 -08:00
Maksim Panchenko 7eaf63a118 [BOLT] Fix data race while running split functions pass
Summary:
In BinaryContext::calculateEmittedSize(), after the temporary code
emission, we have to perform a cleanup and mark all symbols used
during the emission as undefined and unregistered (so that we can emit
them again later). The cleanup is happening even for symbols that were
referenced and not defined by emitted code.

If all emitted symbols are local, there is no risk that one thread will
define a symbol while some other thread will undefine it in its cleanup
code. Such behavior is expected as local symbols can only be referenced
within the containing function and each function is processed in one
thread. However, secondary entry points have associated global symbols
and if we emit them, then it is possible for a thread to undefine
a symbol while the other thread had defined it and was in the process of
emitting the fragment with it. In such case, a data race may happen and
the thread that contains the definition of the symbol may define it
twice causing a redefinition error.

To avoid the data race, we skip the emission of secondary entry global
symbols when emitting code used only for the size estimation.

(cherry picked from FBD24986007)
2020-11-16 14:34:51 -08:00
Sergey Pupyrev 1e9b733008 a new version of hfsort+
Summary:
A faster and better version of function reordering:
- fixed a bug when some computed probabilities were negative;
- changed an O(n^2) loop to a priority queue to find a candidate of chains to merge

(cherry picked from FBD24571208)
2020-11-14 13:18:58 -08:00
Amir Ayupov 6401af89c7 [BOLT] Support jump tables in split fragments with entries pointing back to parent functions
Summary:
Support jump tables belonging to split fragments with entries
pointing back to parent functions.
While skipping such families of functions, make sure to use the
topmost fragment to ignore its fragments.

(cherry picked from FBD24907438)
2020-11-12 11:54:51 -08:00
Amir Ayupov e8234b3b98 [BOLT] Add invalid offset for a JT entry pointing to a fragment
Summary:
In a jump table identification, register an invalid offset for jump table
entries pointing to function fragments.
These invalid offsets have no effect other than padding the jump
table size, calculated as `max(OffsetEntries, Entries)`.
Correct jump table size is required in strict mode (enabled by default
in aggregation mode by `perf2bolt`) in accounting of all PC-relative
relocations in data.
Functions containing these jump tables with invalid offsets are
marked to be ignored immediately afterwards in
`populateJumpTables`.

(cherry picked from FBD24897464)
2020-11-12 11:54:44 -08:00
Amir Ayupov 157129b751 [BOLT] Debug logging in analyzeJumpTable
Summary:
Added debug logging in/around `analyzeJumpTable`:
- Dump jump table entries as they are being processed:
```BOLT-DEBUG: analyzeJumpTable in read_encoded_value_with_base/2(*2)
  * Checking 0x428ff40 -> OK: real entry
  * Checking 0x428ff44 -> OK: real entry
  * Checking 0x428ff48 -> OK: real entry
  * Checking 0x428ff4c -> OK: real entry
  * Checking 0x428ff50 -> OK: real entry
  * Checking 0x428ff54 -> OK: address in split fragment
  * Checking 0x428ff58 -> OK: address in split fragment
  * Checking 0x428ff5c -> OK: address in split fragment
  * Checking 0x428ff60 -> OK: address in split fragment
  * Checking 0x428ff64 -> OK: real entry
  * Checking 0x428ff68 -> OK: real entry
  * Checking 0x428ff6c -> OK: real entry
  * Checking 0x428ff70 -> OK: real entry
BOLT-DEBUG: analyzeJumpTable in classify_object_over_fdes/1(*2)
  * Checking 0x428ff74 -> OK: real entry
  ...
```
- Dump skipped functions:
```
Skipping _ZNK6icu_676number4impl12RoundingImpl5applyERNS1_15DecimalQuantityER10UErrorCode.part.2/1(*2) family
Ignoring _ZNK6icu_676number4impl12RoundingImpl5applyERNS1_15DecimalQuantityER10UErrorCode.part.2/1(*2)
Ignoring _ZNK6icu_676number4impl12RoundingImpl5applyERNS1_15DecimalQuantityER10UErrorCode.part.2.cold.3/1(*2)
Skipping _ZNK6icu_676number4impl12RoundingImpl5applyERNS1_15DecimalQuantityER10UErrorCode family
Ignoring _ZNK6icu_676number4impl12RoundingImpl5applyERNS1_15DecimalQuantityER10UErrorCode
Ignoring _ZNK6icu_676number4impl12RoundingImpl5applyERNS1_15DecimalQuantityER10UErrorCode.cold.4/1(*2)
```
- Dump values of unclaimed PC-relative relocations in data.

(cherry picked from FBD24898172)
2020-11-12 11:54:38 -08:00
Amir Ayupov c0cb550536 Minimize X86/shrinkwrapping-critedge test case
Summary: Minimized test case while preserving the CFG subgraph with an issue

(cherry picked from FBD24871063)
2020-11-10 21:22:57 -08:00
Amir Ayupov e54d389799 [BOLT] Disable DynoStats printing after SCTC
Summary:
Introduce new BinaryFunction flag `IsCanonicalCFG`, which gets
unset by SCTC pass. Make DynoStats collection conditional on this
new flag.
SCTC leaves CFG in a state where branch counters of BBs with tail
calls/conditional tail calls are not available (except via annotations,
which get stripped by `lower-annotations`). Without branch
counters, DynoStats are invalid.

(cherry picked from FBD24558050)
2020-11-10 10:51:23 -08:00
Amir Ayupov c36b71686c Improve cold fragment name matching
Summary:
Fix cold fragment name matching regex by replacing existing
regexes `.*\.cold\..*` and  `.*\.cold`
and combining them into `.*\.cold(\.\d)?`,
applied to restored name (with BOLT-added suffixes stripped)

This allows matching names like "execute_stack_op.cold/1", which
previously weren't recognized.

(cherry picked from FBD24804880)
2020-11-09 12:38:51 -08:00
Amir Ayupov f86a78a4e7 Lost in rebase: call registerFragment with a reference to TargetBF
Summary: Fixes broken build due to a lost dereferencing

(cherry picked from FBD24799948)
2020-11-06 12:22:22 -08:00
Amir Ayupov 2b09d672ce Conservatively handle jump tables in split functions
Summary:
- Allow jump table entries to point to locations inside the function and its fragments.
Reasoning behind this is that jump table identification has the logic of stopping at entry which belongs to a function different from the one originally referencing jump table. This assumption is invalid for jump tables with entries pointing to both parent function and cold fragments, leading to "unclaimed PC-relative relocations" assertion.

- Add fragment identification heuristic based on function name regex and contiguous jump table entries.
Currently, parent-to-fragment relationship is set up based on interprocedural references – direct references from the parent function. These references don't include references through jump table.
Additionally, some fragments are only reachable through jump table. In that case, in order to fully consume jump table, add parent-to-fragment relationship during `analyzeJumpTable` using the following heuristics:
  1. Fragment is identified as such based on name (contains `.cold.` part), but
  2. Parent function is not set – no direct interprocedural references to that fragment, and
  3. Fragment has the name of the form <parent>.cold(.\d+)

* For split functions with jump table entries spanning parent and fragments, mark parent and all fragments as ignored.

(cherry picked from FBD24456904)
2020-11-06 11:19:03 -08:00
Amir Ayupov dc48354f71 processInterproceduralReferences: record references to cold fragments as entry points
Summary:
For interprocedural references to fragments, record them as
fragment entry points. Not registering these entry points leads to
UCE removing the blocks and "Undefined temporary symbol"
assertion.

(cherry picked from FBD24511281)
2020-11-06 10:57:47 -08:00
Amir Ayupov 5452287710 Extract BinaryContext::registerFragment
Summary: registerFragment to be reused in adding fragments reachable only through jump tables.

(cherry picked from FBD24656651)
2020-11-06 10:27:33 -08:00
Vladislav Khmelevsky 58460460d9 [BOLT][PR] Handle TLS relocations on AArch64
Summary:
Some of the TLS relocatios like R_AARCH64_TLSDESC_ADR_PAGE21 must be
handled by bolt and should not be skipped by the removed condition. Some
of the TLS relocations like R_AARCH64_TLS_TPREL64 could really be skipped
here, but AFAIU this condition was added as part of BOLT its self optimization, so
to prevent future problems here my suggestion is not to add another condition
like "isTLS(RType) && isTLSRelocatable(RType)", but just remove it since
absense of this condition should not broke any other TLS relocation.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei

Pull Request resolved: https://github.com/facebookincubator/BOLT/pull/103
GitHub Author: Vladislav Khmelevsky <Vladislav.Khmelevskyi@huawei.com>

(cherry picked from FBD24745928)
2020-11-04 16:45:58 -08:00
Maksim Panchenko 4f4239ceba [BOLT] Fix C++ exceptions for shared objects
Summary:
Fix several issues to make C++ exceptions work in shared objects:
  * Set MCObjectFileInfo PIC type based on the input binary type.
  * Support indirect (DW_EH_PE_indirect) encoding while writing
    exception Type Table.
  * Use different LPStart value and landing pad encoding for .so's.
  * Disable splitting of exception-handling code for .so's because of
    the new encoding.

(cherry picked from FBD24698765)
2020-11-04 11:44:02 -08:00
Rafael Auler c1bb4dcb2b [BOLT] Remove threaded EliminateUnreachableBlock version
Summary:
EliminateUnreachableBlocks has a data race because it depends
on BinaryContext::computeCodeSize. computeCodeSize supports independent
Emitters, enabling a lock-free execution. Unfortunately, that is almost
as expensive as the lock. Removing the boilerplate code for
parallellization of this pass turned out to be the best alternative: no
races and slightly better execution time for HHVM.

(cherry picked from FBD24716250)
2020-11-03 11:28:59 -08:00
Rafael Auler 37921b489a [BOLT] Please sanitizers
Summary:
In BinaryContext, we had StringRef holding a reference to
an r-value std::string. This triggers clang's address sanitizer
warnings. In MCPlusBuilder we had a left shift overflowing a type,
which is undefined behavior. Similarly, in CallGraph, we had a hash
function shifting a negative value, which is also UB. The last two
triggers the UB sanitizer.

(cherry picked from FBD24661045)
2020-10-30 15:11:52 -07:00
Rafael Auler 3e78082c1b [DOCS] Add instrumentation instructions to README
Summary: Add basic instructions on how to instrument a binary.

(cherry picked from FBD24660183)
2020-10-30 14:45:30 -07:00
Rafael Auler eb12d719ac [BOLT] Fix no-asserts build
Summary: Only use dump() method under DEBUG() macro.

(cherry picked from FBD24666481)
2020-10-30 19:59:07 -07:00
Maksim Panchenko 6b185cccf4 [BOLT] Always keep dynamic symbols defined
Summary:
Some symbols in .dynsym will be erroneously marked as belonging to a
non-allocatable section that BOLT can remove. In that case, keep the
original invalid index for such symbols instead of setting the UNDEF
index.

(cherry picked from FBD24488677)
2020-10-22 16:35:29 -07:00
Amir Ayupov 5f2f96c4c9 Add pass number to dot dump filename
Summary:
Change .dot dumps filename format from
  <function>-<passname>.dot
to
  <function>-<passidx>_<passname>.dot
This change helps navigate dumps by making the pass order explicit.
Example:
  execute_stack_op.cold.6-1(*2)-00_build-cfg.dot
  execute_stack_op.cold.6-1(*2)-01_validate-internal-calls.dot
  execute_stack_op.cold.6-1(*2)-02_strip-rep-ret.dot
  ...

(cherry picked from FBD24452903)
2020-10-21 17:08:32 -07:00
Maksim Panchenko d91add0bfe [BOLT] Fix PatchEntries pass
Summary:
While refactoring the pass, I removed the important transactional
property of the patching process. Restore it.

(cherry picked from FBD24440214)
2020-10-21 12:31:09 -07:00
Maksim Panchenko d6d88399fc [BOLT] Enable lite mode by default with relocations
Summary:
When optimizing input with relocations, make it faster and less
memory-hungry with lite mode.

(cherry picked from FBD24374241)
2020-10-17 15:09:06 -07:00
Rafael Auler e4396c41da [BOLT] Ignore __hot_start, __hot_end from input
Summary:
When -hot-text is on, do not read __hot_start and __hot_end
from input (inserted by a linker script with the intent of ordering
functions). This can confuse BOLT into creating a function with this
name depending on which address the symbol lands and we will assert
when trying to emit our own __hot_start/__hot_end with symbol
redefinition.

(cherry picked from FBD24366636)
2020-10-17 00:50:27 -07:00
Alexander Shaposhnikov 6133d2598b Inject a hook into the entry point on MachO
Summary:
This diff is a preparation for loading the runtime on MachO.
The proposed schema is the following:

1/  Function "bolt_instr_setup" is placed into the predefined section "setup" (in the final setting this function will be coming from the instrumentation runtime but we still will be placing it into this section).

2/ In the instrumentation pass we create a symbol "bolt_instr_setup" and inject the corresponding call into the beginning of the function representing the entry point of the binary.

(cherry picked from FBD24329530)
2020-10-15 01:39:35 -07:00
Maksim Panchenko f15532c2aa [BOLT][DWARF] Streamline processing of DWARF unit DIEs
Summary:
Do not store processed DWARF DIEs, but instead process them while
reading one at a time.

Reduces memory consumption when updating debug info by 10%-25%.

(cherry picked from FBD24327029)
2020-10-16 00:11:24 -07:00
Alexander Shaposhnikov bbd9d610fe Add first bits to cross-compile the runtime for OSX
Summary: Add first bits to cross-compile the runtime for OSX.

(cherry picked from FBD24330977)
2020-10-15 03:51:56 -07:00