Since we now re-write .debug_info the DWARF CU Offsets can change.
Just like for .debug_aranges the GDB Index will need to be updated.
Reviewed By: Amir, maksfb
Differential Revision: https://reviews.llvm.org/D118273
This patch reverts patch "DWARFv5 default: Switch bolt tests to use
DWARFv4 since Bolt doesn't support v5 yet" and places the -gdwarf-4 flag
to the global cflags config file.
Reviewed By: Amir
Differential Revision: https://reviews.llvm.org/D118283
This patch adds unit testing support for BOLT. In order to do this we will need at least do this changes on the code level:
* Make createMCPlusBuilder accessible externally
* Remove positional InputFilename argument to bolt utlity sources
And prepare the cmake and lit for the new tests.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
Reviewed By: maksfb, Amir
Differential Revision: https://reviews.llvm.org/D118271
This patch fixes the removal of unreachable uncondtional branch located
after return instruction.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
Reviewed By: Amir
Differential Revision: https://reviews.llvm.org/D117677
In case the case the DW_AT_ranges tag already exists for the object the
low pc values won't be updated and will be incorrect in
after-bolt binaries.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
Differential Revision: https://reviews.llvm.org/D117216
Matching an exact byte offset is fragile if a different version of compiler
is used (e.g. distro clang).
Resolves an issue with running with BOLT_CLANG_EXE + clang-12
Reviewed By: maksfb
Differential Revision: https://reviews.llvm.org/D117440
Summary:
Fix missing string header file inclusion and link_fdata find
problem in lit tests. Change root-level tests to require
linux. Re-enable Windows in our root CMakeLists.txt.
(cherry picked from FBD33296290)
Summary:
The patch moves the shortenInstructions and nop remove to separate binary
passes. As a result when llvm-bolt optimizations stage will begin the
instructions of the binary functions will be absolutely the same as it
was in the binary. This is needed for the golang support by llvm-bolt.
Some of the tests must be changed, since bb alignment nops might create
unreachable BBs in original functions.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD32896517)
Summary:
This patch adds AArch64 relocations handling in case updating of
debug sections is enabled
Elvina Yakubova,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD33077609)
Summary:
Gracefully handle binaries with split functions where two fragments are folded
into one, resulting in a fragment with two parent functions.
This behavior is expected in GCC8+ with -O2 optimization level, where both
function splitting and ICF are enabled by default.
On the BOLT side, the changes are:
- BinaryFunction: allow multiple parent fragments:
- `ParentFragment` --> `ParentFragments`,
- `setParentFragment` --> `addParentFragment`.
- BinaryContext:
- `populateJumpTables`: mark fragments to be skipped later,
- `registerFragment`: add a name heuristic check, return false if it failed,
- `processInterproceduralReferences`: check if `registerFragment`
succeeded, otherwise issue a warning,
- `skipMarkedFragments`: move out fragment traversal and skipping from
`populateJumpTables` into a separate function.
This change fixes an issue where unrelated functions might be registered
as fragments:
```
BOLT-WARNING: interprocedural reference between unrelated fragments:
bad_gs/1(*2) and amd_decode_mce.cold.27/1(*2)
```
(Linux kernel binary)
(cherry picked from FBD32786688)
Summary:
Currently, RuntimeDyld will not allocate a section without relocations
even if such a section is marked allocatable and defines symbols.
When we emit .debug_line for compile units with unchanged code, we
output original (input) data, without relocations. If all units are
emitted in this way, we will have no relocations in the emitted
.debug_line. RuntimeDyld will not allocate the section and as a result
we will write an empty .debug_line section.
To workaround the issue, always emit a relocation of RELOC_NONE type
when emitting raw contents to debug_line.
(cherry picked from FBD32909869)
Summary:
Some optimizations may remove all instructions in a basic block.
The pass will cleanup the CFG afterwards by removing empty basic
blocks and merging duplicate CFG edges.
The normalized CFG is printed under '-print-normalized' option.
(cherry picked from FBD32774360)
Summary:
TailDuplication::isInCacheLine makes the assumption that the block
has a valid layout index, which is not the case for unreachable blocks.
Add a check for a valid layout index.
(cherry picked from FBD32659755)
Summary:
The push and pop instructions might have wrong reorder due to this
error. Thanks rafaelauler for the provided test case.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD32478348)
Summary:
Import the test. The assembly input has three functions with associated fdata.
The old link_fdata.sh script only replaces the symbol names with symbol values,
whereas fdata format expects to have symbol offsets against the anchor symbol.
Introduce the link_fdata.py script which is able to parse the input and produce
either an offset or an absolute symbol value.
(cherry picked from FBD32256351)
Summary:
The majority of tests in LLVM projects are using - instead of _ in the name,
i.e. `check-something.test` is preferred over `check_something.test`.
It makes sense for us to adopt the same naming scheme for our future tests and
to rename existing ones.
(cherry picked from FBD32185879)
Summary:
BOLT meta test using merge-fdata tool.
This tests BOLT instrumentation for a non-trivial binary, running instrumented
binary, and using the instrumentation profile for BOLT optimizations.
The results are verified between original, instrumented, and optimized binaries.
Additional tested features: boltdiff mode and merge-fdata for two profiles.
merge-fdata tool is linked with relocs on Linux to support this test.
(cherry picked from FBD32141812)
Summary:
Added new functionality of dumping simple functions into assembly.
This includes:
- function control flow (basic blocks, instructions),
- profile information as `FDATA` directives, to be consumed by link_fdata,
- data labels,
- CFI directives,
- symbols for callee functions,
- jump table symbols.
Envisioned usage:
1. Find a function that triggers BOLT crash (e.g. with `bughunter.sh`).
2. Generate reproducer asm source for that function (using `-funcs`).
3. Attach it to an issue.
4. Reduce and include as a test case.
Current limitations:
1. Emitted assembly won't match input file relocations.
2. No DWARF support.
3. Data is not emitted.
(cherry picked from FBD32746857)
Summary:
Change cmake config in BOLT to only support Linux. In other
platforms, we print a warning that we won't build BOLT. Change
configs to determine whether we will build BOLT runtime libs. This
only happens in x86 hosts. If true, we will build the runtime and
enable bolt-runtime tests. New tests that depend on the bolt_rt lib
needs to be marked REQUIRES:bolt-runtime. I updated the relevant
tests. Fix cmake to do not crash when building llvm with a target
that BOLT does not support.
(cherry picked from FBD31935760)
Summary:
Add lit.local.cfg to X86 and AArch64 folders.
Fix host_arch in lit config for AArch64.
Fix AArch64 and X86 tests.
Elvina Yakubova,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD31702068)
Summary:
In non-PIC binaries compiler could save absolute addresses in constant
isalnd which we should handle properly. This patch adds relocations
handling in constant islands.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD31416848)
Summary:
Change sed separator to allow replacing symbols with slash in the name.
This is required for symbol names produced by BOLT which include
"/1" suffix.
(cherry picked from FBD31324540)
Summary:
This patch adds R_AARCH64_TSTBR14 and R_AARCH64_CONDBR19 relocations
support in order to handle condition branches, cbz/cnbz and tbz/tbnz
instructions correctly
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD31416734)
Summary:
The ELF::R_AARCH64_TLSDESC_LD64_LO12 and
ELF::R_AARCH64_TLSDESC_ADR_PAGE21 relocations might also be relaxed to
mov instructions, handle these cases
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD31353063)
Summary:
When the compiler emits line table program, it emits EOS using the label
at the end of the containing code section. Since each compilation unit
has its own set of code sections it works as expected (* see the excerpt
from the standard below). However, in BOLT the code from many CUs is
combined into a common section, such as hot text or cold text.
As a result, the symbol at the end of the section may point way past the
code sequence for a given unit.
Since we can emit functions in any order, we conservatively emit
end-of-sequence at the end of every emitted function.
Fixes a problem while intermixing source code with disassembly in
binutils' objdump.
(*) DWARF v4 6.2.5.3:
"Every line number program sequence must end with a DW_LNE_end_sequence
instruction which creates a row whose address is that of the byte after
the last target machine instruction of the sequence."
(cherry picked from FBD31347870)
Summary:
Create bolt/test/runtime folder and move tests that execute the binary.
Move lit.local.cfg with host_arch check to the corresponding folder.
Addresses issue facebookincubator/BOLT#132.
AArch64/tls.c shows a different behavior with clang hence marked as XFAIL
TODO: add a check for non-exec tests for a corresponding LLVM_TARGETS_TO_BUILD.
(cherry picked from FBD31132234)
Summary:
There are few problems found when dealing with TLS relocations for
aarch64.
* RewriteInstance.cpp
** While analyzing TLS relocation we don't have to modify
SymbolAddress (which is the offset from the TLS section), so we need to
just skip verifiction
** The non-got related TLS relocations on aarch64 might be skipped too
** The forse relocation must be applied for GOT relocations on
Aarch64. The symbol adress for GOT relocation might no be pointing
on GOT section (for example ADRP GOT may point to the wrong section,
since GOT table is not page-aligned), so we won't try to get section by
the symbol address.
* Relocation.cpp - Remove R_AARCH64_TLSLE_ADD_TPREL_HI12 and
R_AARCH64_TLSLE_ADD_TPREL_LO12_NC from isGOT check, since they are not
got-related relocations
* BinaryFunction.h
** Remove R_AARCH64_TLSLE_ADD_TPREL_HI12 and
R_AARCH64_TLSLE_ADD_TPREL_LO12_NC from adding to relocation list, since
this is actually an offset in TLS section and BOLT does not change it we
don't need to do something with this relocations, the value won't change
in new binary files
** Refactor the code, separating aarch64 and x86 relocations
* AArch64MCPlusBuilder.cpp
** Add forgotten LO12 relocations to switch case to getTargetExprFor
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD31003349)
Summary:
LLVM started printing warnings when DWARFDebugInfoEntry::extractFast()
is invoked trying to read a DIE past the current unit limits. This
results in verbose warnings from BOLT which are harmless but confusing
to the user. Check the boundaries before calling the API above.
(cherry picked from FBD31097271)
Summary:
We have a problem where we will emit sections that we are not supposed
to emit (with no output offset assigned). This will make us write at
file offset 0 and corrupt the first sections in the binary (usually
.interp section will be corrupted and bash will refuse to run the
binary).
This only happens in non-reloc mode when using JTS_BASIC and when we
do not emit a function that has a jump table (if it gets too large).
Using -update-debug-sections will trigger the pass
check-large-functions, which will mark large funcs as non-simple
and will hide this bug.
(cherry picked from FBD30882012)
Summary:
This commit introduces TryLock usage for SimpleHashTable getter to
avoid deadlock and relax syscalls usage which causes significant
overhead in runtime.
The old behavior left under -conservative-instrumentation option passed
to instrumentation library.
Also, this commit includes a corresponding test case: instrumentation of
executable which performs indirect calls from common code and signal
handler.
Note: in case if TryLock was failed to acquire the lock - this indirect
call will not be accounted in the resulting profile.
Vasily Leonenko,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30821949)
Summary:
This commit adds checking if maxIndividualTestTime is availabe on
the platform. If available - it sets per test timeout to 60sec and
declares lit-max-individual-test-time feature for further checking
by particular test cases.
Based on https://reviews.llvm.org/D64251 implementation.
Vasily Leonenko,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30821986)
Summary:
Three way branches commonly appear
in HHVM. They have one test and then two jumps. The
jump's destinations are not currently optimized.
This pass attempts to optimize which is the first branch.
(cherry picked from FBD30460441)
Summary:
There are 2 problems found when handling ADR instruction:
1. When extracting value from the ADR instruction we need to do
it another way, then we do it for ADRP instruction.
2. When creating target expression the VariantKind should be other for
ADR instruction.
And we introduces R_AARCH64_ADR_PREL_LO21,
R_AARCH64_TLSDESC_ADR_PREL21 and R_AARCH64_ADR_PREL_PG_HI21_NC
relocations support.
Also this patch introduces AdrPass, which will replace non-local
pointing ADR instructions with ADRP + ADD instructions sequence due to
small offset range of ADR instruction, so after BOLT magic there are no
guarantees that ADR instruction will still be in the range of
just +- 1MB from its target. The instruction replacement needs
relocations to be avalailable, so we won't remove "IsFromCode"
relocations after disassembly from BF anymore. Also we need original
offset of ADR instruction to be available so we add offset annotation
for these instructions.
The last thing this patch adds is ARM testing directory, which will be
used only on ARM testing servers. The common tests (non-assembler tests
which are platform-independent) might be moved from the X86 directory to
the parent one in the future, so such tests could be tested on both X86
and ARM machines.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30497379)
Summary:
Added a function in TailDuplication
that will do Constant and Copy Propagation for blocks that
we duplicated as a part of tail duplication. Added supporting
functions to MCPlusBuilder to find src registers and replace
registers
(cherry picked from FBD30231907)
Summary:
This patch is part of preparation for golang support. The golang symbols
might have spaces in the name (for example "type..eq.[10]interface {}").
Since fdata uses spaces as a field separator such names brakes the fdata
format, so we need to escape whitespaces and backslashes in symbol names
using the backslash character.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD29999491)
Summary:
This commit adds dummy tests for checking instrumentation
support for PIE executables and shared libraries.
Vasily Leonenko,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD30092729)
Summary:
The linker can generate 8- or 16-byte entries in .plt.got and .plt.sec
sections. On X86, the main differentiator is the presence of endbr64
instruction at the beginning of the entry. Detect the instruction and
adjust the size accordingly.
(cherry picked from FBD29847639)
Summary:
Created a binary pass that records how many
times tail duplication would be used and how many cache
misses it would theoretically stop
(cherry picked from FBD29619858)
Summary:
A binary can contain multiple PLT sections with different name and
attributes (such as an entry size). Extend the support to .plt.sec and
refactor the code to make future extensions simpler.
(cherry picked from FBD29502107)
Summary:
Our YAML objects contain references to dynamic relocations via .dynamic,
but there are no corresponding relocation sections. Change .dynamic
contents to specify no dynamic relocations.
(cherry picked from FBD29502108)
Summary:
When we fold a function in relocation mode, make sure to clear its state
to avoid emitting relocations against undefined symbols.
(cherry picked from FBD29245320)
Summary:
While printing debug info for instructions, we should use line tables
from the corresponding DWARF CU which could be different from the
containing function CU in case of inlined instructions.
(cherry picked from FBD28908324)
Summary:
This patch introduces LoopInversionPass. Its main purpose is to ensure
that the loop layout is optimal depending on the profile information. So
if profile information shows that the loop is used, the unconditional
jump instruction must be executed only once and vice-versa. Please take
a look to the pass header file and test for more details.
Also change link_fdata script a bit, to be able to change FDATA prefix,
like FileCheck does.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
PR facebookincubator/BOLT#153
(cherry picked from FBD28391811)
Summary:
Implemented support for Debug Fission.
For the most part it doesn't impact Monolithic execution path.
One area that was changed is the DW_AT_low_pc/DW_AT_high_pc conversion. Before it was to DW_AT_ranges/DW_AT_low_pc, now DW_AT_low_pc is kept in same place.
Another more visible impact is in Skeleton CU the DW_AT_low_pc is replaced with DW_AT_ranges_base if it's not originally present and bolt converted ranges conversion inside the dwo units.
Output of this are multiple .dwo files with updated debug information.
(cherry picked from FBD29569788)
Summary:
Since gcc/ld could produce and expect PIE files we need to pass -no-pie option to avoid linking errors for tests.
Vladislav Khmelevsky,
Advanced Software Technology Lab, Huawei
(cherry picked from FBD28360045)
Summary:
Reorder-blocks optimization pass doesn't take into account that
available offset for legacy Jcc instructions (for example,
JRCXZ - operand 8 bits) has to be less than 255 bytes.
It's rare case and to exclude such functions with unsupported
instructions from optimization passes added extra checking
Alexey Moksyakov
Advanced Software Technology Lab, Huawei
(cherry picked from FBD28264117)
Summary:
The user may wish to run BOLT for printing statistics only
(i.e. to check that the profile is valid). Add an option to run BOLT
without writing any output file, similar to a dry run. This option
is triggered by supplying -o with "/dev/null".
(cherry picked from FBD29568632)
Summary:
During the initial indirect jump analysis, we used to assert that the
discovered jump table type matched the pattern of the corresponding
instruction sequence. E.g., for PIC jump table memory we expected the
PIC jump table instruction sequence. The assertions were too
conservative, as in the case of a mismatch we can mark the indirect jump
as having an unknown control flow. That should be sufficient to either
skip the function processing or rely on relocation information for
possible recovery of the control flow.
(cherry picked from FBD27255816)
Summary:
Fix a bug with instrumentation when trying to instrument
functions that share a jump table with multiple indirect
jumps. Usually, each indirect jump that uses a JT will have its own
copy of it. When this does not happen, we need to duplicate the jump
table safely, so we can split the edges correctly (each copy of the
jump table may have different split edges). For this to happen, we
need to correctly match the sequence of instructions that perform the
indirect jump to identify the base address of the jump table and patch
it to point to the new cloned JT. It was reported to us a case in
which the compiler generated suboptimal code to do an indirect jump
which our matcher failed to identify.
Fixesfacebookincubator/BOLT#126
(cherry picked from FBD27065579)
Summary:
Whenever BOLT encounters a data reference in code, it tries to convert
it into <Object+Offset> form. The primary reason behind this approach is
to support read-only data-reordering optimization. However, with the
current level of the linker and compiler support we don't have enough
information to always correctly restore the original <Object+Offset>.
E.g. with zero-sized symbols we have to speculate that the actual size
of the underlying object extends to the next symbol. Most of the time,
there will be an object pointed by a zero-sized symbol and even
if we are guessing incorrectly, there will be no harm in creating
references of such form.
The problem happens when there's no object corresponding to the original
symbol and the next object is an (unmarked) jump table:
A: # <- zero-sized object
.LJUMP_TABLE:
.long <entry1>
.long <entry2>
....
.LB:
.long 21
.LC:
.long 42
The jump table will be moved and all references past it (up to the next
named object) will be incorrectly updated.
We should not speculate about the size of A in a case like that and
treat all discovered data objects (and thus references) independently.
(cherry picked from FBD27005660)
Summary:
There is no real link between CU and TU, so relying on fact
that address are the same, and we are updating all of them.
(cherry picked from FBD28112114)
Summary:
This commit is the first step in rebasing all of BOLT
history in the LLVM monorepo. It also solves trivial build issues
by updating BOLT codebase to use current LLVM. There is still work
left in rebasing some BOLT features and in making sure everything
is working as intended.
History has been rewritten to put BOLT in the /bolt folder, as
opposed to /tools/llvm-bolt.
(cherry picked from FBD33289252)
Summary:
This fixes a bug with shrink wrapping when trying to move
push-pops in a function where we are not allowed to modify the
stack layout for alignment reasons. In this bug, we failed to
propagate alignment requirement upwards in the call graph from
function A to B when: (1) there is a cycle in the call graph and
(2) the distance from A to B is greater than 1 in the call graph
and (3) there is a node in the path from A to B, not including
A or B, that does not access parameters in the stack.
(cherry picked from FBD25315977)
Summary:
Fix corner case of insertion of updated CFI with unset `PrevBB`.
Handle it in the same way as inserting past hot-cold split point.
(cherry picked from FBD24943911)
Summary:
Support jump tables belonging to split fragments with entries
pointing back to parent functions.
While skipping such families of functions, make sure to use the
topmost fragment to ignore its fragments.
(cherry picked from FBD24907438)
Summary:
Introduce new BinaryFunction flag `IsCanonicalCFG`, which gets
unset by SCTC pass. Make DynoStats collection conditional on this
new flag.
SCTC leaves CFG in a state where branch counters of BBs with tail
calls/conditional tail calls are not available (except via annotations,
which get stripped by `lower-annotations`). Without branch
counters, DynoStats are invalid.
(cherry picked from FBD24558050)
Summary:
- Allow jump table entries to point to locations inside the function and its fragments.
Reasoning behind this is that jump table identification has the logic of stopping at entry which belongs to a function different from the one originally referencing jump table. This assumption is invalid for jump tables with entries pointing to both parent function and cold fragments, leading to "unclaimed PC-relative relocations" assertion.
- Add fragment identification heuristic based on function name regex and contiguous jump table entries.
Currently, parent-to-fragment relationship is set up based on interprocedural references – direct references from the parent function. These references don't include references through jump table.
Additionally, some fragments are only reachable through jump table. In that case, in order to fully consume jump table, add parent-to-fragment relationship during `analyzeJumpTable` using the following heuristics:
1. Fragment is identified as such based on name (contains `.cold.` part), but
2. Parent function is not set – no direct interprocedural references to that fragment, and
3. Fragment has the name of the form <parent>.cold(.\d+)
* For split functions with jump table entries spanning parent and fragments, mark parent and all fragments as ignored.
(cherry picked from FBD24456904)
Summary:
For interprocedural references to fragments, record them as
fragment entry points. Not registering these entry points leads to
UCE removing the blocks and "Undefined temporary symbol"
assertion.
(cherry picked from FBD24511281)
Summary:
When -hot-text is on, do not read __hot_start and __hot_end
from input (inserted by a linker script with the intent of ordering
functions). This can confuse BOLT into creating a function with this
name depending on which address the symbol lands and we will assert
when trying to emit our own __hot_start/__hot_end with symbol
redefinition.
(cherry picked from FBD24366636)
Summary:
When placing restore instructions in the shrink wrapping pass,
we typically put them right before the last instruction of a block at
the dominance frontier. If this instruction happened to have a prefix,
because the MC lib separates prefix into separate MCInsts, we would
accidentally put a load between a prefix and another instruction. Fix
this.
(cherry picked from FBD24295324)
Summary:
In analyzeRelocations, we extract the result of the relocation
from binary code to recreate the target of it in a few special cases.
For R_X86_64_32S relocations, however, we were neglecting the
possibility of the encoded value in the instruction to be negative.
(cherry picked from FBD24096347)
Summary:
Fix issue with splitting critical edges originating at
the same BB in ShrinkWrapping::splitFrontierCritEdges.
Splitting of critical edges originating at the same FromBB
wasn't handled correctly as the Frontier at index corresponding
to FromBB was overwritten with basic blocks created for
multiple DestinationBBs.
(cherry picked from FBD23232398)
Summary:
Right now, the SAVE_ALL sequence executed upon entry of both
of our runtime libs (hugify and instrumentation) will cause the stack to
not be aligned at a 16B boundary because it saves 15 8-byte regs. Change
the code sequence to adjust for that. The compiler may generate code
that assumes the stack is aligned by using movaps instructions, which
will crash.
(cherry picked from FBD22744307)
Summary:
If no profile data is provided, but only a user-provided order
file for functions, fix the placement of the __hot_end symbol.
(cherry picked from FBD22713265)
Summary:
Do not fail/assert when trying to reorder blocks that terminate
with JRCXZ/JECXZ/LOOP instructions. We cannot invert the condition of
these instructions, so just treat them accordingly in fixBranches().
(cherry picked from FBD22487107)
Summary:
Re-add tests removed because they used to depend on yaml2obj.
Rewrite them with an assembler (llvm-mc) and use the system linker to
produce a valid ELF as input to BOLT.
(cherry picked from FBD22323449)
Summary:
Some functions could be called at an address inside their function body.
Typically, these functions are written in assembly as C/C++ does not
have a multi-entry function concept. The addresses inside a function
body that could be referenced from outside are called secondary entry
points.
In BOLT we support processing functions with secondary/multiple entry
points. We used to mark basic blocks representing those entry points
with a special flag. There was only one problem - each basic block has
exactly one MCSymbol associated with it, and for the most efficient
processing we prefer that symbol to be local/temporary. However, in
certain scenarios, e.g. when running in non-relocation mode, we need
the entry symbol to be global/non-temporary.
We could create global symbols for secondary points ahead of time when
the entry point is marked in the symbol table. But not all such entries
are properly marked. This means that potentially we could discover an
entry point only after disassembling the code that references it, and
it could happen after a local label was already created at the same
location together with all its references. Replacing the local symbol
and updating the references turned out to be an error-prone process.
This diff takes a different approach. All basic blocks are created with
permanently local symbols. Whenever there's a need to add a secondary
entry point, we create an extra global symbol or use an existing one
at that location. Containing BinaryFunction maps a local symbol of a
basic block to the global symbol representing a secondary entry point.
This way we can tell if the basic block is a secondary entry point,
and we emit both symbols for all secondary entry points. Since secondary
entry points are quite rare, the overhead of this approach is minimal.
Note that the same location could be referenced via local symbol from
inside a function and via global entry point symbol from outside.
This is true for both primary and secondary entry points.
(cherry picked from FBD21150193)
Summary:
Indirect calls that use RSP to compute the target address would
break in instrumentation mode because we were adding instructions that
changed the stack pointer. Fix this.
(cherry picked from FBD20883791)
Summary:
Shrink wrapping has a mode where it will directly move push
pop pairs, instead of replacing them with stores/loads. This is an
ambitious mode that is triggered sometimes, but whenever matching with
a push, it would operate with the assumption that the restoring
instruction was a pop, not a load, otherwise it would assert. Fix this
assertion to bail nicely back to non-pushpop mode (use regular store and
load instructions).
(cherry picked from FBD20085905)
Summary:
I noticed when setting up a new repository for bolt that bolt tests
would fail unexpectedly when running `ninja check-bolt` and
`ninja check-llvm`. This turns out to be because dependencies for bolt
binaries were not specified in the CMake configuration so they were not
built before running the tests. This diff adds the dependencies to the
CMake configuration for check-bolt and check-llvm so that bolt binaries
are built before running tests.
(cherry picked from FBD17919505)
Summary:
The regular perf2bolt aggregation job is to read perf output directly.
However, if the data is coming from a database instead of perf, one
could write a query to produce a pre-aggregated file. This function
deals with this case.
The pre-aggregated file contains aggregated LBR data, but without binary
knowledge. BOLT will parse it and, using information from the
disassembled binary, augment it with fall-through edge frequency
information. After this step is finished, this data can be either
written to disk to be consumed by BOLT later, or can be used by BOLT
immediately if kept in memory.
File format syntax:
{B|F|f} [<start_id>:]<start_offset> [<end_id>:]<end_offset> <count>
[<mispred_count>]
B - indicates an aggregated branch
F - an aggregated fall-through (trace)
f - an aggregated fall-through with external origin - used to disambiguate
between a return hitting a basic block head and a regular internal
jump to the block
<start_id> - build id of the object containing the start address. We can
skip it for the main binary and use "X" for an unknown object. This will
save some space and facilitate human parsing.
<start_offset> - hex offset from the object base load address (0 for the
main executable unless it's PIE) to the start address.
<end_id>, <end_offset> - same for the end address.
<count> - total aggregated count of the branch or a fall-through.
<mispred_count> - the number of times the branch was mispredicted.
Omitted for fall-throughs.
Example
F 41be50 41be50 3
F 41be90 41be90 4
f 41be90 41be90 7
B 4b1942 39b57f0 3 0
B 4b196f 4b19e0 2 0
(cherry picked from FBD8887182)
Summary:
When a given function B, located after function A, references
one of A's basic blocks, it registers a new global symbol at the
reference address and update A's Labels vector via
BinaryFunction::addEntryPoint(). However, we don't update A's branch
targets at this point. So we end up with an inconsistent CFG, where the
basic block names are global symbols, but the internal branch operands
are still referencing the old local name of the corresponding blocks
that got promoted to an entry point. This patch fix this by detecting
this situation in addEntryPoint and iterating over all instructions,
looking for references to the old symbol and replacing them to use the
new global symbol (since this is now an entry point).
Fixesfacebookincubator/BOLT#26
(cherry picked from FBD8728407)
Summary:
While removing unreachable blocks, we may decide to remove a
block that is listed as a target in a jump table entry. If we do that,
this label will be then undefined and LLVM assembler will crash.
Mitigate this for now by not removing such blocks, as we don't support
removing unnecessary jump tables yet.
Fixesfacebookincubator/BOLT#20
(cherry picked from FBD8730269)
Summary:
Create folders and setup to make LIT run BOLT-only tests. Add
a test example. This will add a new make/ninja rule "check-bolt" that
the user can invoke to run LIT on this folder.
(cherry picked from FBD8595786)
Alexander Yermolovich