Summary:
BOLT sources are being moved under tools/llvm-bolt/src
and tools/llvm-bolt will contain more files such as LICENSE.txt,
README.txt, etc.
Remove trailing white spaces from our sources.
Create llvm.patch by running
> git diff f137ed238db11440f03083b1c88b7ffc0f4af65e include lib > \
tools/llvm-bolt/llvm.patch
README.txt has instructions on checking out sources and applying the
patch.
(cherry picked from FBD7878380)
Summary:
New profile writer was crashing as functions were lacking a profile
flags. Fix it by requiring flags when marking function as profiled.
Generate new profile for clang. The new profile has more coverage and
results in better overall improvement from BOLT. It was generated by
merging multiple runs of:
% perf record -e cycles:u -j any,u -F32000 -- \
./clang bf.cpp -O2 -std=c++11 -c -o /tmp/bf.o
(cherry picked from FBD7798580)
Summary:
Refactor MCInst comparison code to support target-dependent
functionality. This was necessary because AArch64 uses MCTargetExprs
that only the AArch64 backend knows how to unpack it and compare. Also
fix a bug where a relocation against a constant island would make BOLT
create a fixed reference against a code location in a similar way to
read-only data, so when we asked to -use-old-text, the code would break
for this particular HHVM function
(_ZN5folly2io4zlib18defaultZlibOptionsEv) because the reference now
contains invalid data, since the original .text was overwritten. Finally,
fix a bug with -update-debug-sections on AArch64 where the update
loop wasn't expecting a function with zero basic blocks, which can
happen on AArch64 because some functions contain just a constant
island.
(cherry picked from FBD7679244)
Summary: Modifying parameters of block reordering algorithm that result in better performance. Additionally extending some cache-related metrics
(cherry picked from FBD7578336)
Summary:
Whenever building BOLT in an AArch64 box, we need to make sure
we do not run tests that are excluse for x86. This diff also adds a tag
for expensive tests, so the user can disable them, which is useful when
using a memory-constrained machine to run BOLT tests. It also removes
ifdefs that caused BOLT to behave diferently when running in a non-x86
host. Finally, it changes a case where we depended on updated libstdc++
implementation for insert to make the codebase more friendly with boxes
that do not have the newer version of the lib.
(cherry picked from FBD7625715)
Summary:
Restore the optimization with some modifications:
* Only enabled in relocation mode.
* Covers instructions other than TEST/CMP.
* Prints missed macro-fusion opportunities for input.
* By default enabled for all hot code.
* Without profile enabled for all code.
The new command-line option:
-align-macro-fusion - fix instruction alignment for macro-fusion (x86 relocation mode)
=none - do not insert alignment no-ops for macro-fusion
=hot - only insert alignment no-ops on hot execution paths (default)
=all - always align instructions to allow macro-fusion
(cherry picked from FBD7644042)
Summary:
Since BOLT can use relocations in the binary automatically, it's not
always clear if we are operating in relocation mode or not. This diff
adds "BOLT-INFO" message indicating if the relocation mode in ON.
(cherry picked from FBD7557492)
Summary:
Expanded YAML profile format to support different kinds of profile
including LBR and non-LBR (and memevents in the future).
The profile now starts with a header that includes the profile
description. "profile-flags" field includes either "lbr" or "sample",
but not both at the same time. It could also include "memevent" in
addition to other flags.
For now, the only way to generate non-LBR YAML profile is through
conversion. Once task is done, it should be possible to use
perf2bolt for it.
(cherry picked from FBD7595693)
Summary:
merge-fdata now operates on .fdata files in YAML format. The previous
format is not supported, which means that non-LBR data could not be
merged and memory data has to be merged with "cat" command.
(cherry picked from FBD7544031)
Summary:
This diff has 3 fixes. First fixes the way relocations are read
and interpreted for AArch64, so the references are preserved correctly.
Second, it fixes constant islands to be able to live in the very first
address of a function (which means there is no code, but this function
contains just a constant island).
Third, it fixes function splitting to do not outline entry points for
AArch64. This was done because some functions may load pointers to its
internal basic blocks, issueing a short-range ADR instruction to do so
without its pair ADRP (since the size of the function is supposed to
be small). But when we move this block to a cold region, that is not
the case anymore. Since blocks with a reference are marked as entry
points, we conservatively disable outlining for them in AArch64.
(cherry picked from FBD7505067)
Summary:
Change the way annotations are stored and processed.
Embed annotation type/index into immediate value stored as an operand.
This limits the effective range of values that could be stored as
annotations to 56 bits, which is still plenty for most integer types
that we use and for pointers on real systems. High 8 bits are reserved
for storing annotation type/index.
Expand the interface for general annotations to include reference to
annotations by index. The main purpose of this interface is to improve
performance of annotations that are used by heavy (>O(N)) algorithms,
such as data flow analysis.
For -frame-opt pass, new memory usage and processing times are slightly
better compared to those before refactoring.
(cherry picked from FBD7492017)
Summary:
Use MCPlus::getNumPrimeOperands() to get the real number of operands
on MCInst. Alternatively, use MCInstrDesc::getNumOperands().
(cherry picked from FBD7507666)
Summary:
When we erase invalid/unreachable basic blocks, we have to remove them
from a list of predecessors of regular blocks, otherwise the CFG will be
left in a broken state containing references to removed basic blocks.
(cherry picked from FBD7464292)
Summary:
We verify that relocation information matches a value stored in a
binary, i.e. "ExtractedValue == SymbolValue + Addend". However, because
of the size of the relocation, and the fact that an addend is always
of type int64_t, we have to sign-extend the extracted value, and then we
might get mismatch in higher bits in certain scenarios. Hence, we should
only compare values that are truncated to a relocation size.
Discovered while processing hhvm binary with modified compiler flags.
(cherry picked from FBD7462559)
Summary:
Getting a forward iterator from reverse iterator was implemented
incorrectly. For some reason erase worked on it, but it's clearly wrong
and printing the instruction (before the deletion) results in an error.
(cherry picked from FBD7457457)
Summary:
Changes that we made to MCInst, MCOperand, MCExpr, etc. are now all
moved into tools/llvm-bolt. That required a change to the way we handle
annotations and any extra operands for MCInst.
Any MCPlus information is now attached via an extra operand of type
MCInst with an opcode ANNOTATION_LABEL. Since this operand is MCInst, we
attach extra info as operands to this instruction. For first-level
annotations use functions to access the information, such as
getConditionalTailCall() or getEHInfo(), etc. For the rest, optional or
second-class annotations, use a general named-annotation interface such
as getAnnotationAs<uint64_t>(Inst, "Count").
I did a test on HHVM binary, and a memory consumption went down a little
bit while the runtime remained the same.
(cherry picked from FBD7405412)
Summary:
Some improvements for CFG construction:
- getting rid of fallthrough-inferrence, as this is already
done DataAggregator;
- adjusting block counts for blocks with non-zero outgoing edges
to make sure they're not outlined;
- making sure that all functions (including non-simple ones) are
reordered and placed in the hot section.
The main goal of the diff is to make sure that constructed CFG graphs
exactly correspond to the input profile data.
(cherry picked from FBD7323205)
Summary:
The binary had some unexpected ovelapping symbols:
.str.34.llvm.2944770977690351622/1 address = 0x48e9ec7, next address =
0x48e9ed2, size = 21
PG.LC135/1 address = 0x48e9ed2, next address = 0x48e9eef, size = 29
BOLT wasn't expecting this type of overlap when generating HOLE symbols,
so it was asserting. I've changed the code to deal with this case.
I'll need to change the reordering pass to mark these types of symbols
as unmoveable for now.
(cherry picked from FBD7304195)
Summary: This assertion was making sure that when we patched up symbol sizes that we wouldn't modify the size of a symbol that has already had its size set. The issue here is that private symbols are sometimes composed of multiple objects internally (e.g. jump tables). In this particular case a jump table started at the same address as the private data blob it was contained in. Currently, there isn't any good way of differentiating symbols that start at the same address (except possibly using multimaps for certain data structures). I'm hacking around it by modifying the assertion to ignore jump tables and skip setting the size when it has already been set. This shouldn't affect any existing optimizations since the only thing that depended on sizes is data reordering and that currently ignores jump tables and private data blobs.
(cherry picked from FBD7269207)
Summary:
Refactor architecture-specific code out of llvm into llvm-bolt.
Introduce MCPlusBuilder, a class that is taking over MCInstrAnalysis
responsibilities, i.e. creating, analyzing, and modifying instructions.
To access the builder use BC->MIB, i.e. substitute MIA with MIB.
MIB is an acronym for MCInstBuilder, that's what MCPlusBuilder used
to be. The name stuck, and I find it better than MPB.
Instructions are still MCInst, and a bunch of BOLT-specific code still
lives in LLVM, but the staff under Target/* is significantly reduced.
(cherry picked from FBD7300101)
Summary:
In new ORC, the sequence of how sections are allocated and loaded is
changed. Now everything is delayed until emitAndFinalize() is called,
and all actions are supposed to happen via notification functors.
There are two functors that we pass to new ObjectLinkingLayer object.
One is used to notify when objects are loaded, and the other - once they
are finalized. We use the first one to remap sections to proper
addresses, and that's the earliest place where we can do it. However,
ORC decides to update symbols right before that, and as a result they
are updated with non-mapped values.
There are two possible fixes for that. This diff postpones the update to
the symbol table until the notifier is called. I don't know what other
tools depend on the existing sequence, and the proper fix may involve
creating a third notifier to be called before the symbol table update.
(cherry picked from FBD7280973)
Summary:
Rebased version revealed a mistake when computing the dataflow
for the "remove-unused-stores" optimization. This is disabled in prod but
it doesn't hurt to fix it, so the tests for the rebased bolt go green
again.
(cherry picked from FBD7253418)
Summary:
This is a simple bolt-based tool that instantiates two
RewriteInstances objects and compares them. Add a method to
RewriteInstance to enable us to compare two objects. Include a mechanism
to match functions from binary 1 to binary 2 and finally print the
largest differences in profiling data from one binary to another.
(cherry picked from FBD6517076)
Summary:
This makes it possible to do adjustments to all functions based on
information gained during disassembly. E.g. if we detect an entry point
after the CFG for a function is constructed, we have to take a
conservative approach, and mark such function as non-simple. Now we have
this information before building the CFG. This could also be used to do
other processing/post-processing on disassembled functions that might
affect CFG construction of other functions (e.g. early detection of
functions that never return).
The drawback of this approach is that we lose cache locality and some
processing performance as a result. I've measured 5 second difference
on HHVM binary.
(cherry picked from FBD7258466)
Summary:
This is preparation work for static data reordering.
I've created a new class called BinaryData which represents a symbol
contained in a section. It records almost all the information relevant
for dealing with data, e.g. names, address, size, alignment, profiling
data, etc. BinaryContext still stores and manages BinaryData objects
similar to how it managed symbols and global addresses before. The
interfaces are not changed too drastically from before either. There is
a bit of overlap between BinaryData and BinaryFunction. I would have
liked to do some more refactoring to make a BinaryFunctionFragment that
subclassed from BinaryData and then have BinaryFunction be composed or
associated with BinaryFunctionFragments.
I've also attempted to use (symbol + offset) for when addresses are
pointing into the middle of symbols with known sizes. This changes the
simplify rodata loads optimization slightly since the expression on an
instruction can now also be a (symbol + offset) rather than just a symbol.
One of the overall goals for this refactoring is to make sure every
relocation is associated with a BinaryData object. This requires adding
"hole" BinaryData's wherever there are gaps in a section's address space.
Most of the holes seem to be data that has no associated symbol info. In
this case we can't do any better than lumping all the adjacent hole
symbols into one big symbol (there may be more than one actual data
object that contributes to a hole). At least the combined holes should
be moveable.
Jump tables have similar issues. They appear to mostly be sub-objects
for top level local symbols. The main problem is that we can't recognize
jump tables at the time we scan the symbol table, we have to wait til
disassembly. When a jump table is discovered we add it as a sub-object
to the existing local symbol. If there are one or more existing
BinaryData's that appear in the address range of a newly created jump
table, those are added as sub-objects as well.
(cherry picked from FBD6362544)
Summary:
Fix a few ShrinkWrapping bugs:
- Using push-pop mode in a function that required aligned stack
- Correctly update the edges in jump tables after splitting critical
edges
- Fix stack pointer restores based on RBP + offset, when we change the
stack layout in push-pop mode.
(cherry picked from FBD6755232)
Summary:
Fix a bug introduced by rebasing with respect to aligned ULEBs.
This wasn't breaking anything but it is good to keep LDSA aligned.
(cherry picked from FBD7094742)
Summary:
This is a big refactoring of the section handling code. I've removed the SectionInfoMap and NoteSectionInfo and stored all the associated info about sections in BinaryContext and BinarySection classes. BinarySections should now hold all the info we care about for each section. They can be initialized from SectionRefs but don't necessarily require one to be created. There are only one or two spots that needed access to the original SectionRef to work properly.
The trickiest part was making sure RewriteInstance.cpp iterated over the proper sets of sections for each of it's different types of processing. The different sets are broken down roughly as allocatable and non-alloctable and "registered" (I couldn't think up a better name). "Registered" means that the section has been updated to include output information, i.e. contents, file offset/address, new size, etc. It may help to have special iterators on BinaryContext to iterate over the different classes to make things easier. I can do that if you guys think it is worthwhile.
I found pointee_iterator in the llvm ADT code. Use that for iterating over BBs in BinaryFunction rather than the custom iterator class.
(cherry picked from FBD6879086)
Summary:
When we move a jump table to either hot or cold new section
(-jump-tables=move), we rely on a number of taken branches from the table
to decide if it's hot or cold. However, if the function is non-simple, we
always get 0 count, and always move the table to the cold section.
Instead, we should make a conservative decision based on the execution
count of the function.
(cherry picked from FBD7058127)
Summary:
Speed of cache+ by skipping mallocs on vectors.
Although this change speeds up the algorithm by 2x, this is still not
enough for some binaries where some functions have ~2500 hot basic
blocks. Hence, introduce a threshold for expensive optimizations in
CachePlusReorderAlgorithm. If the number of hot basic blocks exceeds
the threshold (2048 by default), we use a cheaper version, which is
quite fast.
(cherry picked from FBD6928075)
Summary:
Do a better job of recording fall-through branches in new profile mode
(-prof-compat-mode=0). For this we need to record offsets for all
instructions that are last in the containing basic block.
Change the way we convert conditional tail calls. Now we never reverse
the condition. This is required for better profile matching.
The original approach of preserving the direction was controversial
to start with.
Add "-infer-fall-throughs" option (on by default) to allow disabling
inference of fall-through edge counts.
(cherry picked from FBD6994293)
Summary:
Prioritize functions with 100% name match when doing LTO "fuzzy"
name matching. Avoid re-assigning profile to a function.
(cherry picked from FBD6992179)
Summary:
In relocation mode trap on entry to any function that has AVX-512
instructions. This is controlled by "-trap-avx512" option which is on
by default. If the option is disabled and AVX-512 instruction is seen
in relocation mode, then we abort while re-writing the binary.
(cherry picked from FBD6893165)
Summary:
This commit includes all code necessary to make BOLT working again
after the rebase. This includes a redesign of the EHFrame work,
cherry-pick of the 3dnow disassembly work, compilation error fixes,
and port of the debug_info work. The macroop fusion feature is not
ported yet.
The rebased version has minor changes to the "executed instructions"
dynostats counter because REP prefixes are considered a part of the
instruction it applies to. Also, some X86 instructions had the "mayLoad"
tablegen property removed, which BOLT uses to identify and account
for loads, thus reducing the total number of loads reported by
dynostats. This was observed in X86::MOVDQUmr. TRAP instructions are
not terminators anymore, changing our CFG. This commit adds compensation
to preserve this old behavior and minimize tests changes. debug_info
sections are now slightly larger. The discriminator field in the line
table is slightly different due to a change upstream. New profiles
generated with the other bolt are incompatible with this version
because of different hash values calculated for functions, so they will
be considered 100% stale. This commit changes the corresponding test
to XFAIL so it can be updated. The hash function changes because it
relies on raw opcode values, which change according to the opcodes
described in the X86 tablegen files. When processing HHVM, bolt was
observed to be using about 800MB more memory in the rebased version
and being about 5% slower.
(cherry picked from FBD7078072)
Summary:
This fixes the increased memory consumption introduced in an earlier
diff while I was working on new profiling infra.
The increase came from a delayed release of memory allocated to
intermediate structures used to build CFG. In this diff we release
them ASAP, and don't keep them for all functions at the same time.
(cherry picked from FBD6890067)
Summary:
Limiting "Offset" annotation only to instructions that actually
need it, improves the memory consumption on HHVM binary by 1GB.
(cherry picked from FBD6878943)
Summary:
SCTC was incorrectly swapping BranchInfo when reversing the branch condition. This was wrong because when we remove the successor BB later, it removes the BranchInfo for that BB. In this case the successor would be the BB with the stats we had just swapped.
Instead leave BranchInfo as it is and read the branch count from the false or true branch depending on whether we reverse or replace the branch, respectively. The call to removeSuccessor later will remove the unused BranchInfo we no longer care about.
(cherry picked from FBD6876799)
Summary: Register all sections with BinaryContext. Store all sections in a set ordered by (address, size, name). Add two separate maps to lookup sections by address or by name. Non-allocatable sections are not stored in the address->section map since they all "start" at 0.
(cherry picked from FBD6862973)
Summary:
Handle types CU list in `updateGdbIndexSection`.
It looks like the types part of `.gdb_index` isn't empty when `-fdebug-types-section` is used. So instead of aborting, we copy the part to new `.gdb_index` section.
(cherry picked from FBD6770460)
Summary:
When we read profile for functions, we initialize counts for entry
blocks first, and then populate counts for all blocks based
on incoming edges.
During the second phase we ignore the entry blocks because we expect
them to be already initialized. For the primary entry at offset 0 it's
the correct thing to do, since we treat all incoming branches as calls
or tail calls. However, for secondary entries we only consider external
edges to be from calls and don't increase entry count if an edge
originates from inside the function. Thus we need to update the
secondary entry basic block counts with internal edges too.
(cherry picked from FBD6836817)
Summary:
A test is asserting on impossible addresses coming from
perf.data, instead of just reporting it as bad data. Fix this behavior.
(cherry picked from FBD6835590)
Summary:
Speeding up cache+ algorithm.
The idea is to find and merge "fallthrough" successors before main
optimization. For a pair of blocks, A and B, block B is the fallthrough
successor of A, if (i) all jumps (based on profile) from A goes to B
and (ii) all jumps to B are from A.
Such blocks should be adjacent in an optimal ordering, and should
not be considered for splitting. (This gives the speed up).
The gap between cache and cache+ reduced from ~2m to ~1m.
(cherry picked from FBD6799900)
Summary:
Refactor the relocation anaylsis code. It should be a little better at validating
that the relocation value matches up with the symbol address + addend stored in the
relocation (except on aarch64). It is also a little better at finding the symbol
address used to do the lookup in BinaryContext, rather than just using symbol
address + addend.
(cherry picked from FBD6814702)
Summary: Add BinarySection class that is a wrapper around SectionRef. This is refactoring work for static data reordering.
(cherry picked from FBD6792785)
Summary:
Rewrite how data/code markers are interpreted, so the code
can have constant islands essentially anywhere. This is necessary to
accomodate custom AArch64 assembly code coming from mozjpeg. Allow
any function to refer to the constant island owned by any other
function. When this happens, we pull the constant island from the
referred function and emit it as our own, so it will live nearby
the code that refers to it, allowing us to freely reorder functions
and code pieces. Make bolt more strict about not changing anything
in non-simple ARM functions, as we need to preserve offsets for
those functions we don't interpret their jump tables (currently
any function with jump tables in ARM is non-simple and is left
untouched).
(cherry picked from FBD6402324)
Maksim Panchenko