Commit Graph

2307 Commits

Author SHA1 Message Date
Diego Novillo 10cf124bb9 SamplePGO - Reduce memory utilization by 10x.
DenseMap is the wrong data structure to use for sample records and call
sites.  The keys are too large, causing massive core memory growth when
reading profiles.

Before this patch, a 21Mb input profile was causing the compiler to grow
to 3Gb in memory.  By switching to std::map, the compiler now grows to
300Mb in memory.

There still are some opportunities for memory footprint reduction. I'll
be looking at those next.

llvm-svn: 255389
2015-12-11 23:21:38 +00:00
Artur Pilipenko 7ae49ac619 PruneEH pass incorrectly reports that a change was made
Reviewed By: reames

Differential Revision: http://reviews.llvm.org/D14097

llvm-svn: 255343
2015-12-11 16:30:26 +00:00
Teresa Johnson 9f2ff9c669 [ThinLTO] Debug message cleanup (NFC)
Added some missing spaces between the module identifier and the start of
the debug message. Also added a ":" after the module identifier to make
this look a little nicer.

llvm-svn: 255259
2015-12-10 16:39:07 +00:00
Sanjoy Das ccd14566e2 Add arg_begin() and arg_end() to CallInst and InvokeInst; NFCI
- This simplifies the CallSite class, arg_begin / arg_end are now
   simple wrapper getters.

 - In several places, we were creating CallSite instances solely to call
   arg_begin and arg_end.  With this change, that's no longer required.

llvm-svn: 255226
2015-12-10 06:39:02 +00:00
Rafael Espindola 9edc3b8403 Don't assign a temporary string to a StringRef.
Should fix the windows debug and asan bots.

llvm-svn: 255149
2015-12-09 20:41:10 +00:00
Teresa Johnson 7f961e14d3 [ThinLTO] FunctionImport pass can take a const index pointer (NFC)
llvm-svn: 255140
2015-12-09 19:39:47 +00:00
Mehdi Amini 7e88d0da38 The current importing scheme is processing one function at a time,
loading the source Module, linking the function in the destination
module, and destroying the source Module before repeating with the
next function to import (potentially from the same Module).

Ideally we would keep the source Module alive and import the next
Function needed from this Module. Unfortunately this is not possible
because the linker does not leave it in a usable state.

However we can do better by first computing the list of all candidates
per Module, and only then load the source Module and import all the
function we need for it.

The trick to process callees is to materialize function in the source
module when building the list of function to import, and inspect them
in their source module, collecting the list of callees for each
callee.

When we move the the actual import, we will import from each source
module exactly once. Each source module is loaded exactly once.
The only drawback it that it requires to have all the lazy-loaded
source Module in memory at the same time.

Currently this patch already improves considerably the link time,
a multithreaded link of llvm-dis on my laptop was:

  real  1m12.175s  user  6m32.430s sys  0m10.529s

and is now:

  real  0m40.697s  user  2m10.237s sys  0m4.375s

Note: this is the full link time (linker+Import+Optimizer+CodeGen)

Differential Revision: http://reviews.llvm.org/D15178

From: Mehdi Amini <mehdi.amini@apple.com>
llvm-svn: 255100
2015-12-09 08:17:35 +00:00
Sanjoy Das 48945cdc15 [OperandBundles] Have PruneEH work correct with operand bundles.
For an invoke with operand bundles, the [op_begin(), op_end()-3] range
can contain things other than invoke arguments.  This change teaches
PruneEH to use arg_begin() and arg_end() explicitly.

llvm-svn: 255073
2015-12-08 23:16:52 +00:00
Mehdi Amini 5411d0510c Fix/Improve Debug print in FunctionImport pass
From: Mehdi Amini <mehdi.amini@apple.com>
llvm-svn: 255071
2015-12-08 23:04:19 +00:00
Mehdi Amini d16c8065ff Remove caching in FunctionImport: a Module can't be reused after being linked from
The Linker destroys the source module (API change coming to make it explicit)

From: Mehdi Amini <mehdi.amini@apple.com>
llvm-svn: 255064
2015-12-08 22:39:40 +00:00
Teresa Johnson 5fcbdb717c [ThinLTO] Support for specifying function index from pass manager
Summary:
Add a field on the PassManagerBuilder that clang or gold can use to pass
down a pointer to the function index in memory to use for importing when
the ThinLTO backend is triggered. Add support to supply this to the
function import pass.

Reviewers: joker.eph, dexonsmith

Subscribers: davidxl, llvm-commits, joker.eph

Differential Revision: http://reviews.llvm.org/D15024

llvm-svn: 254926
2015-12-07 19:21:11 +00:00
Mehdi Amini 311fef6ea5 clang-format FunctionImport after refactoring (NFC)
From: Mehdi Amini <mehdi.amini@apple.com>
llvm-svn: 254585
2015-12-03 02:58:14 +00:00
Mehdi Amini c8c551701e Refactor FunctionImporter::importFunctions with a helper function to process the Worklist (NFC)
This precludes some more functional changes to perform bulk imports.

From: Mehdi Amini <mehdi.amini@apple.com>
llvm-svn: 254583
2015-12-03 02:37:33 +00:00
David Majnemer 70497c696a Move EH-specific helper functions to a more appropriate place
No functionality change is intended.

llvm-svn: 254562
2015-12-02 23:06:39 +00:00
Mehdi Amini ffe2e4aae0 Change ModuleLinker to take a set of GlobalValues to import instead of a single one
For efficiency reason, when importing multiple functions for the same Module,
we can avoid reparsing it every time.

Differential Revision: http://reviews.llvm.org/D15102

From: Mehdi Amini <mehdi.amini@apple.com>
llvm-svn: 254486
2015-12-02 04:34:28 +00:00
Mehdi Amini a11bdc8ef7 Modify FunctionImport to take a callback to load modules
When linking static archive, there is no individual module files to
load. Instead they can be mmap'ed and could be initialized from a
buffer directly. The callback provide flexibility to override the
scheme for loading module from the summary.

Differential Revision: http://reviews.llvm.org/D15101

From: Mehdi Amini <mehdi.amini@apple.com>
llvm-svn: 254479
2015-12-02 02:00:29 +00:00
Rafael Espindola 0e309fe860 Use references now that it is natural to do so.
The linker never takes ownership of a module or changes which module it
is refering to, making it natural to use references.

llvm-svn: 254449
2015-12-01 19:50:54 +00:00
Teresa Johnson 430110cc0b [ThinLTO] Wrap dbgs() output in DEBUG macro
Missed in a couple places.

llvm-svn: 254422
2015-12-01 17:12:10 +00:00
Teresa Johnson d582f5b3f8 [ThinLTO] Remove stale comment (NFC)
Stale as of r254036 which added basic profitability check.

llvm-svn: 254421
2015-12-01 16:45:23 +00:00
Diego Novillo 7ff0a174d1 SamplePGO - Do not use std::to_string in diagnostics.
This fixes buildbots in systems that std::to_string is not present. It
also tidies the output of the diagnostic to render doubles a bit better
(thanks Ben Kramer for help with string streams and format).

llvm-svn: 254261
2015-11-29 18:23:26 +00:00
Diego Novillo 84f06cc835 SamplePGO - Add initial support for inliner annotations.
This adds two thresholds to the sample profiler to affect inlining
decisions: the concept of global hotness and coldness.

Functions that have accumulated more than a certain fraction of samples at
runtime, are annotated with the InlineHint attribute. Conversely,
functions that accumulate less than a certain fraction of samples, are
annotated with the Cold attribute.

This is very similar to the hints emitted by Clang when using
instrumentation profiles.

Notice that this is a very blunt instrument. A function may have
globally collected a significant fraction of samples, but that does not
necessarily mean that every callsite for that function is hot.

Ideally, we would annotate each callsite with the samples collected at
that callsite. This way, the inliner can incorporate all these weights
into its cost model.

Once the inliner offers this functionality, we can change the hints
emitted here to a more precise per-callsite annotation. For now, this is
providing some measure of speedups with our internal benchmarks. I've
observed speedups of up to 23% (though the geo mean is about 3%). I expect
these numbers to improve as the inliner gets better annotations.

llvm-svn: 254212
2015-11-27 23:14:51 +00:00
Diego Novillo b579240875 SamplePGO - Fix default threshold for hot callsites.
Based on testing of internal benchmarks, I'm lowering this threshold to
a value of 0.1%.  This means that SamplePGO will respect 99.9% of the
original inline decisions when following a profile.

The performance difference is noticeable in some tests. With the
previous threshold, the speedups over baseline -O2 was about 0.63%. With
the new default, the speedups are around 3% on average.

The point of this threshold is not to do more aggressive inlining. When
an inlined callsite crosses this threshold, SamplePGO will redo the
inline decision so that it can better apply the input profile.

By respecting most original inline decisions, we can apply more of the
input profile because the shape of the code follows the profile more
closely.

In the next series, I'll be looking at adding some inline hints for the
cold callsites and for toplevel functions that are hot/cold as well.

llvm-svn: 254211
2015-11-27 23:14:49 +00:00
Rafael Espindola 8934577171 Disallow aliases to available_externally.
They are as much trouble as aliases to declarations. They are requiring
the code generator to define a symbol with the same value as another
symbol, but the second symbol is undefined.

If representing this is important for some optimization, we could add
support for available_externally aliases. They would be *required* to
point to a declaration (or available_externally definition).

llvm-svn: 254170
2015-11-26 19:22:59 +00:00
Rong Xu 25c106b347 [PGO] Revert revision r254021,r254028,r254035
Revert the above revision due to multiple issues.

llvm-svn: 254040
2015-11-24 23:49:08 +00:00
Teresa Johnson 3930361969 [ThinLTO] Add option to limit importing based on instruction count
Add a simple initial heuristic to control importing based on the number
of instructions recorded in the function's summary. Add option to
control the limit, and test using option.

llvm-svn: 254036
2015-11-24 22:55:46 +00:00
Diego Novillo 0b6985a3c6 SamplePGO - Add test for hot/cold inlined functions.
When the original binary is executed and sampled, the resulting profile
contains information on the original inline stack. We currently follow
the original inline plan if we notice that the inlined callsite has more
than 0 samples to it.

A better way is to determine whether the callsite is actually worth
inlining. If the callsite accumulates a small fraction of the samples
spent in the parent function, then we don't want to bother inlining it
(as it means that the callsite is actually cold).

This patch introduces a threshold expressed in percentage of samples
in relation to the parent function.  If the callsite uses less than N%
of the total samples used by its parent, the original inline decision is
not re-applied.

I've set the threshold to the very arbitrary value of 5%. I'm yet to do
any actual experiments to see what's a good value. I wanted to separate
the basic mechanism from the tuning.

llvm-svn: 254034
2015-11-24 22:38:37 +00:00
Rong Xu 1b665ca707 [PGO] MST based PGO instrumentation infrastructure
This patch implements a minimum spanning tree (MST) based instrumentation for
PGO. The use of MST guarantees minimum number of CFG edges getting
instrumented. An addition optimization is to instrument the less executed
edges to further reduce the instrumentation overhead. The patch contains both the
instrumentation and the use of the profile to set the branch weights.

Differential Revision: http://reviews.llvm.org/D12781

llvm-svn: 254021
2015-11-24 21:31:25 +00:00
Teresa Johnson d450da3281 [ThinLTO] Refactor function body scan during importing into helper (NFC)
llvm-svn: 254020
2015-11-24 21:15:19 +00:00
Teresa Johnson 130de7af7f [ThinLTO] Enable iterative importing in FunctionImport pass
Analyze imported function bodies and add any new external calls to
the worklist for importing. Currently no controls on the importing
so this will end up importing everything possible in the call tree
below the importing module. Basic profitability checks coming next.

Update test to check for iteratively inlined functions.

llvm-svn: 254011
2015-11-24 19:55:04 +00:00
Teresa Johnson 17626654fd [ThinLTO] Fix FunctionImport alias checking and test
Skip imports for weak_any aliases as well. Fix the test to check
non-import of weak aliases and functions, and import of normal alias.

llvm-svn: 253991
2015-11-24 16:10:43 +00:00
Ismail Donmez 65487e2d7e Fix build after r253954
llvm-svn: 253969
2015-11-24 09:48:09 +00:00
Mehdi Amini 42418aba58 Add a FunctionImporter helper to perform summary-based cross-module function importing
Summary:
This is a helper to perform cross-module import for ThinLTO. Right now
it is importing naively every possible called functions.

Reviewers: tejohnson

Subscribers: dexonsmith, llvm-commits

Differential Revision: http://reviews.llvm.org/D14914

From: Mehdi Amini <mehdi.amini@apple.com>
llvm-svn: 253954
2015-11-24 06:07:49 +00:00
Diego Novillo 243ea6a7d6 SamplePGO - Add coverage tracking for samples.
The existing coverage tracker counts the number of records that were used
from the input profile. An alternative view of coverage is to check how
many available samples were applied.

This way, if the profile contains several records with few samples, it
doesn't really matter much that they were not applied. The more
interesting records to apply are the ones that contribute many samples.

llvm-svn: 253912
2015-11-23 20:12:21 +00:00
Diego Novillo 1ca881c4bb SamplePGO - Clear coverage tracking when clearing per-function data.
llvm-svn: 253877
2015-11-23 16:30:17 +00:00
Diego Novillo 39ab68f39b SamplePGO - Use newly introduced local variable. NFC.
llvm-svn: 253868
2015-11-23 15:24:13 +00:00
Diego Novillo 5fb49e5c5f SamplePGO - Do not count never-executed inlined functions when computing coverage.
If a function was originally inlined but not actually hot at runtime,
its samples will not be counted inside the parent function. This throws
off the coverage calculation because it expects to find more used
records than it should.

Fixed by ignoring functions that will not be inlined into the parent.
Currently, this is inlined functions with 0 samples.  In subsequent
patches, I'll change this to mean "cold" functions.

llvm-svn: 253716
2015-11-20 21:46:38 +00:00
Tilmann Scheller 925b193eed Revert "[FunctionAttrs] Remove redundant assignment."
This reverts r253661.

Turns out that the assignment is not redundant (despite the Clang static analyzer claiming the opposite).

The variable is being used by the lambda function AddUsersToWorklistIfCapturing().

llvm-svn: 253696
2015-11-20 19:17:10 +00:00
Diego Novillo df544a098a SamplePGO - Add line offset and discriminator information to sample reports.
While debugging some sampling coverage problems, I found this useful:
When applying samples from a profile, it helps to also know what line
offset and discriminator the sample belongs to. This makes it easy to
correlate against the input profile.

llvm-svn: 253670
2015-11-20 15:39:42 +00:00
Tilmann Scheller 1e929f97f6 [FunctionAttrs] Remove redundant assignment.
Identified by the Clang static analyzer.

llvm-svn: 253661
2015-11-20 12:51:58 +00:00
James Molloy 1d695a09dd [GlobalOpt] Localize some globals that have non-instruction users
We currently bail out of global localization if the global has non-instruction users. However, often these can be simple bitcasts or constant-GEPs, which we can easily turn into instructions before localizing. Be a bit more aggressive.

llvm-svn: 253584
2015-11-19 18:04:33 +00:00
James Molloy 0ecdbe7d6b [FunctionAttrs] Provide a mechanism for adding function attributes from the command line
This provides a way to force a function to have certain attributes from the command line. This can be useful when debugging or doing workload exploration, where manually editing IR is tedious or not possible (due to build systems etc).

The syntax is -force-attribute=function_name:attribute_name

All function attributes are parsed except alignstack as it requires an argument.

llvm-svn: 253550
2015-11-19 08:49:57 +00:00
James Molloy 9ad4f22538 [LTO] Add an early run of functionattrs
Because we internalize early, we can potentially mark a bunch of functions as norecurse. Do this before globalopt.

llvm-svn: 253451
2015-11-18 11:24:42 +00:00
Elena Demikhovsky 3ec9e15ad4 Vector of pointers in function attributes calculation
While setting function attributes we check all instructions that may access memory. For a call instruction we check all arguments. The special check is required for pointers.
I added vector-of-pointers to the call arguments types that should be checked.

Differential Revision: http://reviews.llvm.org/D14693

llvm-svn: 253363
2015-11-17 19:30:51 +00:00
James Molloy d4d2357f26 [GlobalOpt] Address post-commit review comments on r253168
Address Duncan Exon Smith's comments on D14148, which was added after the patch had been LGTM'd and committed:
  * clang-format one area where whitespace diffs occurred.
  * Add a threshold to limit the store/load dominance checks as they are quadratic.

llvm-svn: 253192
2015-11-16 10:16:22 +00:00
Benjamin Kramer 83709b1c1e Move helper classes into anonymous namespaces. NFC.
llvm-svn: 253189
2015-11-16 09:01:28 +00:00
James Molloy 9c7d4d8855 [GlobalOpt] Demote globals to locals more aggressively
Global to local demotion can speed up programs that use globals a lot. It is particularly useful with LTO, when the entire call graph is known and most functions have been internalized.

For a global to be demoted, it must only be accessed by one function and that function:
  1. Must never recurse directly or indirectly, else the GV would be clobbered.
  2. Must never rely on the value in GV at the start of the function (apart from the initializer).

GlobalOpt can already do this, but it is hamstrung and only ever tries to demote globals inside "main", because C++ gives extra guarantees about how main is called - once and only once.

In LTO mode, we can often prove the first property (if the function is internal by this point, we know enough about the callgraph to determine if it could possibly recurse). FunctionAttrs now infers the "norecurse" attribute for this reason.

The second property can be proven for a subset of functions by proving that all loads from GV are dominated by a store to GV. This is conservative in the name of compile time - this only requires a DominatorTree which is fairly cheap in the grand scheme of things. We could do more fancy stuff with MemoryDependenceAnalysis too to catch more cases but this appears to catch most of the useful ones in my testing.

llvm-svn: 253168
2015-11-15 14:21:37 +00:00
James Molloy 33e7345886 [GlobalOpt] Make sure all debug lines end with '\n'
GlobalVariable::print() used to emit a newline. It hasn't for a while now, but these debug lines weren't updated.

llvm-svn: 253030
2015-11-13 11:05:13 +00:00
James Molloy ea31ad3b27 [GlobalOpt] Coding style - remove function names from doxygen comments
Suggested by Mehdi in the review of D14148.

llvm-svn: 253029
2015-11-13 11:05:07 +00:00
Akira Hatanaka 5af7ace4ee Revert r252990.
Some of the buildbots are still failing.

llvm-svn: 252999
2015-11-13 01:44:32 +00:00
Akira Hatanaka c7dfb76fe7 Provide a way to specify inliner's attribute compatibility and merging.
This reapplies r252949. I've changed the type of FuncName to be
std::string instead of StringRef in emitFnAttrCompatCheck.

Original commit message for r252949:

Provide a way to specify inliner's attribute compatibility and merging
rules using table-gen. NFC.

This commit adds new classes CompatRule and MergeRule to Attributes.td,
which are used to generate code to check attribute compatibility and
merge attributes of the caller and callee.

rdar://problem/19836465

llvm-svn: 252990
2015-11-13 01:23:11 +00:00