I believe this patch eliminates all remaining uses of _context
or _linkingContext variable names. Consistent naming improves
readability.
llvm-svn: 234645
The Native file format was designed to be the fastest on-memory or
on-disk file format for object files. The problem is that no one
is working on that. No LLVM tools can produce object files in
the Native, thus the feature of supporting the format is useless
in the linker.
This patch removes the Native file support. We can add it back
if we really want it in future.
llvm-svn: 234641
It's not our business to resolve those undefined symbols.
We just trust the linker will load the library and its dependencies
correctly, which is actually what happens, modulo bugs in the dynamic
linker itself.
PR: 23035
Differential Revision: http://reviews.llvm.org/D8886
llvm-svn: 234378
canParse took three parameters -- file magic, filename extension and
memory buffer. All but YAMLReader ignored the second parameter.
This patch removes the parameter.
llvm-svn: 234080
The expression evaluation is needed when interpreting linker scripts, in order
to calculate the value for new symbols or to determine a new position to load
sections in memory. This commit extends Expression nodes from the linker script
AST with evaluation functions, and also contains a unit test.
http://reviews.llvm.org/D8156
llvm-svn: 231707
All readers except PE/COFF reader create layout-after edges to preserve
the original symbol order. PE/COFF uses layout-before edges as primary
edges for no reason.
This patch makes PE/COFF reader to create layout-after edges.
Resolver is updated to recognize reverse edges of layout-after edges
in the garbage collection pass.
Now we can retire layout-before edges. I don't do that in this patch
because if I do, I would have updated many tests to replace all
occurrrences of "layout-before" with "layout-after". So that's a TODO.
llvm-svn: 231615
Atoms with fallback atoms are never be added to the symbol table.
However, we added such atoms to _undefines array. We had to call
isCoalescedAway to identify and skip them. We should just stop
adding them in the first place.
This seems to make the linker ~1% faster in my test case.
llvm-svn: 231552
If an undefined symbol is added to the symbol table by the previous
call of SymbolTable::add, SymbolTable::isDefined will always return
false for the same symbol.
llvm-svn: 231551
This is yet another optimization patch. Previously we called
SymbolTable::isDefined() and SymbolTable::findByName() from a very
frequently executed function. Because isDefined calls findByName,
findByName is called twice on each iteration.
findByName is not a cheap function. It computes a hash value for a
given symbol name. When linking C++ programs, it can be expensive
because of C++ mangled long symbols.
This patch reduces the number of call from 2 to 1. Performance
improvements by this patch was larger than I expected. Linking time
of chrome.dll gets almost 5% shorter.
llvm-svn: 231549
In the resolver, we maintain a list of undefined symbols, and when we
visit an archive file, we check that file if undefined symbols can be
resolved using files in the archive. The archive file class provides
find() function to lookup a symbol.
Previously, we call find() for each undefined symbols. Archive files
may be visited multiple times if they are in a --start-group and
--end-group. If we visit a file M times and if we have N undefined
symbols, find() is called M*N times. I found that that is one of the
most significant bottlenecks in LLD when linking a large executable.
find() is not a very cheap operation because it looks up a hash table
for a given string. And a string, or a symbol name, can be pretty long
if you are dealing with C++ symbols.
We can eliminate the bottleneck.
Calling find() with the same symbol multiple times is a waste. If a
result of looking up a symbol is "not found", it stays "not found"
forever because the symbol simply doesn't exist in the archive.
Thus, we should call find() only for newly-added undefined symbols.
This optimization makes O(M*N) O(N).
In this patch, all undefined symbols are added to a vector. For each
archive/shared library file, we maintain a start position P. All
symbols [0, P) are already searched. [P, end of the vector) are not
searched yet. For each file, we scan the vector only once.
This patch changes the order in which undefined symbols are looked for.
Previously, we iterated over the result of _symbolTable.undefines().
Now we iterate over the new vector. This is a benign change but caused
differences in output if remaining undefines exist. This is why some
tests are updated.
The performance improvement of this patch seems sometimes significant.
Previously, linking chrome.dll on my workstation (Xeon 2.4GHz 8 cores)
took about 70 seconds. Now it takes (only?) 30 seconds!
http://reviews.llvm.org/D8091
llvm-svn: 231434
_reverseRef is a multimap from atoms to atoms. The map contains
reverse edges of "layout-before" and "group" edges for dead-stripping.
The type of the variable was DenseMap<Atom *, DenseSet<Atom *>>.
This patch changes that to std::unordered_multimap<Atom *, Atom *>.
A DenseMap with a value type of DenseSet was not fast. Inserting 900k
items to the map took about 1.6 seconds on my workstation.
unordered_multimap on the other hand took only 0.6 seconds.
Use of the map also got faster -- originally markLive took 1.3 seconds
in the same test case, and it now took 1.0 seconds. In total we shove
off 1.3 seconds out of 27 seconds in that test case.
llvm-svn: 231432
We maintain a map from symbols to archive files for the archive file
pre-loading. That map is created at the beginning of the resolve()
and is never updated. However, the input file list may be updated by
File::beforeLink(). This is a patch to update the map after beforeLink.
llvm-svn: 231395
Merge::mergeByLargestSection is half-baked since it's defined
in terms of section size, there's no way to get the section size
of an atom.
Currently we work around the issue by traversing the layout edges
to both directions and calculate the sum of all atoms reachable.
I wrote that code but I knew it's hacky. It's even not guaranteed
to work. If you add layout edges before the core linking, it
miscalculates a size.
Also it's of course slow. It's basically a linked list traversal.
In this patch I added DefinedAtom::sectionSize so that we can use
that for mergeByLargestSection. I'm not very happy to add a new
field to DefinedAtom base class, but I think it's legitimate since
mergeByLargestSection is defined for section size, and the section
size is currently just missing.
http://reviews.llvm.org/D7966
llvm-svn: 231290
File objects are not really const in the resolver. We set ordinals to
them and call beforeLink hooks. Also, File's member functions marked
as const are not really const. ArchiveFile never returns the same
member file twice, so it remembers files returned before. find() has
side effects.
In order to deal with the inconsistencies, we sprinkled const_casts
and marked member varaibles as mutable.
This patch removes const from there to reflect the reality.
llvm-svn: 231212
std::promise and std::future in old version of libstdc++ are buggy.
I think that's the reason why LLD tests were flaky on Ubuntu 13
buildbots until we disabled file preloading.
In this patch, I implemented very simple future and used that in
FileArchive. Compared to std::promise and std::future, it lacks
many features, but should serve our purpose.
http://reviews.llvm.org/D8025
llvm-svn: 231153
Previously we didn't call the hook on a file in an archive, which
let the PE/COFF port fail to link files in archives. It was a
simple mistake. Added a call to the hook and also added a test to
catch that error.
const_cast is an unfortunate hack. Files in the resolver are usually
const, but they are not actually const objects, since they are
mutated if either a file is taken from an archive (an archive file
does never return the same file twice) or the beforeLink hook is
called. Maybe we should just remove const from there -- because they
are not const.
llvm-svn: 230808
It is observed that the function throws std::future_error on a few buildbots.
That cannot be easily reproducible on local machines. Kill the feature
temporarily to see if this is going to fix the buildbot issue.
llvm-svn: 230735
This fixes a linker crash (found out while testing --gc-sections,
testcase provided by Rafael Avila de Espindola).
While this behaviour was found while testing ELF, it' not necessarily
ELF specific and this change is (apparently) harmless on all the
other drivers.
Differential Revision: D7823
Reviewed by: ruiu
llvm-svn: 230614
Use the environment variable "LLD_RUN_ROUNDTRIP_TEST" in the test that you want
to disable, as
RUN: env LLD_RUN_ROUNDTRIP_TEST= <run>
This was a patch that I made, but I find this a better way to accomplish what we
want to do.
llvm-svn: 228376
This caused some tests to fail on FreeBSD, and Mac OS X.
Some std::sort() implementations will check for strict-weak-ordering
by comparing with the same element, or will compare an element to
itself for 1-element sequence. Take care of this case. Thanks to
chandlerc for explaning that to me.
Reviewed by: ruiu
llvm-svn: 227709
Previously we applied the LayoutPass to order atoms and then
apply elf::ArrayOrderPass to sort them again. The first pass is
basically supposed to sort atoms in the normal fashion (which
is to sort symbols in the same order as the input files).
The second pass sorts atoms in {init,fini}_array.<priority> by
priority.
The problem is that the LayoutPass is overkill. It analyzes
references between atoms to make a decision how to sort them.
It's slow, hard to understand, and above all, it doesn't seem
that we need its feature for ELF in the first place.
This patch remove the LayoutPass from ELF pass list. Now all
reordering is done in elf::OrderPass. That pass sorts atoms by
{init,fini}_array, and if they are not in the special section,
they are ordered as the same order as they appear in the command
line. The new code is far easier to understand, faster, and
still able to create valid executables.
Unlike the previous layout pass, elf::OrderPass doesn't count
any attributes of an atom (e.g. permissions) except its
position. It's OK because the writer takes care of them if we
have to.
This patch changes the order of final output, although that's
benign. Tests are updated.
http://reviews.llvm.org/D7278
llvm-svn: 227666
That kind of reference was used only in ELFFile, and the use of
that reference there didn't seem to make sense. All test still
pass (after adjusting symbol names) without that code. LLD is
still be able to link LLD and Clang. Looks like we just don't
need this.
http://reviews.llvm.org/D7189
llvm-svn: 227259
Before this patch there was a cyclic dependency between lldCore and
lldReaderWriter. Only lldConfig could be built as a shared library.
* Moved Reader and Writer base classes into lldCore.
* The following shared libraries can now be built:
lldCore
lldYAML
lldNative
lldPasses
lldReaderWriter
Differential Revision: http://reviews.llvm.org/D7105
From: Greg Fitzgerald <garious@gmail.com>
llvm-svn: 226732
I believe the original code is valid, but on Windows it failed with an
assertion error saying "Expression: vector iterator is not decrementable."
Don't use rbegin and rend to workaround that error.
llvm-svn: 226706
We used to manage the state whether we are in a group or not
using a counter. The counter is incremented by one if we jump from
end-group to start-group, and decremented by one if we don't.
The counter was assumed to be either zero or one, but obviously it
could be negative (if there's a group which is not repeated at all).
This is a fix for that issue.
llvm-svn: 226632
LLD parses archive file index table only at first. When it finds a symbol
it is looking for is defined in a member file in an archive file, it actually
reads the member from the archive file. That's done in the core linker.
That's a single-thread process since the core linker is single threaded.
If your command line contains a few object files and a lot of archive files
(which is quite often the case), LLD hardly utilizes hardware parallelism.
This patch improves parallelism by speculatively instantiating archive
file members. At the beginning of the core linking, we first create a map
containing all symbols defined in all members, and each time we find a
new undefined symbol, we instantiate a member file containing the
symbol (if such file exists). File instantiation is side effect free, so this
should not affect correctness.
This is a quick benchmark result. Time to link self-link LLD executable:
Linux 9.78s -> 8.50s (0.86x)
Windows 6.18s -> 4.51s (0.73x)
http://reviews.llvm.org/D7015
llvm-svn: 226336
This patch makes File::parse() multi-thread safe. If one thread is running
File::parse(), other threads will block if they try to call the same method.
File::parse() is idempotent, so you can safely call multiple times.
With this change, we don't have to wait for all worker threads to finish
in Driver::link(). Previously, Driver::link() calls TaskGroup::sync() to
wait for all threads running File::parse(). This was not ideal because
we couldn't start the resolver until we parse all files.
This patch increase parallelism by making Driver::link() to not wait for
worker threads. The resolver calls parse() to make sure that the file
being read has been parsed, and then uses the file. In this approach,
the resolver can run with the parser threads in parallel.
http://reviews.llvm.org/D6994
llvm-svn: 226281
InputElement was named that because it's an element of an InputGraph.
It's losing the origin because the InputGraph is now being removed.
InputElement's subclass is FileNode, that naming inconsistency needed
to be fixed.
llvm-svn: 226147
These changes depended on r225674 and had been rolled back in r225859.
Because r225674 has been re-submitted, it's safe to re-submit them.
llvm-svn: 226132
r225764 broke a basic functionality on Mac OS. This change reverts
r225764, r225766, r225767, r225769, r225814, r225816, r225829, and r225832.
llvm-svn: 225859
getNextFile used to have a complex logic to determine which file
should be processed by the Resolver on next iteration.
Now, it is just a sequential accessor to the internal array and
provides no sensible feature.
This patch also removes InputGraph::getGroupSize and InputGraph::
skipGroup to simplify the code.
llvm-svn: 225832
PECOFF was the only user of the API, and the reason why we created
the API is because, although the driver creates a list of input files,
it has no knowledge on what files are being created. It was because
everything was hidden behind the InputGraph abstraction.
Now the driver knows what that's doing. We no longer need this
indirection to get the file list being processed.
llvm-svn: 225767
This is a part of InputGraph cleanup to represent input files as a flat
list of Files (and some meta-nodes for group etc.)
We cannot achieve that goal in one gigantic patch, so I split the task
into small steps as shown below.
(Recap the progress so far: Currently InputGraph contains a list of
InputElements. Each InputElement contain one File (that used to have
multiple Files, but I eliminated that use case in r223867). Files are
currently instantiated in Driver::link(), but I already made a change
to separate file parsing from object instantiation (r224102), so we
can safely instantiate Files when we need them, instead of wrapping
a file with the wrapper class (FileNode class). InputGraph used to
act like a generator class by interpreting groups by itself, but it's
now just a container of a list of InputElements (r223867).)
1. Instantiate Files in the driver and wrap them with WrapperNode.
WrapperNode is a temporary class that allows us to instantiate Files
in the driver while keep using the current InputGraph data structure.
This patch demonstrates how this step 1 looks like, using Core driver
as an example.
2. Do the same thing for the other drivers.
When step 2 is done, an InputGraph consists of GroupEnd objects or
WrapperNodes each of which contains one File. Other types of
FileNode subclasses are removed.
3. Replace InputGraph with std::vector<std::unique_ptr<InputElement>>.
InputGraph is already just a container of list of InputElements,
so this step removes that needless class.
4. Remove WrapperNode.
We need some code cleanup between each step, because many classes
do a bit odd things (e.g. InputGraph::getGroupSize()). I'll straight
things up as I need to.
llvm-svn: 225313
ReaderErrorCategory was used only at one place. We now have a
DynamicErrorCategory for this kind of one-time error, so use it.
The calling function doesn't really care the type of an error, so
ReaderErrorCategory was actually dead code.
llvm-svn: 224245
These member functions returns either no_more_files error or a File object.
We could simply return a nullptr instead of a no_more_files.
This function will be removed soon as a part of InputGraph cleanup.
I had to do that step by step.
llvm-svn: 224208
The documentation of parseFile() said that "the resulting File
object may take ownership of the MemoryBuffer." So, whether or not
the ownership of a MemoryBuffer would be taken was not clear.
A FileNode (a subclass of InputElement, which is being deprecated)
keeps the ownership if a File doesn't take it.
This patch makes File always take the ownership of a buffer.
Buffers lifespan is not always the same as File instances.
Files are able to deallocate buffers after parsing the contents.
llvm-svn: 224113
This reverts commit r223330 because it broke Darwin and ELF
linkers in a way that we couldn't have caught with the existing
test cases.
llvm-svn: 223373
The aim of this patch is to reduce the excessive abstraction from
the InputGraph. We found that even a simple thing, such as sorting
input files (Mach-O) or adding a new file to the input file list
(PE/COFF), is nearly impossible with the InputGraph abstraction,
because it hides too much information behind it. As a result,
we invented complex interactions between components (e.g.
notifyProgress() mechanism) and tricky code to work around that
limitation. There were many occasions that we needed to write
awkward code.
This patch is a first step to make it cleaner. As a first step,
this removes Group class from the InputGraph. The grouping feature
is now directly handled by the Resolver. notifyProgress is removed
since we no longer need that. I could have cleaned it up even more,
but in order to keep the patch minimum, I focused on Group.
SimpleFileNode class, a container of File objects, is now limited
to have only one File. We shold have done this earlier.
We used to allow putting multiple File objects to FileNode.
Although SimpleFileNode usually has only one file, the Driver class
actually used that capability. I modified the Driver class a bit,
so that one FileNode is created for each input File.
We should now probably remove SimpleFileNode and directly store
File objects to the InputGraph in some way, because a container
that can contain only one object is useless. This is a TODO.
Mach-O input files are now sorted before they are passe to the
Resolver. DarwinInputGraph class is no longer needed, so removed.
PECOFF still has hacky code to add a new file to the input file list.
This will be cleaned up in another patch.
llvm-svn: 223330
RoundTripPasses should always be called in DEBUG mode if the environment
variable "LLD_RUN_ROUNDTRIP_TEST" is set.
Flavors should not be able to override this behavior.
llvm-svn: 223073
This would allow other flavor specific contexts to override the default value,
if they want to optionally run the round trip passes.
There is some information lost like the original file owner of the atom with
RoundTripPasses. The Gnu flavor needs this information inorder to implement
LinkerScript matching and for other diagnostic outputs such as Map files.
The flag also can be used to record information in the Atom if the information
to the Writer needs to be conveyed through References too.
llvm-svn: 222983
On darwin in final linked images, the __TEXT segment covers that start of the
file. That means in memory a process can see the mach_header (and load commands)
for every loaded image in a process. There are APIs that take and return the
mach_header addresses as a way to specify a particular loaded image.
For completeness, any code can get the address of the mach_header of the image
it is in by using &__dso_handle. In addition there are mach-o type specific
symbols like __mh_execute_header.
The linker needs to supply a definition for any of these symbols if used. But
the address the symbol it resolves to is not in any section. Instead it is the
address of the start of the __TEXT segment.
I needed to make a small change to SimpleFileNode to not override
resetNextIndex() because the Driver creates a SimpleFileNode to hold the
internal/implicit files that the context/writer can create. For some reason
SimpleFileNode overrode resetNextIndex() to do nothing instead of reseting
the index (which mach-o needs if the internal file is an archive).
llvm-svn: 221822
Darwin uses two-level-namespace lookup for symbols which means the static
linker records where each symbol must be found at runtime. Thus defining a
symbol in a dylib loaded earlier will not effect where symbols needed by
later dylibs will be found. Instead overriding is done through a section
of type S_INTERPOSING which contains tuples of <interposer, interposee>.
llvm-svn: 221421
The job of the CompactUnwind pass is to turn __compact_unwind data (and
__eh_frame) into the compressed final form in __unwind_info. After it's done,
the original atoms are no longer relevant and should be deleted (they cause
problems during actual execution, quite apart from the fact that they're not
needed).
llvm-svn: 221301
The darwin linker operates differently than the gnu linker with respect to
libraries. The darwin linker first links in all object files from the command
line, then to resolve any remaining undefines, it repeatedly iterates over
libraries on the command line until either all undefines are resolved or no
undefines were resolved in the last pass.
When Shankar made the InputGraph model, the plan for darwin was for the darwin
driver to place all libraries in a group at the end of the InputGraph. Thus
making the darwin model a subset of the gnu model. But it turns out that does
not work because the driver cannot tell if a file is an object or library until
it has been loaded, which happens later.
This solution is to subclass InputGraph for darwin and just iterate the graph
the way darwin linker needs.
llvm-svn: 220330
Previously, we would not check the target machine type and the module (object)
machine type. Add a check to ensure that we do not attempt to use an object
file with a different target architecture.
This change identified a couple of tests which were incorrectly mixing up
architecture types, using x86 input for a x64 target. Adjust the tests
appropriately. The renaming of the input and the architectures covers the
changes to the existing tests.
One significant change to the existing tests is that the newly added test input
for x64 uses the correct user label prefix for X64.
llvm-svn: 219093
Rather than a series of cascading ifs, use a switch statement to convert the
error code to a string. This has the benefit of allowing the compiler to inform
us if we ever add a new error code but fail to update the string representation.
Add in stringified versions for a couple of missing InputGraphErrors.
llvm-svn: 219089
The mergeByContent attribute on DefinedAtoms triggers the symbol table to
coalesce atoms with the exact same content. The problem is that atoms can also
have a required custom section. The coalescing should never change the custom
section of an atom.
The fix is to only consider to atoms to have the same content if their
sectionChoice() and customSectionName() attributes match.
llvm-svn: 218893
No functionality change. This removes a down-cast from LinkingContext to
MachOLinkingContext.
Also, remove const from LinkingContext::createImplicitFiles() to remove
the need for another const cast. Seems reasonable for createImplicitFiles()
to need to modify the context (MachOLinkingContext does).
llvm-svn: 218796
The darwin linker has the -demangle option which directs it to demangle C++
(and soon Swift) mangled symbol names. Long term we need some Diagnostics object
for formatting errors and warnings. But for now we have the Core linker just
writing messages to llvm::errs(). So, to enable demangling, I changed the
Resolver to call a LinkingContext method on the symbol name.
To make this more interesting, the demangling code is done via __cxa_demangle()
which is part of the C++ ABI, which is only supported on some platforms, so I
had to conditionalize the code with the config generated HAVE_CXXABI_H.
llvm-svn: 218718
This is a minimally useful pass to construct the __unwind_info section in a
final object from the various __compact_unwind inputs. Currently it doesn't
produce any compressed pages, only works for x86_64 and will fail if any
function ends up without __compact_unwind.
rdar://problem/18208653
llvm-svn: 218703
defined in a shared library.
Now LLD does not export a strong defined symbol if it coalesces away a
weak symbol defined in a shared library. This bug affects all ELF
architectures and leads to segfault:
% cat foo.c
extern int __attribute__((weak)) flag;
int foo() { return flag; }
% cat main.c
int flag = 1;
int foo();
int main() { return foo() == 1 ? 0 : -1; }
% clang -c -fPIC foo.c main.c
% lld -flavor gnu -target x86_64 -shared -o libfoo.so ... foo.o
% lld -flavor gnu -target x86_64 -o a.out ... main.o libfoo.so
% ./a.out
Segmentation fault
The problem is caused by the fact that we lose all information about
coalesced symbols after the `Resolver::resolve()` method is finished.
The patch solves the problem by overriding the
`LinkingContext::notifySymbolTableCoalesce()` method and saving names
of coalesced symbols. Later in the `buildDynamicSymbolTable()` routine
we use this information to export these symbols.
llvm-svn: 217363
This is the one interesting aspect from:
http://reviews.llvm.org/D4965
These hooks are useful for flavor specific processing, such as recording that
a DefinedAtom replaced a weak SharedLibraryAtom.
llvm-svn: 216122
insertElementAt(x, END) does the identical thing as addInputElement(x),
so the only reasonable use of insertElementAt is to call it with the
other possible argument, BEGIN. That means the second parameter of the
function is just redundant. This patch is to remove the second
parameter and rename the function accordingly.
llvm-svn: 213821
COFF supports a feature similar to ELF's section groups. This
patch implements it.
In ELF, section groups are identified by their names, and they are
treated somewhat differently from regular symbols. In COFF, the
feature is realized in a more straightforward way. A section can
have an annotation saying "if Nth section is linked, link this
section too."
I added a new reference type, kindAssociate. If a target atom is
coalesced away, the referring atom is removed by Resolver, so that
they are treated as a group.
Differential Revision: http://reviews.llvm.org/D4028
llvm-svn: 211106
This code was never being used and any use of it would look fairly strange.
For example, it would try to map a NativeReaderError::file_malformed to
std::errc::invalid_argument.
llvm-svn: 210913
Rui Ueyama