This is part of a larger effort to get the Stream code moved
up to Support. I don't want to do it in one large patch, in
part because the changes are so big that it will treat everything
as file deletions and add, losing history in the process.
Aside from that though, it's just a good idea in general to
make small changes.
So this change only changes the names of the Stream related
source files, and applies necessary source fix ups.
llvm-svn: 296211
I added this log message to test the /msvclto option, but
this output might confuse FileCheck. This patch attempts to fix
it by removing it.
llvm-svn: 295793
LLD is a multi-threaded program. errs() or outs() are not guaranteed
to be thread-safe (they are actually not).
LLD's message(), log() or error() are thread-safe. We should use them.
llvm-svn: 295787
Behavior races on ErrorCount. If the enqueued paths are evaluated
eagerly (in enqueuePath) then the behavior is as the test expects. But
they may not be evaluated until the future is waited on, in run() -
which is after the early return/exit on ErrorCount. (this causes the
test to fail (because in the "/ERRORCOUNT:XYZ" test, no other errors
are printed), at least for me, on linux)
This reverts commit r295507.
llvm-svn: 295590
Summary: This adds support for reporting multiple errors in a single invocation of lld-link. The limit defaults to 20 and can be changed with the /ERRORLIMIT command line parameter, or set to unlimited by passing a value of 0.
Reviewers: pcc, ruiu
Reviewed By: ruiu
Differential Revision: https://reviews.llvm.org/D29691
llvm-svn: 295507
Some PDBs or object files can contain references to other PDBs
where the real type information lives. When this happens,
all type indices in the original PDB are meaningless because
their records are not there.
With this patch we add the ability to pull type info from those
secondary PDBs.
Differential Revision: https://reviews.llvm.org/D29973
llvm-svn: 295382
LLVM defines `PTHREAD_LIB` which is used by AddLLVM.cmake and various projects
to correctly link the threading library when needed. Unfortunately
`PTHREAD_LIB` is defined by LLVM's `config-ix.cmake` file which isn't installed
and therefore can't be used when configuring out-of-tree builds. This causes
such builds to fail since `pthread` isn't being correctly linked.
This patch attempts to fix that problem by renaming and exporting
`LLVM_PTHREAD_LIB` as part of`LLVMConfig.cmake`. I renamed `PTHREAD_LIB`
because It seemed likely to cause collisions with downstream users of
`LLVMConfig.cmake`.
llvm-svn: 294690
Summary: This adds an option to save temporary files generated during link-time optimization. This can be useful for debugging.
Reviewers: ruiu, pcc
Reviewed By: ruiu, pcc
Subscribers: mehdi_amini
Differential Revision: https://reviews.llvm.org/D29518
llvm-svn: 294498
If `/debugtypes` is used to omit the codeview information, we would not
have constructed the debug info codeview record which is used to tie the
PDB to the binary. In such a case, rub out the GUID and Age fields.
llvm-svn: 294279
This patch defines a new command line option, /MSVCLTO, to LLD.
If that option is given, LLD invokes link.exe to link LTO-generated
object files. This is hacky but useful because link.exe can create
PDB files.
Differential Revision: https://reviews.llvm.org/D29526
llvm-svn: 294234
Summary: The COFF linker previously implemented link-time optimization using an API which has now been marked as legacy. This change refactors the COFF linker to use the new LTO API, which is also used by the ELF linker.
Reviewers: pcc, ruiu
Reviewed By: pcc
Subscribers: mgorny, mehdi_amini
Differential Revision: https://reviews.llvm.org/D29059
llvm-svn: 293967
Previously, mergeTypeStreams returns only true or false, so it was
impossible to know the reason if it failed. This patch changes the
function signature so that it returns an Error object.
Differential Revision: https://reviews.llvm.org/D29362
llvm-svn: 293820
Previously, we were printing out something like this for
sections/symbols with alignment 16
0000000000201000 0000000000000182 10 .data
which I think confusing. I think printing it in decimal is better.
Differential Revision: https://reviews.llvm.org/D29258
llvm-svn: 293685
Summary: This copies over some functionality we have in ELF/Error.{cpp,h} and makes it available in COFF/Error.{cpp,h}
Reviewers: pcc, rafael, ruiu
Subscribers:
Differential Revision: https://reviews.llvm.org/D28692
llvm-svn: 292240
Previous code had a bug that if the program exits with an assert() or
fail() before the control reaches end of writeMapFile(), it leaves a
temporary file, because FileRemover's dtor isn't called in that case.
I could fix that by removeFileOnSignal() and other functions, but
I think we can simply write to the result file directly. I think
that is straightforward and easy to understand.
Additionally, that allows something like `-Map /dev/null` or a bash
hack such as `-Map >(grep symbol-im-looking-for)`. Previously,
that kind of things didn't work.
Differential Revision: https://reviews.llvm.org/D28714
llvm-svn: 292041
Summary: When we encouter a relocation type we don't know how to handle, this change causes us to print out the hexadecimal value of the relocation type. This makes troubleshooting a little easier.
Reviewers: ruiu, zturner
Differential Revision: https://reviews.llvm.org/D28576
llvm-svn: 291962
This patch is to merge type info in multiple .debug$T sections.
One mystery that needs to be solved is that it is not clear how
the MSVC linker uses TPI and IPI streams. Both streams contain
type info, and it is not obvious what kind of record should go
which.
dumppdb command in microsoft-pdb repository prints out IPI stream
contents as "IDs" and TPI stream as "TYPES", but looks like the tool
don't really care about which stream type recrods were read from.
For now, in this patch, I emit all type records to TPI stream.
It might just work with other tools. If not, we need to investigate
it more.
llvm-svn: 291739
Previously the type dumper itself was passed around to a lot of different
places and manipulated in ways that were more appropriate on the type
database. For example, the entire TypeDumper was passed into the symbol
dumper, when all the symbol dumper wanted to do was lookup the name of a
TypeIndex so it could print it. That's what the TypeDatabase is for --
mapping type indices to names.
Another example is how if the user runs llvm-pdbdump with the option to
dump symbols but not types, we still have to visit all types so that we
can print minimal information about the type of a symbol, but just without
dumping full symbol records. The way we did this before is by hacking it
up so that we run everything through the type dumper with a null printer,
so that the output goes to /dev/null. But really, we don't need to dump
anything, all we want to do is build the type database. Since
TypeDatabaseVisitor now exists independently of TypeDumper, we can do
this. We just build a custom visitor callback pipeline that includes a
database visitor but not a dumper.
All the hackery around printers etc goes away. After this patch, we could
probably even delete the entire CVTypeDumper class since really all it is
at this point is a thin wrapper that hides the details of how to build a
useful visitation pipeline. It's not a priority though, so CVTypeDumper
remains for now.
After this patch we will be able to easily plug in a different style of
type dumper by only implementing the proper visitation methods to dump
one-line output and then sticking it on the pipeline.
Differential Revision: https://reviews.llvm.org/D28524
llvm-svn: 291724
We were starting to get some name clashes between llvm-pdbdump
and the common CodeView framework, so I took this opportunity
to rename a bunch of files to more accurately describe their
usage. This also helps in llvm-pdbdump to distinguish
between different files and whether they are used for pretty
dump mode or raw dump mode.
llvm-svn: 291627
This is how we use TarWriter in LLD. Now LLD does not append
a file extension, so you need to pass `--reproduce foo.tar`
instead of `--reproduce foo`.
Differential Revision: https://reviews.llvm.org/D28103
llvm-svn: 291210
The PDB GUID, Age, and version are tied together by the RSDS record in
the binary. Pass along the BuildId information into the createPDB to
allow us to tie the binary and the PDB together.
llvm-svn: 290975
Assert that the size of the MD5 result is the same size as the signature
field being populated. Use the sizeof operator to determine the size of
the field being written rather than hardcoding it to the magic number
16. NFC.
llvm-svn: 290764
I thought for a while about how to remove it, but it looks like we
can just copy the file for now. Of course I'm not happy about that,
but it's just less than 50 lines of code, and we already have
duplicate code in Error.h and some other places. I want to solve
them all at once later.
Differential Revision: https://reviews.llvm.org/D27819
llvm-svn: 290062
File system operations were still dominating the profile on Windows. In this
case we were spending a significant amount of our time repeatedly searching
for libraries as a result of processing linker directives. Address this
by caching whether we have already found a library with a given name. For
chrome_child.dll:
Before: 10.53s
After: 6.88s
Differential Revision: https://reviews.llvm.org/D27840
llvm-svn: 289915
Profiling revealed that the majority of lld's execution time on Windows was
spent opening and mapping input files. We can reduce this cost significantly
by performing these operations asynchronously.
This change introduces a queue for all operations on input file data. When
we discover that we need to load a file (for example, when we find a lazy
archive for an undefined symbol, or when we read a linker directive to
load a file from disk), the file operation is launched using a future and
the symbol resolution operation is enqueued. This implies another change
to symbol resolution semantics, but it seems to be harmless ("ninja All"
in Chromium still succeeds).
To measure the perf impact of this change I linked Chromium's chrome_child.dll
with both thin and fat archives.
Thin archives:
Before (median of 5 runs): 19.50s
After: 10.93s
Fat archives:
Before: 12.00s
After: 9.90s
On Linux I found that doing this asynchronously had a negative effect on
performance, probably because the cost of mapping a file is small enough that
it becomes outweighed by the cost of managing the futures. So on non-Windows
platforms I use the deferred execution strategy.
Differential Revision: https://reviews.llvm.org/D27768
llvm-svn: 289760
Enable building lld as a standalone project. This is motivated by the desire to
package lld for inclusion in a linux distribution. This allows building lld
against an existing paired llvm installation. Now that lld is usable on x86_64,
it makes sense to revive this configuration to allow distributions to package
it.
llvm-svn: 289421
This patch replaces the symbol table's object and archive queues, as well as
the convergent loop in the linker driver, with a design more similar to the
ELF linker where symbol resolution directly causes input files to be added to
the link, including input files arising from linker directives. Effectively
this removes the last vestiges of the old parallel input file loader.
Differential Revision: https://reviews.llvm.org/D27660
llvm-svn: 289409
Using a set here caused us to take about 1 second longer to write the symbol
table when linking chrome_child.dll. With this I consistently get better
performance on Windows with the new symbol table.
Before r289280 and with r289183 reverted (median of 5 runs): 17.65s
After this change: 17.33s
On Linux things look even better:
Before: 10.700480444s
After: 5.735681610s
Differential Revision: https://reviews.llvm.org/D27648
llvm-svn: 289408
This ports the ELF linker's symbol table design, introduced in r268178,
to the COFF linker.
Differential Revision: http://reviews.llvm.org/D21166
llvm-svn: 289280
The former option bases the filename on the output name, e.g. if the
link output is a.exe, the map will be written to a.map. This matches the
behaviour of link.exe's /MAP option and is useful for creating a map
file of each executable when building a large project.
Differential Revision: https://reviews.llvm.org/D27595
llvm-svn: 289271
Profiling revealed that we were spending 5% of our time linking
chrome_child.dll just in this call to toString().
Differential Revision: https://reviews.llvm.org/D27628
llvm-svn: 289270
I don't think the data I add to a TPI stream in this patch is correct,
but at least it can be displayed using llvm-pdbdump. Until I add more
streams to a PDB file, I'm not able to know whether the data will be
accepted by MSVC tools or not.
llvm-svn: 289183
Previously, we had different way to stringize SymbolBody and InputFile
to construct error messages. This patch defines overloaded function
toString() so that we don't need to memorize all these different
function names.
With that change, it is now easy to include demangled names in error
messages. Now, if there is a symbol name conflict, we'll print out
both mangled and demangled names.
llvm-svn: 288992
LLD used to take 11.73 seconds to link Clang. Now it is 6.94 seconds.
MSVC link takes 83.02 seconds. Note that ICF is enabled by default on
Windows, so a low latency ICF is more important than in ELF.
llvm-svn: 288487
Associative sections are sections that need to be linked if their associated
sections are linked. Associative sections are used to append auxiliary data
such as debug info.
Previously, we compared all associative sections when comparing two comdat
sections. Because usually assocative sections are not mergeable sections,
we missed a lot of mergeable sections. MSVC linker doesn't seem to check
the identity of associative sections.
This patch makes LLD to ignore associative sections when doing ICF.
llvm-svn: 288483
rL287555 introduces a link error when building with BUILD_SHARED_LIBS:
undefined reference to llvm::codeview::CVSymbolDumper::dump(),
and more...
The functions are available in libDebugInfoCodeView, from LLVM.
Patch by Visoiu Mistrih Francis!
llvm-svn: 287837
Previously, we discarded .debug$ sections. This patch adds them to
files so that PDB.cpp can access them.
This patch also adds a debug option, /dumppdb, to dump debug info
fed to createPDB so that we can verify that valid data has been passed.
llvm-svn: 287555
This patch updates a couple places where add_dependencies was being explicitly called to add dependencies on intrinsics_gen to instead use the DEPENDS named parameter. This cleanup is needed for a patch I'm working on to add a dependency debugging mode to the build system.
llvm-svn: 287205
createManifestRes was generating a MemoryBuffer from a TemporaryFile,
keeping the data but removing the file, before passing the file path
to CVTRES.exe, leading to the following error:
CVTRES : fatal error CVT1101: cannot open 'C:\Users\user\AppData\
Local\Temp\lld-output-resource-bfee19.res' for reading
With this, we instead create a new TemporaryFile before passing it to cvtres.
Patch from Rudy Pons!
llvm-svn: 287034
Object files compiled with cl.exe /GL contain intermediate code for LTO.
We can't (and don't want to) interpret such code, but we should print
out a user-friendly error message.
Differential Revision: https://reviews.llvm.org/D26647
llvm-svn: 286921
The MSVC linker relies on this invariant to produce a valid import
table. More ASan tests pass in a stage 2 build now. They still fail when
using LLD since there are no PDBs for the dynamic ASan runtime.
llvm-svn: 286499
Following the lazy reference might bring in an object file that depends
on bitcode files that weren't part of the LTO step.
Differential Revision: https://reviews.llvm.org/D25461
llvm-svn: 283989
With this, "llvm-pdbdump yaml -ipi-stream" prints out an IPI stream.
Previously it crashed because it can't handle the case where IPI
stream doesn't exist.
llvm-svn: 283392
I do not fully understand how to use these classes yet, but
seems like these arguments are not used, since without them
all tests still pass. In order to simplify the situation,
I'll remove them now.
llvm-svn: 283174
Handle this in the exact same way as IMAGE_REL_AMD64_SECREL
and IMAGE_REL_I386_SECREL.
Differential revision: https://reviews.llvm.org/D24608
llvm-svn: 282531
So that it is clear that FileOutputBuffer does not depend on
PDB file builder. Eventually we will have to to get the file size
info from the file builder to create a file with the exact size.
NFC.
llvm-svn: 282454
Change the way we calculate the build id to use MD5 to give reproducible build
ids. Previously we would generate random bytes for the build id GUID.
llvm-svn: 281079
Previously, we created temporary files using llvm::sys::fs::createTemporaryFile
and removed them using llvm::FileRemover. This is error-prone as it is easy to
forget creating FileRemover instances after creating temporary files.
There is actually a temporary file leak bug.
This patch introduces a new class, TemporaryFile, to manage temporary files
in the RAII style.
Differential Revision: https://reviews.llvm.org/D24176
llvm-svn: 280510
Summary:
UBSan complains like the following:
tools/lld/COFF/Writer.cpp:97:15: runtime error: null pointer passed as argument 2, which is declared to never be null
The reason is that the vector could be empty.
Reviewers: rsmith
Subscribers: Eugene.Zelenko, kcc
Differential Revision: https://reviews.llvm.org/D24050
llvm-svn: 280259
The IMAGE_FILE_HEADER structure contains a (RVA, size) to an array of
COFF_DEBUG_DIRECTORY records. Each one of these records contains an RVA to a OMF
Debug Directory. These OMF debug directories are derived into newer types such
as PDB70, PDB20, etc. This constructs a PDB70 structure which will allow us to
associate a GUID with a build to actually tie debug information.
llvm-svn: 280012
Reorder the table setup to mirror the indices corresponding to them. This means
that the table values are filled out as per the enumeration ordering. Doing so
makes it easier to identify a particular table. NFC.
llvm-svn: 278199
Add the support infrastructure for the /debugtype option which takes a comma
delimited list of debug info to generate. The defaults are based on other
options potentially (/driver or /profile). This sets up the infrastructure to
allow us to emit RSDS records to get "build id" equivalents on COFF (similar to
binutils).
llvm-svn: 278056
Don't blindly OR in the new value, but clear the existing one, since it can be
nonzero. Read out the existing value before, and add into the desired offset.
(The add is done outside of the applyMOV, to handle potential overflow between
the two.)
Patch by Martin Storsjö!
llvm-svn: 277846
The opcode for the bl branches can initially be F000 F800, i.e.
the J1 and J2 bits are already set. Therefore mask these bits out
before or'ing in the new bits.
Patch by Martin Storsjö!
llvm-svn: 277836
This flag is implemented similarly to --reproduce in the ELF linker.
This patch implements /linkrepro by moving the cpio writer and associated
utility functions to lldCore, and using that implementation in both linkers.
One COFF-specific detail is that we store the object file from which the
resource files were created in our reproducer, rather than the resource
files themselves. This allows the reproducer to be used on non-Windows
systems for example.
Differential Revision: https://reviews.llvm.org/D22418
llvm-svn: 276719
lld currently relies on lib.exe in order to generate an empty import library.
The "empty" import library consists of 5 members:
- first linker member
- second linker member
- Import Descriptor
- NULL Import Descriptor
- NULl Thunk
The first two entries (first and second linker members) are string tables which
are never updated. Therefore, they may as well as not be present. A subsequent
change to add that is probably warranted. However, this does not prevent the
use of the linker.
The Import Descriptor is the content which is most important. It provides an
Import Name Table entry for the library (as specified by the LIBRARY directive
in the DEF file). Additionally, it contains undefined references to the NULL
Import Descriptor and the library NULL Thunk Data. This ensures that the linker
will pull in the subsequent objects from the import library for the link. The
Import Descriptor has a single symbol (__IMPORT_DESCRIPTOR_<Library>) which
contains 3 relocations, one to the INT (Import Name Table) entry, one to the ILT
(Import Lookup Table) entry, and one to the IAT (Import Address Table) entry.
The NULL Import Descriptor is the last import descriptor and terminates the
import descriptor array. It contains a single symbol
(__NULL_IMPORT_DESCRIPTOR).
The NULL Thunk contains a single symbol (\x7f<Library>_NULL_THUNK_DATA) and
provides the terminator for the ILT and IAT.
These files are currently constructed manually following the example of the
Short Import Library format. This is arguably less than ideal, and it may be
possible to use MCAssembler and feed it the fragments to construct the object.
The major difference between the LIB (LINK) generated objects and the ones
generated here is that they are all one section shorter (.debug$S) as they do
not contain the debug information and one symbol shorter (@comp.id) as they do
not contain the RICH signature.
Move the logic related to the librarian into a new source file (Librarian.cpp).
llvm-svn: 275242
Manifest file is a separate or embedded XML file having metadata
of an executable. As it is XML, it can contain various types of
information. Probably the most popular one is to request escalated
priviledges.
Usually the linker creates an XML file and embed that file into
an executable. However, there's a way to supply an XML file from
command line. /manifestniput is it.
Apparently it is over-designed here, but if you supply two or more
manifest files, then the linker needs to merge the files into a
single XML file. A good news is that we don't need to do that ourselves.
MT.exe command can do that, so we call the command from the linker
in this patch.
llvm-svn: 266704
With the llvm change in r265606 this is the matching needed change to the lld
code now that createBinary() is returning Expected<...> .
llvm-svn: 265607
This flag disables link.exe's crash handler so that normal windows error
reporting and crash dumping occurs. For now it is reasonable for LLD to
ignore the flag.
Chromium is currently using this flag to collect minidumps of link.exe
crashing, and it breaks the LLD build.
llvm-svn: 264439
Some declarations of memcpy (like glibc's for example) are attributed
with notnull which makes it UB for NULL to get passed in, even if the
memcpy count is zero.
To account for this, guard the memcpy with an appropriate precondition.
This should fix the last UBSan bug, exposed by the test suite, in the
COFF linker.
llvm-svn: 263919
LLD type-punned an integral type and a pointer type using a pointer
field. This is problematic because the pointer type has alignment
greater than some of the integral values.
This would be less problematic if a union was used but it turns out the
integral values are only present for a short, transient, amount of time.
Let's remove this undefined behavior by skipping the punning altogether
by storing the state in a separate memory location: a vector which
informs us which symbols to process for weak externs.
llvm-svn: 263918
This fixes a test which exposed an ASan issue.
We assumed that a symbol's section number had a corresponding section
without performing validation.
llvm-svn: 263558
The load configuration directory is a structure whose size varies as the
OS gains additional functionality. To account for this, the structure's
layout begins with a size field; this allows loaders to know which
fields are available.
However, LLD hard-coded the sizes (112 bytes for 64-bit and 64 for
32-bit). This means that we might not inform the loader of all the
pertinent fields or we might claim that there are more fields than are
actually present.
To correctly account for this, the size field must be loaded from the
_load_config_used symbol.
N.B. The COFF spec is either wrong or out of date, the load
configuration directory is not correctly documented in the
specification: it omits the size field.
llvm-svn: 263543
The TLS directory has a different layout depending on the bitness of the
machine the image will run on. LLD would always use the 64-bit TLS
directory for the data directory entry instead of an appropriately sized
TLS directory.
llvm-svn: 263539
Now that DarwinLdDriver is the only derived class of Driver.
This patch merges them and actually removed the class because
they can now just be non-member functions. This change simplifies
a common header, Driver.h.
http://reviews.llvm.org/D17788
llvm-svn: 262502
DLL export tables usually contain dllexport'ed symbol RVAs so that
applications which use the DLLs can find symbols from the DLLs.
However, there's a minor feature to "forward" DLL symbols to other
DLLs.
If you set an RVA to a string whose form is "<dllname>.<symbolname>"
(e.g. "KERNEL32.ExitProcess") instead of symbol RVA to the export
table, the loader interprets that as a forwarder symbol, and resolve
that symbol from the specified DLL.
This patch implements that feature.
llvm-svn: 257243
In a UI such as XCode, LLVM source files are in 'libraries' while clang
files are in 'clang libraries'.
This change moves the lld source to 'lld libraries' to make code browsing easier.
It should be NFC as the build itself is still the same, just the structure in a
UI differs.
llvm-svn: 257001
MSVC linker considers PDB files created with this patch valid.
So you don't have to remove PDB files created by lld before
running MSVC linker.
This patch has no test since llvm-pdbdump dislikes PDB files
with no metadata streams.
llvm-svn: 255039
Before this patch, we created an empty PDB file if /debug option is
specified. For MSVC linker, such PDB file is completely broken, and
linker exits without doing anything as soon as it finds an empty PDB
file.
A PDB file created in this patch has the correct file signature.
MSVC linker still thinks that the file is broken, but it then removes
and replaces with its output.
This is an initial patch to support PDB in LLD. We aim to support
PDB in order to make it 100% compatible with MSVC linker. PDB support
is the last missing piece.
llvm-svn: 254796
If a section symbol is not external, that COMDAT section should never
be merge with other sections in other compilation unit. Previously,
we didn't take visibility into account.
Note that COMDAT sections with non-external visibility makes sense
because they can be removed by dead-stripping.
Fixes https://llvm.org/bugs/show_bug.cgi?id=25686
llvm-svn: 254578
There was a threading issue in the ICF code for COFF. That seems like
a venign bug in the sense that it doesn't produce an incorrect output,
but it oftentimes misses reducible sections. As a result, mergeable
sections could remain in outputs, which makes the output nondeterministic.
Basically the algorithm we are using for ICF is this: We group sections
so that identical sections will eventually be in the same group. Initially,
all sections are in one group. We split the group by relocation targets
until we get a convergence (if relocation targets are in different gruops,
the sections are different). Once a group is split, they will never be
merged.
Each section has a group ID. That variable itself is atomic, so there's
no threading issue at the level that we can use thread sanitizer.
The point is, when we split a group, we re-assign new group IDs to group
of sections. That are multiple separate writes to atomic varaibles.
Thus, splitting a group is not an atomic operation, and there's a small
chance that the other thread observes inconsistent group IDs.
Over-splitting is always "safe", so it will never create incorrect output.
I suspect that the nondeterminism stems from that point. However, I
cannot prove or fix that at this moment, so I'm going to avoid using
threads here.
llvm-svn: 251300
There's actually a room to improve this patch. Instead of not merging
sections that have different alignements, we can choose the section that
has the largest alignment requirement among all sections that are otherwise
considered the same. Then all section alignments are satisfied, so we can
merge them.
I don't know if that improvement could make any difference for real-world
input, so I'll leave it alone. Would be interesting to revisit later.
llvm-svn: 248581
This is an LLD extension to MSVC link.exe command line. MSVC linker
does not write symbol tables for executables. We do unless no /debug
option is given.
There's a situation that we want to enable debug info but don't want
to emit the symbol table. One example is when we are comparing output
file size. With this patch, you can tell the linker to not create
a symbol table by just specifying /nosymtab.
llvm-svn: 248225
std::distance(C->Relocs.end(), C->Relocs.begin()) is the same as NumRelocs
which is already added to the hash value. What we are missing here is the
section size.
llvm-svn: 248202
This patch fixes a regression introduced by r247964. Relocations that
are referring the same symbol should be considered equal, but they
were not if they were pointing to non-section chunks.
llvm-svn: 248132
Previously, InputFile::parse() was run in batch. We construct a list
of all input files and call parse() on each file using parallel_for_each.
That means we cannot start parsing files until we get a complete list
of input files, although InputFile::parse() is safe to call from anywhere.
This patch makes it asynchronous. As soon as we add a file to the symbol
table, we now start parsing the file using std::async().
This change shortens self-hosting time (650 ms) by 28 ms. It's about 4%
improvement.
llvm-svn: 248109
I made the field an atomic pointer in hope that we would be able to
parallelize the symbol resolver soon, but that's not going to happen
soon. This patch reverts that change for the sake of readability.
llvm-svn: 248104
InputFile::parse() can be called in parallel with other calls of
the same function. By doing that, time to self-link improves from
741 ms to 654 ms or 12% faster.
This is probably the last low hanging fruit in terms of parallelism.
Input file parsing and symbol table insertion takes 450 ms in total.
If we want to optimize further, we probably have to parallelize
symbol table insertion using concurrent hashmap or something.
That's doable, but that's not easy, especially if you want to keep
the exact same semantics and linking order. I'm not going to do that
at least soon.
Anyway, compared to r248019 (the change before the first attempt for
parallelism), we achieved 36% performance improvement from 1022 ms
to 654 ms. MSVC linker takes 3.3 seconds to link the same program.
MSVC's ICF feature is very slow for some reason, but even if we
disable the feature, it still takes about 1.2 seconds.
Our number is probably good enough.
llvm-svn: 248078
Self-hosting took 801 ms on my machine. Of which this function took
69 ms. Now it takes 37 ms. That is about 4% overall performance
improvement.
llvm-svn: 248052
The LLD's ICF algorithm is highly parallelizable. This patch does that
using parallel_for_each.
ICF accounted for about one third of total execution time. Previously,
it took 324 ms when self-hosting. Now it takes only 62 ms.
Of course your mileage may vary. My machine is a beefy 24-core Xeon machine,
so you may not see this much speedup. But this optimization should be
effective even for 2-core machine, since I saw speedup (324 ms -> 189 ms)
when setting parallelism parameter to 2.
llvm-svn: 248038
Previously, ICF created a vector for each SectionChunk. The vector
contained pointers to successors, which are namely associative sections
and COMDAT relocation targets. The reason I created vectors is because
I thought that that would make section comparison faster.
It did make the comparison faster. When self-linking, for example, it
saved about 10 ms on each iteration. The time we spent on constructing
the vectors was 124 ms. If we iterate more than 12 times, return from
the investment exceeds the initial cost.
In reality, it usually needs 5 iterations. So we shouldn't construct
the vectors.
llvm-svn: 247963
equalsConstants() is the heaviest function in ICF, and that consumes
more than half of total ICF execution time. Of which, section content
comparison accounts for roughly one third.
Previously, we compared section contents at the beginning of the
function after comparing their checksums. The comparison is very
likely to succeed because when the control reaches that comparison,
their checksums are always equal. And because checksums are 64-bit
CRC, they are unlikely to collide.
We compared relocations and associative sections after that.
If they are different, the time we spent on byte-by-byte comparison
of section contents were wasted.
This patch moves the comparison at the end of function. If the
comparison fails, the time we spent on relocation comparison are
wasted, but as I wrote it's very unlikely to happen.
LLD took 1198 ms to link itself to produce a 27.11 MB executable.
Of which, ICF accounted for 536 ms. This patch cuts it by 90 ms,
which is 17% speedup of ICF and 7.5% speedup overall. All numbers
are median of ten runs.
llvm-svn: 247961
Basically the concept of "liveness" is for sections (or chunks in LLD
terminology) and not for symbols. Symbols are always available or live,
or otherwise it indicates a link failure.
Previously, we had isLive() and markLive() methods for DefinedSymbol.
They are confusing methods. What they actually did is to act as a proxy
to backing section chunks. We can simplify eliminate these methods
and call section chunk's methods directly.
llvm-svn: 247869
Only live symbols are written to the symbol table. Because isLive()
returned false if dead-stripping was disabled entirely, only
non-COMDAT sections were written to the symbol table. This patch fixes
the issue.
llvm-svn: 247856
This patch defines ICF class and defines ICF-related functions as
members of the class. By doing this we can move code that are
related only to ICF from SectionChunk to the newly-defined class.
This also eliminates a global variable "NextID".
llvm-svn: 247802
This is a patch to make LLD to be on par with MSVC in terms of ICF
effectiveness. MSVC produces a 27.14MB executable when linking LLD.
LLD previously produced a 27.61MB when self-linking. Now the size
is reduced to 27.11MB. Note that without ICF the size is 29.63MB.
In r247387, I implemented an algorithm that handles section graphs
as cyclic graphs and merge them using SCC. The algorithm did not
always work as intended as I demonstrated in r247721. The new
algortihm implemented in this patch is different from the previous
one. If you are interested the details, you want to read the file
comment of ICF.cpp.
llvm-svn: 247770
Previously, LLD's ICF couldn't merge cyclic graphs. That was unfortunate
because, in COFF, cyclic graphs are not exceptional at all. That is
pretty common.
In this patch, sections are grouped by Tarjan's strongly connected
component algorithm to get acyclic graphs. And then we try to merge
SCCs whose outdegree is zero, and remove them from the graph. This
makes other SCCs to have outdegree zero, so we can repeat the
process until all SCCs are removed. When comparing two SCCs, we handle
cycles properly.
This algorithm works better than previous one. Previously, self-linking
produced a 29.0MB executable. It now produces a 27.7MB. There's still some
gap compared to MSVC linker which produces a 27.1MB executable for the
same input. So the gap is narrowed, but still LLD is not on par with MSVC.
I'll investigate that later.
llvm-svn: 247387
Identical COMDAT Folding is a feature to merge COMDAT sections
by contents. Two sections are considered the same if their contents,
relocations, attributes, etc, are all the same.
An interesting fact is that MSVC linker takes "iterations" parameter
for ICF because the algorithm they are using is iterative. Merging
two sections could make more sections to be mergeable because
different relocations could now point to the same section. ICF is
repeated until we get a convergence (until no section can be merged).
This algorithm is not fast. Usually it needs three iterations until a
convergence is obtained.
In the new algorithm implemented in this patch, we consider sections
and relocations as a directed acyclic graph, and we try to merge
sections whose outdegree is zero. Sections with outdegree zero are then
removed from the graph, which makes other sections to have outdegree
zero. We repeat that until all sections are processed. In this
algorithm, we don't iterate over the same sections many times.
There's an apparent issue in the algorithm -- the section graph is
not guaranteed to be acyclic. It's actually pretty often cyclic.
So this algorithm cannot eliminate all possible duplicates.
That's OK for now because the previous algorithm was not able to
eliminate cycles too. I'll address the issue in a follow-up patch.
llvm-svn: 246878
Previously, we calculated our own hash values for section contents.
Of coruse that's slow because we had to access all bytes in sections.
Fortunately, COFF objects usually contain hash values for COMDAT
sections. We can use that to speed up Identical COMDAT Folding.
llvm-svn: 246869
The option is added in MSVC 2015, and there's no documentation about
what the option is. This patch is to ignore the option for now, so that
at least LLD is usable with MSVC 2015.
llvm-svn: 246780
This patch fixes a subtle incompatibility with MSVC linker.
MSVC linker preserves the original spelling of a DLL in the
import descriptor table. LLD previously converted all
characters to lowercase. Usually this difference is benign,
but if a program explicitly checks for DLL file names, the
program could fail.
llvm-svn: 246620
In r246424, I made a change that disables non-DLL to export
symbols. It turned out that the change was not correct. Both
DLLs and executables are able to export symbols (although the
latter is relatively rare). This change restores the feature.
llvm-svn: 246537
I have totally no idea why, but MSVC linker is sensitive about
file names of archive members. If we do not make import library
file names to the same as the DLL name, MSVC link *crashes*
when it is processing the library file. This patch is to set
the same name.
llvm-svn: 246535
The rules for dllexported symbols are overly complicated due to
x86 name decoration, fuzzy symbol resolution, and the fact that
one symbol can be resolved by so many different names. The rules
are probably intended to be "intuitive", so that users don't have
to understand the name mangling schemes, but it seems that it can
lead to unintended symbol exports.
To make it clear what I'm trying to do with this patch, let me
write how the export rules are subtle and complicated.
- x86 name decoration: If machine type is i386 and export name
is given by a command line option, like /export:foo, the
real symbol name the linker has to search for is _foo because
all symbols are decorated with "_" prefixes. This doesn't happen
on non-x86 machines. This automatic name decoration happens only
when the name is not C++ mangled.
However, the symbol name exported from DLLs are ones without "_"
on all platforms.
Moreover, if the option is given via .drectve section, no
symbol decoration is done (the reason being that the .drectve
section is created by a compiler and the compiler should always
know the exact name of the symbol, I guess).
- Fuzzy symbol resolution: In addition to x86 name decoration,
the linker has to look for cdecl or C++ mangled symbols
for a given /export. For example, it searches for not only
_foo but also _foo@<number> or ??foo@... for /export:foo.
Previous implementation didn't get it right. I'm trying to make
it as compatible with MSVC linker as possible with this patch
however the rules are. The new code looks a bit messy to me, but
I don't think it can be simpler due to the ad-hoc-ness of the rules.
llvm-svn: 246424
This is exposed via a new flag /opt:lldltojobs=N, where N is the number of
code generation threads.
Differential Revision: http://reviews.llvm.org/D12309
llvm-svn: 246342
lib.exe has a feature to create import library files (which contain
short import files) from module-definition files. Previously, we were
using that feature, but it turned out that the feature is not complete
for us.
There seems no way to specify "Import Types" in module-definition file.
lib.exe always adds "_" to given symbols and specify IMPORT_NAME_UNDECORATE.
We need more fine-grainded control on that value.
This patch teaches LLD to create short import files itself.
We are still using lib.exe, but the use of the tool is limited to create
empty import library files. We then create short import files and add them
to the empty files as new members.
This patch does not intend to change the functionality. LLD produces
the same import libraries as before. I'll make another change to create
different import libraries in a follow-up patch.
llvm-svn: 246292
__NULL_IMPORT_DESCRIPTOR is a symbol used by MSVC liner to construct
the import descriptor table. We do not use the symbol. Previously,
we had code to skip that symbol. That code does not actually do
anything meaningful because no one is referencing the symbol, the
symbol would naturally be ignored. This patch stops recognizing
the symbol.
llvm-svn: 245280
Previously, weak external symbols could reference only symbols that
appeared before them. Although that covers almost all use cases
of weak externals, there are object files out there which contains
weak externals that have forward references.
This patch supports such weak externals.
llvm-svn: 245258
There are some DLLs whose initializers depends on other DLLs'
initializers. The initialization order matters for them.
MSVC linker uses the order of the libraries from the command line.
LLD used ASCII-betical order. So they were incompatible.
This patch makes LLD compatible with MSVC.
llvm-svn: 245201
This is more convenient than the offset from the start of the file as we
don't have to worry about it changing when we move the output section.
This is a port of r245008 from ELF.
llvm-svn: 245018
Sections must start at page boundaries in memory, but they
can be aligned to sector boundaries (512-bytes) on disk.
We aligned them to 4096-byte boundaries even on disk, so we
wasted disk space a bit.
llvm-svn: 244691
MSVC 2015's load configuration object (__load_config_used) contains
references to these symbols. I don't fully understand how it works,
but looks like these symbols are linker-defined ones. So I define them
here in the Driver. With this patch, LLD can self-host with MSVC 2015.
This patch is to link MSVC 2015-produced object files. It does not
implement Control Flow Protection. If I understand correctly, the
linker has to create a bitmap of function entry point addresses for
the CFG runtime. We don't do that yet. Produced executables will not
be protected by CFG.
llvm-svn: 244425
SymbolTable::find(mangle(X)) is equivalent to SymbolTable::findUnderscore(X)
except that the latter is slightly efficient as that doesn't allocate a new
string.
llvm-svn: 244377
This has a few advantages
* Less C++ code (about 300 lines less).
* Less machine code (about 14 KB of text on a linux x86_64 build).
* It is more debugger friendly. Just set a breakpoint on the exit function and
you get the complete lld stack trace of when the error was found.
* It is a more robust API. The errors are handled early and we don't get a
std::error_code hot potato being passed around.
* In most cases the error function in a better position to print diagnostics
(it has more context).
llvm-svn: 244215
Various parameters are passed implicitly using Config global variable
already. Output file path is no different from others, so there was no
special reason to handle that differnetly.
This patch changes the signature of writeResult(SymbolTable *, StringRef)
to writeResult(SymbolTable *).
llvm-svn: 244180
We were printing an error but exiting with 0.
Not sure how to test this. We could add a no-winlib feature,
but that is probably not worth it.
llvm-svn: 244109
I don't remember why I thought that only functions are subject
of garbage collection, but the comment here said so, which is
not correct. Moreover, the code just below the comment does not
do what the comment says -- it handles non-COMDAT, non-function
sections as GC root. As a result, it just handles non-COMDAT
sections as GC root.
This patch cleans that up by removing SectionChunk::isRoot and
use isCOMDAT instead.
llvm-svn: 243700
We want to convince the NT loader not to map these sections into memory.
A good first step is to move them to the end of the executable.
Differential Revision: http://reviews.llvm.org/D11655
llvm-svn: 243680
We create a module-definition file and give that to lib.exe to
create an import library file. A module-definition has to be
syntactically and semantically correct, of course.
There was a case that we created a module-definition file that
lib.exe would complain for duplicate entries. If a user gives
an unmangled and mangled name for the same symbol, we would end
up having two duplicate lines for the mangled name in a module-
definition file.
This patch fixes that issue by uniquefying entries by mangled
symbol name.
llvm-svn: 243587
Windows ARM is the thumb ARM environment, and pointers to thumb code
needs to have its LSB set. When we apply relocations, we need to
adjust the LSB if it points to an executable section.
llvm-svn: 243560
SECREL should sets the 32-bit offset of the target from the beginning
of *target's* output section. Previously, the offset from the beginning
of source's output section was used instead.
SECTION means the target section's index, and not the source section's
index. This patch fixes that issue too.
llvm-svn: 243535
I don't fully understand the rationale behind the name mangling
scheme used for the DLL export table and the import library.
Why only leading "_" is dropped for the import library while
both "_" and "@" are dropped from DLL symbol table? But this seems
to be what MSVC linker does.
llvm-svn: 243490
The linker is now able to link not only LLVM/Clang/LLD for x86 but
even larger programs. I confirmed that it successsfully linked Chrome
for x86. Because the browser is a pretty large program, I think I can
say that the linker is now mostly feature complete. (I'm pretty sure
that there are hidden bugs somewhere, but they shouldn't be significant.)
llvm-svn: 243377
Previously, we ignore /merge option if /debug is specified
because I thought that was MSVC linker did. This was wrong.
/merge shouldn't be ignored even in debug mode.
llvm-svn: 243375
Zachary Turner