MSVC linker is able to link an object file created from the following code.
Note that __imp_hello is not defined anywhere.
void hello() { printf("Hello\n"); }
extern void (*__imp_hello)();
int main() { __imp_hello(); }
Function symbols exported from DLLs are automatically mangled by appending
__imp_ prefix, so they have two names (original one and with the prefix).
This "feature" seems to simulate that behavior even for non-DLL symbols.
This is in my opnion very odd feature. Even MSVC linker warns if you use this.
I'm adding that anyway for the sake of compatibiltiy.
llvm-svn: 240620
Getting an iterator to the relocation table is very hot operation
in the linker. We do that not only to apply relocations but also
to mark live sections and to do ICF.
libObject's interface is slow. By caching pointers to the first
relocation table entries makes the linker 6% faster to self-link.
We probably need to fix libObject as well.
llvm-svn: 240603
Identical COMDAT Folding (ICF) is an optimization to reduce binary
size by merging COMDAT sections that contain the same metadata,
actual data and relocations. MSVC link.exe and many other linkers
have this feature. LLD achieves on per with MSVC in terms produced
binary size with this patch.
This technique is pretty effective. For example, LLD's size is
reduced from 64MB to 54MB by enaling this optimization.
The algorithm implemented in this patch is extremely inefficient.
It puts all COMDAT sections into a set to identify duplicates.
Time to self-link with/without ICF are 3.3 and 320 seconds,
respectively. So this option roughly makes LLD 100x slower.
But it's okay as I wanted to achieve correctness first.
LLD is still able to link itself with this optimization.
I'm going to make it more efficient in followup patches.
Note that this optimization is *not* entirely safe. C/C++ require
different functions have different addresses. If your program
relies on that property, your program wouldn't work with ICF.
However, it's not going to be an issue on Windows because MSVC
link.exe turns ICF on by default. As long as your program works
with default settings (or not passing /opt:noicf), your program
would work with LLD too.
llvm-svn: 240519
Chunks are basically unnamed chunks of bytes, and we don't like
to give them names. However, for logging or debugging, we want to
know symbols names of functions for COMDAT chunks. (For example,
we want to print out "we have removed unreferenced COMDAT section
which contains a function FOOBAR.")
This patch is to do that.
llvm-svn: 240484
Previously, we added files in directive sections to the symbol
table as we read the sections, so the link order was depth-first.
That's not compatible with MSVC link.exe nor the old LLD.
This patch is to queue files so that new files are added to the
end of the queue and processed last. Now addFile() doesn't parse
files nor resolve symbols. You need to call run() to process
queued files.
llvm-svn: 240483
Before this change, you got to cast a symbol to DefinedRegular and then
call isCOMDAT() to determine if a given symbol is a COMDAT symbol.
Now you can just use isa<DefinedCOMDAT>().
As to the class definition of DefinedCOMDAT, I could remove duplicate
code from DefinedRegular and DefinedCOMDAT by introducing another base
class for them, but I chose to not do that to keep the class hierarchy
shallow. This amount of code duplication doesn't worth to define a new
class.
llvm-svn: 240319
DLLs are usually resolved at process startup, but you can
delay-load them by passing /delayload option to the linker.
If a /delayload is specified, the linker has to create data
which is similar to regular import table.
One notable difference is that the pointers in a delay-load
import table are originally pointing to thunks that resolves
themselves. Each thunk loads a DLL, resolve its name, and then
overwrites the pointer with the result so that subsequent
function calls directly call a desired function. The linker
has to emit thunks.
llvm-svn: 240250
.pdata section contains a list of triplets of function start address,
function end address and its unwind information. Linkers have to
sort section contents by function start address and set the section
address to the file header (so that runtime is able to find it and
do binary search.)
This change seems to resolve all but one remaining test failures in
check{,-clang,-lld} when building the entire stuff with clang-cl and
lld-link.
llvm-svn: 240231
This is a case that one mistake caused a very mysterious bug.
I made a mistake to calculate addresses of common symbols, so
each common symbol pointed not to the beginning of its location
but to the end of its location. (Ouch!)
Common symbols are aligned on 16 byte boundaries. If a common
symbol is small enough to fit between the end of its real
location and whatever comes next, this bug didn't cause any harm.
However, if a common symbol is larger than that, its memory
naturally overlapped with other symbols. That means some
uninitialized variables accidentally shared memory. Because
totally unrelated memory writes mutated other varaibles, it was
hard to debug.
It's surprising that LLD was able to link itself and all LLD
tests except gunit tests passed with this nasty bug.
With this fix, the new COFF linker is able to pass all tests
for LLVM, Clang and LLD if I use MSVC cl.exe as a compiler.
Only three tests are failing when used with clang-cl.
llvm-svn: 240216
This avoids undefined behaviour caused by an out-of-range access if the
vector is empty, which can happen if an object file's directive section
contains only whitespace.
llvm-svn: 240183
getName() does strlen() on the symbol table, so it's not very fast.
It's not as bad as r239332 because the number of symbols exported
from archive files are fewer than object files, and they are usually
shorter, though.
llvm-svn: 240178
In this linker model, adding an undefined symbol may trigger chain
reactions. It may trigger a Lazy symbol to read a new file.
A new file may contain a directive section, which may contain various
command line options.
Previously, we didn't handle chain reactions well. We visited /include'd
symbols only once, so newly-added /include symbols were ignored.
This patch fixes that bug.
Now, the symbol table is versioned; every time the symbol table is
updated, the version number is incremented. We repeat adding undefined
symbols until the version number does not change. It is guaranteed to
converge -- the number of undefined symbol in the system is finite,
and adding the same undefined symbol more than once is basically no-op.
llvm-svn: 240177
Alternatename option is in the form of /alternatename:<from>=<to>.
It's effect is to resolve <from> as <to> if <from> is still undefined
at end of name resolution.
If <from> is not undefined but completely a new symbol, alternatename
shouldn't do anything. Previously, it introduced a new undefined
symbol for <from>, which resulted in undefined symbol error.
llvm-svn: 240161
We don't want to insert a new symbol to the symbol table while reading
a .drectve section because it's going to be too complicated.
That we are reading a directive section means that we are currently
reading some object file. Adding a new undefined symbol to the symbol
table can trigger a library file to read a new file, so it would make
the call stack too deep.
In this patch, I add new symbol names to a list to resolve them later.
llvm-svn: 240076
Alternatename option is in the form of /alternatename:<from>=<to>.
It is an error if there are two options having the same <from> but
different <to>. It is *not* an error if both are the same.
llvm-svn: 240075
We skip unknown options in the command line with a warning message
being printed out, but we shouldn't do that for .drectve section.
The section is not visible to the user. We should handle unknown
options as an error.
llvm-svn: 240067
The linker has to create an XML file for each executable.
This patch supports that feature.
You can optionally embed an XML file to an executable as .rsrc
section. If you choose to do that (by passing /manifest:embed
option), the linker has to create a textual resource file
containing an XML file, compile that using rc.exe to a binary
resource file, conver that resource file to a COFF file using
cvtres.exe, and then link that COFF file. This patch implements
that feature too.
llvm-svn: 239978
On Windows, we have to create a .lib file for each .dll.
When linking against DLLs, the linker doesn't use the DLL files,
but instead read a list of dllexported symbols from corresponding
lib files.
A library file containing descriptors of a DLL is called an
import library file.
lib.exe has a feature to create an import library file from a
module-definition file. In this patch, we create a module-definition
file and pass that to lib.exe.
We eventually want to create an import library file by ourselves
to eliminate dependency to lib.exe. For now, we just use the MSVC
tool.
llvm-svn: 239937
Module-definition files (.def files) are yet another way to
specify parameters to the linker. You can write a list of dllexported
symbols in module-definition files instead of using /export command
line option. It also supports a few more directives.
The parser code is taken from lib/Driver/WinLinkModuleDef.cpp
with the following modifications.
- variable names are updated to comply with the LLVM coding style.
- Instead of returning parsing results as "directive" objects,
it updates Config object directly.
llvm-svn: 239929
DLL files are in the same format as executables but they have export tables.
The format of the export table is described in PE/COFF spec section 5.3.
A new class, EdataContents, takes care of creating chunks for export tables.
What we need to do is to parse command line flags for dllexports, and then
instantiate the class to create chunks. For the writer, export table chunks
are opaque data -- it just add chunks to .edata section.
llvm-svn: 239869
We are currently handling all combinations of SymbolBody types directly.
This patch is to flip this and Other if Other->kind() < this->kind()
to reduce number of combinations. No functionality change intended.
llvm-svn: 239745
PE/COFF executables/DLLs usually contain data which is called
base relocations. Base relocations are a list of addresses that
need to be fixed by the loader if load-time relocation is needed.
Base relocations are in .reloc section.
We emit one base relocation entry for each IMAGE_REL_AMD64_ADDR64
relocation.
In order to save disk space, base relocations are grouped by page.
Each group is called a block. A block starts with a 32-bit page
address followed by 16-bit offsets in the page. That is more
efficient representation of addresses than just an array of 32-bit
addresses.
llvm-svn: 239710
When we add a chunk to an OutputSection, we always want to create
a backreference from an OutputSection to a Chunk. To make sure
we always do, do that in addChunk(). NFC.
llvm-svn: 239706
Resource files are data files containing i18n messages, icon images, etc.
MSVC has a tool to convert a resource file to a regular COFF file so that
you can just link that file to embed resources to an executable.
However, you can directly pass resource files to the linker. If you do that,
the linker invokes the tool automatically. This patch implements that feature.
llvm-svn: 239704
In the case where either a bitcode file and a regular file or two bitcode
files export a common or comdat symbol with the same name, the linker needs
to pick one of them following COFF semantics. This patch implements a design
for resolving such symbols that pushes most of the work onto either LLD's
regular mechanism for resolving common or comdat symbols or the IR linker's
mechanism for doing the same.
We modify SymbolBody::compare to always prefer non-bitcode symbols, so that
during the initial phase of symbol resolution, the symbol table always contains
a regular symbol in any case where we need to choose between a regular and
a bitcode symbol. In SymbolTable::addCombinedLTOObject, we force export
any bitcode symbols that were initially pre-empted by a regular symbol,
and later use SymbolBody::compare to choose between the regular symbol in
the symbol table and the regular symbol from the combined LTO object file.
This design seems to be sound, so long as the resolution mechanism is defined
to be commutative and associative modulo arbitrary choices between symbols
(which seems to be the case for COFF).
Differential Revision: http://reviews.llvm.org/D10329
llvm-svn: 239563
isRoot, isLive and markLive functions are called very frequently.
Previously, they were virtual functions. This patch make them
non-virtual.
Also this patch checks chunk liveness before calling its mark().
Previously, we did that at beginning of markLive(), so the virtual
function would return immediately if it's live. That was inefficient.
llvm-svn: 239458
The code generator may create references to runtime library symbols such as
__chkstk which were not visible via LTOModule. Handle these cases by loading
the object file from the library, but abort if we end up having loaded any
bitcode objects.
Because loading the object file may have introduced new undefined references,
call reportRemainingUndefines again to detect and report them.
Differential Revision: http://reviews.llvm.org/D10332
llvm-svn: 239386
The LLVM code generator can sometimes synthesize symbols, such as SSE
constants, that are not visible via the LTOModule interface. Allow such
symbols so long as they have definitions.
Differential Revision: http://reviews.llvm.org/D10331
llvm-svn: 239385
This change seems to make the linker about 10% faster.
Reading symbol name is not very cheap because it needs strlen()
on the string table. We were wasting time on reading non-external
symbol names that would never be used by the linker.
llvm-svn: 239332
MSVC profiler reported that this stable_sort takes 7% time
when self-linking. As a result, createSection was taking 10% time.
Now createSection takes 3%. This small change actually makes
the linker a bit but perceptibly faster.
llvm-svn: 239292
We forgot to check for auxiliary symbol's type. So we sometimes read
garbage as associative section definitions.
Associative sections are considered as not live themselves by the
garbage collector because they are live only when associaited sections
are live.
By reading more data (or garbage) as associative section definitions,
we treated more sections as non-GC-roots, that caused the linker to
discard too many sections by mistake. That caused another mysterious
bug (such as some global constructors don't run at all for some reason.)
llvm-svn: 239287
I don't know what the right thing to do here, but at least 1 does
not seem like a correct value. If we do not align common chunks at
all, a small program which calls puts() from global dtors crashes
mysteriously in a kernel32's function.
I believe the crash was caused by symbols overlapping each other,
and my guess is that alignment has something to do with that, but
I am not 100% sure. Needs investigating.
llvm-svn: 239280
Chunk has writeTo function which takes uint8_t *Buf.
writeHeaderTo feels more consistent with that because this member
function also takes uint8_t *Buf.
llvm-svn: 239236
Previously, half of the constructor for .idata contents was in Chunks.cpp
and the rest was in Writer.cpp. This patch moves the latter to Chunks.cpp.
Now IdataContents class manages everything for .idata section.
llvm-svn: 239230
In this design, Chunk is the only thing that knows how to write
its contents to output file as well as how to apply relocations
there. The writer shouldn't know about the details.
llvm-svn: 239216
Not only entry point symbol but also symbols specified by /include
option must be preserved, as they will never be dead-stripped.
http://reviews.llvm.org/D10220
llvm-svn: 239005
Avoid saying this is based on sections because it's not very accurate.
That we don't split section into smaller chunks of data does not mean
that the linker is built on top of that.
In reality, most part of the code do not care about underlying data,
so they are neither based on "atoms" nor sections.
The symbol table only cares about symbol names and their types.
The writer handles list of chunks, which look like just blobs,
and the writer doesn't care what those chunks are backed by.
The only thing that interact with sections is SectionChunk, which is
abstracted away as one type of Chunk.
llvm-svn: 238902
In r238690, I made all files have only MemoryBufferRefs. This change
is to do the same thing for the bitcode file reader. Also updated
a few variable names to match with other code.
llvm-svn: 238782
Symbols exported by DLLs can be imported not by name but by
small number or ordinal. Usually, symbols have both ordinals
and names, and in that case ordinals are called "hints" and
used by the loader as hints.
However, symbols can have only ordinals. They are called
import-by-ordinal symbols. You need to manage ordinals by hand
so that they will never change if you choose to use the feature.
But it's supposed to make dynamic linking faster because
it needs no string comparison. Not sure if that claim still
stands in year 2015, though. Anyways, the feature exists,
and this patch implements that.
llvm-svn: 238780
I'm adding ordinal-only (nameless) imports to the import table.
The chunk for that type is going to be different from LookupChunk.
Without this change, we cannot add objects of the new type to the
vectors.
llvm-svn: 238779
Instead of returning non-categorized errors, return categorized errors.
All uses of make_dynamic_error_code are removed.
Because we don't have error reporting mechanism, I just chose to print out
error messages to stderr, and then return an error object. Not sure if
that's the right thing to do, but at least it seems practical.
http://reviews.llvm.org/D10129
llvm-svn: 238714
Previously, this feature was implemented using a special type of
undefined symbol, in addition to an intricate way to make the resolver
read a virtual file containing that renaming symbols.
Now the feature is directly handled by the symbol table.
The symbol table has a function, rename(), to rename symbols, whose
definition is 4 lines long. Symbol renaming is naturally modeled using
Symbol and SymbolBody.
llvm-svn: 238696
Previously, a MemoryBuffer of a file was owned by each InputFile object.
This patch makes the Driver own all of them. InputFiles now have only
MemoryBufferRefs. This change simplifies ownership managment
(particularly for ObjectFile -- the object owned a MemoryBuffer only when
it's not created from an archive file, because in that case a parent
archive file owned the entire buffer. Now it owns nothing unconditionally.)
llvm-svn: 238690
It does not involve notions of virtual archives or virtual files,
nor store a list of undefined symbols somewhere else to consume them later.
We did that before. In this patch, undefined symbols are just added to
the symbol table, which now can be done in very few lines of code.
llvm-svn: 238681
Previously the main linker routine is just a non-member function.
We store some context information to the Config object.
This patch makes it belong to Driver.
llvm-svn: 238677
`main` is not the only main function in Windows. You can choose one
from these four -- {w,}{WinMain,main}. There are four different entry
point functions for them, {w,}{WinMain,main}CRTStartup, respectively.
The linker needs to choose the right one depending on which `main`
function is defined.
llvm-svn: 238667
Section names were truncated to 8 bytes because the section table's
name field is 8 byte long. This patch creates the string table to
store long names.
llvm-svn: 238661
The new mechanism is less code, and fixes the case where all inputs
are archives.
Differential Revision: http://reviews.llvm.org/D10136
llvm-svn: 238618
Currently we set the field to zero, but as per the spec, we should
set numbers we read from import library files. The loader uses the
values as starting offsets for binary search when looking up imported
symbols from DLL.
llvm-svn: 238562
The previous implementation's driver file is cluttered by lots of
small functions, and it was hard to find important functions.
Make a separate file to prevent that issue.
llvm-svn: 238482
Previously Writer directly handles writes to a file.
Chunks needed to give Writer a continuous chunk of memory.
That was inefficent if you construct data in chunks because
it would require two memory copies (one to construct a chunk
and the other is to write that to a file).
This patch teaches chunk to write directly to a file.
From readability point of view, this is also good because
you no longer have to call hasData() before calling getData().
llvm-svn: 238464
This is an initial patch for a section-based COFF linker.
The patch has 2300 lines of code including comments and blank lines.
Before diving into details, you want to start from reading README
because it should give you an overview of the design.
All important things are written in the README file, so I write
summary here.
- The linker is already able to self-link on Windows.
- It's significantly faster than the existing implementation.
The existing one takes 5 seconds to link LLD on my machine,
while the new one only takes 1.2 seconds, even though the new
one is not multi-threaded yet. (And a proof-of-concept multi-
threaded version was able to link it in 0.5 seconds.)
- It uses much less memory (250MB vs. 2GB virtual memory space
to self-host).
- IMHO the new code is much simpler and easier to read than
the existing PE/COFF port.
http://reviews.llvm.org/D10036
llvm-svn: 238458
Rui Ueyama