Commit Graph

521 Commits

Author SHA1 Message Date
Peter Collingbourne 2612a32ce5 COFF: Numerous fixes for interaction between LTO and weak externals.
We were previously hitting assertion failures in the writer in cases where
a regular object file defined a weak external symbol that was defined by
a bitcode file. Because /export and /entry name mangling were implemented
using weak externals, the same problem affected mangled symbol names in
bitcode files.

The underlying cause of the problem was that weak external symbols were
being resolved before doing LTO, so the symbol table may have contained stale
references to bitcode symbols. The fix here is to defer weak external symbol
resolution until after LTO.

Also implement support for weak external symbols in bitcode files
by modelling them as replaceable DefinedBitcode symbols.

Differential Revision: http://reviews.llvm.org/D10940

llvm-svn: 241391
2015-07-04 05:28:41 +00:00
Rui Ueyama 0569ecfa9b Revert "COFF: Do not use VirtualSize section header field for directive sections."
This reverts commit r241386 because the issue is addressed in LLVM (r241387).

llvm-svn: 241388
2015-07-04 03:27:46 +00:00
Rui Ueyama cffbe7cb55 COFF: Do not use VirtualSize section header field for directive sections.
Looks like clang-cl sets a bogus value to the field, which makes
getSectionContents() to truncate section contents. This patch directly
uses SizeOfRawData field instead of VirtualSize to see if this can
make buildbot green.

llvm-svn: 241386
2015-07-04 02:26:20 +00:00
Rui Ueyama 3126c6c565 Use map::insert instead of checking existence of a key and insert. NFC.
llvm-svn: 241385
2015-07-04 02:00:22 +00:00
Rui Ueyama a3d463df6f COFF: Print directive section contents if /verbose.
llvm-svn: 241384
2015-07-04 01:39:11 +00:00
Rui Ueyama b0398827c2 COFF: Fix bug in garbage collector.
GC root may have non-regular defined symbols, such as DefinedImportThunk,
so this cast<> was a wrong assumption.

llvm-svn: 241382
2015-07-04 01:10:32 +00:00
Rui Ueyama 4b8cdd20fb COFF: Don't print warning message for identical /export options.
llvm-svn: 241379
2015-07-03 23:23:29 +00:00
Peter Collingbourne da2f094bbb COFF: Fix the case where an object defines a weak external and its alias.
This worked before, but only by accident, and only with assertions disabled.
We ended up storing a DefinedRegular symbol in the WeakAlias field,
and never using it as an Undefined.

Differential Revision: http://reviews.llvm.org/D10934

llvm-svn: 241376
2015-07-03 22:03:36 +00:00
Rui Ueyama a51ce71fdf COFF: Call exit(0) on success to not call destructors.
This change cut the link time of chrome.dll from 24 seconds
to 22 seconds (5% gain). When the control reaches end of link(),
all output files have already been written. All in-memory
objects can just vanish. There is no use to call their dtors.

llvm-svn: 241320
2015-07-03 05:31:35 +00:00
Rui Ueyama d8111f2c2e COFF: Fix ordinal-only delay-imported symbols.
DLLs can export symbols only by ordinal, and DLLs are also able to be
delay-loaded. The combination of the two is valid. I didn't expect
that combination. This patch implements that feature.

With this patch, LLD is now able to link a working executable of Chrome
for 64-bit debug build. The browser seemed to be working fine. Chrome is
good for testing because of its variety and size. It contains various
open-source libraries written by various people. The largest file in
Chrome is chrome.dll whose size is 496MB. LLD can link it in 24 seconds.
MSVC linker takes 48 seconds. So it is exactly 2x faster. (I measured
that with debug info and ICF being turned off.)

With this achievement, I think I can say that the new COFF linker is
now mostly feature complete for x86-64 Windows. I believe there are
still many lingering bugs, though.

llvm-svn: 241318
2015-07-03 04:32:49 +00:00
Rui Ueyama 7a247ee242 COFF: Fix a bug that /delayload was case-sensitive.
llvm-svn: 241316
2015-07-03 01:40:14 +00:00
Rui Ueyama 49d6cd35ad COFF: Fix /base option.
Previously, __ImageBase symbol got a different value than the one
specified by /base:<number> because the symbol was created in the
SymbolTable's constructor. When the constructor is called,
no command line options are processed yet, so the symbol was
created always with the initial value. This caused wrong relocations
and thus caused mysterious crashes of some executables linked by LLD.

llvm-svn: 241313
2015-07-03 00:02:19 +00:00
Rui Ueyama 6be9099140 COFF: Define SymbolTable::insert to simplify. NFC.
llvm-svn: 241311
2015-07-02 22:52:33 +00:00
Rui Ueyama 7a333c66be COFF: Fix locally-imported symbols.
Previously, pointers pointed by locally-imported symbols were broken.
It has only 4 bytes although the correct size is 8 byte. This patch
fixes that bug.

llvm-svn: 241295
2015-07-02 20:33:50 +00:00
Rui Ueyama 65813edfe2 COFF: Make symbols satisfy weak ordering.
Previously, SymbolBody::compare(A, B) didn't satisfy weak ordering.
There was a case that A < B and B < A could have been true.
This is because we just pick LHS if A and B are consisdered equivalent.

This patch is to make symbols being weakly ordered. If A and B are
not tie, one of A < B && B > A or A > B && B < A is true.
This is not an improtant property for a single-threaded environment
because everything is deterministic anyways. However, in a multi-
threaded environment, this property becomes important.

If a symbol is defined or lazy, ties are resolved by its file index.
For simple types that we don't really care about their identities,
symbols are compared by their addresses.

llvm-svn: 241294
2015-07-02 20:33:48 +00:00
Rui Ueyama 458d74421b COFF: Merge SymbolTable::find{,Symbol}. NFC
llvm-svn: 241238
2015-07-02 03:59:04 +00:00
Rui Ueyama 85225b0a36 COFF: Infer entry point as early as possible, but not too early.
On Windows, we have four different main functions, {w,}{main,WinMain}.
The linker has to choose a corresponding entry point function among
{w,}{main,WinMain}CRTStartup. These entry point functions are defined
in the standard library. The linker resolves one of them by looking at
which main function is defined and adding a corresponding undefined
symbol to the symbol table.

Object files containing entry point functions conflicts each other.
For example, we cannot resolve both mainCRTStartup and WinMainCRTStartup
because other symbols defined in the files conflict.

Previously, we inferred CRT function name at the very end of name
resolution. I found that that is sometimes too late. If the linker
already linked one of these four archive member objects, it's too late
to change the decision.

The right thing to do here is to infer entry point name after adding
all symbols from command line files and before adding any other files
(which are specified by directive sections). This patch does that.

llvm-svn: 241236
2015-07-02 03:15:15 +00:00
Rui Ueyama 3d4c69c04d COFF: Resolve AlternateNames using weak aliases.
Previously, we use SymbolTable::rename to resolve AlternateName symbols.
This patch is to merge that mechanism with weak aliases, so that we
remove that function.

llvm-svn: 241230
2015-07-02 02:38:59 +00:00
Rui Ueyama 0744e87fad COFF: Rename getReplacement -> repl.
The previous name was too long to my taste.

llvm-svn: 241215
2015-07-02 00:21:11 +00:00
Rui Ueyama 18f8d2c5c0 COFF: Change GCRoot member type from StringRef to Undefined. NFC.
I think Undefined symbols are a bit more convenient than StringRefs
since SymbolBodies are handles for symbols. You can get resolved
symbols for undefined symbols just by calling getReplacmenet without
looking up the symbol table.

llvm-svn: 241214
2015-07-02 00:21:08 +00:00
Rui Ueyama 6bf638e688 COFF: Simplify and rename findMangle. NFC.
Occasionally we have to resolve an undefined symbol to its
mangled symbol. Previously, we did that on calling side of
findMangle by explicitly updating SymbolBody.
In this patch, mangled symbols are handled as weak aliases
for undefined symbols.

llvm-svn: 241213
2015-07-02 00:04:14 +00:00
Rui Ueyama 4897596728 COFF: Chagne weak alias' type from SymbolBody** to SymbolBody*. NFC.
llvm-svn: 241198
2015-07-01 22:32:23 +00:00
Rui Ueyama 4b6698917d COFF: Simplify SymbolTable::findLazy. NFC.
llvm-svn: 241128
2015-06-30 23:46:52 +00:00
Rui Ueyama 8d3010a1a6 COFF: Change the order of adding symbols to the symbol table.
Previously, the order of adding symbols to the symbol table was simple.
We have a list of all input files. We read each file from beginning of
the list and add all symbols in it to the symbol table.

This patch changes that order. Now all archive files are added to the
symbol table first, and then all the other object files are added.
This shouldn't change the behavior in single-threading, and make room
to parallelize in multi-threading.

In the first step, only lazy symbols are added to the symbol table
because archives contain only Lazy symbols. Member object files
found to be necessary are queued. In the second step, defined and
undefined symbols are added from object files. Adding an undefined
symbol to the symbol table may cause more member files to be added
to the queue. We simply continue reading all object files until the
queue is empty.

Finally, new archive or object files may be added to the queues by
object files' directive sections (which contain new command line
options).

The above process is repeated until we get no new files.

Symbols defined both in object files and in archives can make results
undeterministic. If an archive is read before an object, a new member
file gets linked, while in the other way, no new file would be added.
That is the most popular cause of an undeterministic result or linking
failure as I observed. Separating phases of adding lazy symbols and
undefined symbols makes that deterministic. Adding symbols in each
phase should be parallelizable.

llvm-svn: 241107
2015-06-30 19:35:21 +00:00
Peter Collingbourne f7b27d15f2 COFF: Implement SymbolBody::getDebugName() for DefinedBitcode symbols.
Differential Revision: http://reviews.llvm.org/D10827

llvm-svn: 241029
2015-06-30 00:47:52 +00:00
Rui Ueyama c15139bb6d COFF: Make DefinedCOFF one pointer smaller.
The size of this class actually matters because this is the
most popular class among all classes. We create a Defined symbol
for each defined symbol in a symbol table. That can be millions
for a large program. For example, linking LLD instantiates this
class millions times.

llvm-svn: 241025
2015-06-30 00:10:54 +00:00
Peter Collingbourne 79cfd437b5 COFF: Use LTOModule::getLinkerOpts() instead of reading the linker directives ourselves.
llvm-svn: 241020
2015-06-29 23:26:28 +00:00
Rui Ueyama dae1661436 COFF: Split ObjectFile::createSymbolBody into small functions. NFC.
llvm-svn: 241011
2015-06-29 22:16:21 +00:00
Rui Ueyama 579c21537a Move llvm_unreachable out of switch to avoid -Wswitch-covered-defualt.
llvm-svn: 241008
2015-06-29 21:59:34 +00:00
Rui Ueyama 81dd16a1e0 Silence MSVC "not all control paths return a value" warning.
llvm-svn: 241004
2015-06-29 21:46:46 +00:00
Chandler Carruth 64c17c7d67 [opt] Devirtualize the SymbolBody type hierarchy and start compacting
its members into the base class.

First, to help motivate this kind of change, understand that in
a self-link, LLD creates 5.5 million defined regular symbol bodies (and
6 million symbol bodies total). A significant portion of its time is
spent allocating the memory for these symbols, and befor ethis patch
the defined regular symbol body objects alone consumed some 420mb of
memory during the self link.

As a consequence, I think it is worth expending considerable effort to
make these objects as memory efficient as possible. This is the first of
several components of that. This change starts with the goal of removing
the virtual functins from SymbolBody so that it can avoid having a vptr
embedded in it when it already contains a "kind" member, and that member
can be much more compact than a vptr.

The primary way of doing this is to sink as much of the logic that we
would have to dispatch for into data in the base class. As part of this,
I made the various flags bits that will pack into a bitfield with the
kind tag. I also sank the Name down to eliminate the dispatch for that,
and used LLVM's RTTI-style dispatch for everything else (most of which
is cold and so doesn't matter terribly if we get minutely worse lowering
than a vtable dispatch).

As I was doing this, I wanted to make the RTTI-dispatch (which would
become much hotter than before) as efficient as possible, so I've
re-organized the tags somewhat. Notably, the common case (regular
defined symbols) is now zero which we can test for faster.

I also needed to rewrite the comparison routine used during resolving
symbols. This proved to be quite complex as the semantics of the
existing one were very subtle due to the back-and-forth virtual dispatch
caused by re-dispatching with reversed operands. I've consolidated it to
a single function and tried to comment it quite a bit more to help
explain what is going on. However, this may need more comments or other
explanations. It at least passes all the regression tests. I'm not
working on Windows, so I can't fully test it.

With all of these changes, the size of a DefinedRegular symbol on
a 64-bit build goes from 80 bytes to 64 bytes, and we save approximately
84mb or 20% of the memory consumed by these symbol bodies during the
link.

The link time appears marginally faster as well, and the profile hotness
of the memory allocation subsystem got a bit better, but there is still
a lot of allocation traffic.

Differential Revision: http://reviews.llvm.org/D10792

llvm-svn: 241001
2015-06-29 21:35:48 +00:00
Chandler Carruth ee5bf526eb [cleanup] Clean up the flow of creating a symbol body for regular symbols.
This uses a single cast and test to get the section for the symbol, and
uses the cast_or_null<> pattern throughout to handle the known type but
unknown non-null-ness.

No functionality changed.

Differential Revision: http://reviews.llvm.org/D10791

llvm-svn: 241000
2015-06-29 21:32:37 +00:00
Chandler Carruth 59013c387e [opt] Replace the recursive walk for GC with a worklist algorithm.
This flattens the entire liveness walk from a recursive mark approach to
a worklist approach. It also sinks the worklist management completely
out of the SectionChunk and into the Writer by exposing the ability to
iterato over children of a chunk and over the symbol bodies of relocated
symbols. I'm not 100% happy with the API names, so suggestions welcome
there.

This allows us to use a single worklist for the entire recursive walk
and would also be a natural place to take advantage of parallelism at
some future point.

With this, we completely inline away the GC walk into the
Writer::markLive function and it makes it very easy to profile what is
slow. Currently, time is being wasted checking whether a Chunk isa
SectionChunk (it essentially always is), finding (or skipping)
a replacement for a symbol, and chasing pointers between symbols and
their chunks. There are a bunch of things we can do to fix this, and its
easier to do them after this change IMO.

This change alone saves 1-2% of the time for my self-link of lld.exe
(which I'm running and benchmarking on Linux ironically).

Perhaps more notably, we'll no longer blow out the stack for large
links. =]

Just as an FYI, at this point, I/O is starting to really dominate the
profile. Well over 10% of the time appears to be inside the kernel doing
page table silliness. I think a decent chunk of this can be nuked as
well, but it's a little odd as cross-linking in this way isn't really
the primary goal here.

Differential Revision: http://reviews.llvm.org/D10790

llvm-svn: 240995
2015-06-29 21:12:49 +00:00
Chandler Carruth be6e80b012 [opt] Hoist the call throuh SymbolBody::getReplacement out of the inline
method to get a SymbolBody and into the callers, and kill now dead
includes.

This removes the need to have the SymbolBody definition when we're
defining the inline method and makes it a better inline method. That was
the only reason for a lot of header includes here. Removing these and
using forward declarations actually uncovers a bunch of cross-header
dependencies that I've fixed while I'm here, and will allow me to
introduce some *important* inline code into Chunks.h that requires the
definition of ObjectFile.

No functionality changed at this point.

Differential Revision: http://reviews.llvm.org/D10789

llvm-svn: 240982
2015-06-29 18:50:11 +00:00
Rui Ueyama 2d5e917bce COFF: Handle mangled entry symbol name.
Compilers recognize "main" function and don't mangle its name.
But if you use a different function as a user-defined entry name,
and if you didn't define that function with extern C, your entry
point function name is mangled. And the linker has to be able to
find that. This is relatively rare but can happen.

llvm-svn: 240953
2015-06-29 14:43:07 +00:00
Rui Ueyama 0fc26d21bd COFF: Create an empty file for /pdb.
Most build system depends on existence or time stamp of a file.
This patch is to create an empty file for /pdb:<filename> option
just to satisfy some build rules.

llvm-svn: 240948
2015-06-29 14:27:12 +00:00
Rui Ueyama 6b79ed128a COFF: Fix /export.
Mangled dllexported symbols may be defined in a library.
If that's the case, we have to read a member file from the library.

llvm-svn: 240947
2015-06-29 14:27:10 +00:00
Rui Ueyama 45044f47d3 COFF: Fix logic to find default entry name or subsystem.
The previous logic to find default entry name or subsystem does not
seem correct (i.e. was not compatible with MSVC linker). Previously,
default entry name was inferred from CRT functions and user-defined
entry functions. Subsystem was inferred from CRT functions.

Default entry name and subsystem are now inferred based on the
following table. Note that we no longer use CRT functions to infer
them.

               Entry name           Subsystem
  main         mainCRTStartup       console
  wmain        wmainCRTStartup      console
  WinMain      WinMainCRTStartup    windows
  wWinMain     wWinMainCRTStartup   windows

llvm-svn: 240922
2015-06-29 01:03:53 +00:00
Rui Ueyama f5313b3498 COFF: Allow mangled symbols as arguments for /export.
Usually dllexported symbols are defined with 'extern "C"',
so identifying them is easy. We can just do hash table lookup
to look up exported symbols.

However, C++ non-member functions are also allowed to be exported,
and they can be specified with unmangled name. So, if /export:foo
is given, we need to look up not only "foo" but also its all
mangled names. In MSVC mangling scheme, that means that we need to
look up any symbol which starts with "?foo@@Y".

In this patch, we scan the entire symbol table to search for
a mangled symbol. The symbol table is a DenseMap, and that doesn't
support table lookup by string prefix. This is of course very
inefficient. But that should be probably OK because the user
should always add 'extern "C"' to dllexported symbols.

llvm-svn: 240919
2015-06-28 22:16:41 +00:00
Rui Ueyama f24e6f8607 COFF: Undefined weak aliases are not fatal if /force is given.
llvm-svn: 240917
2015-06-28 20:34:09 +00:00
Rui Ueyama 016414f557 COFF: Add a comment.
llvm-svn: 240916
2015-06-28 20:07:08 +00:00
Rui Ueyama a8b60458ea COFF: Add /noentry flag.
This option is sometimes used to create a resource-only DLL that
doesn't need any initialization.

llvm-svn: 240915
2015-06-28 19:56:30 +00:00
Rui Ueyama 95925fd1ab COFF: Support /force flag.
This option is to ignore remaining undefined symbols and force
the linker to create an output file anyways.

The existing code assumes that there's no undefined symbol after
reportRemainingUndefines(). That assumption is legitimate.
I also don't want to mess up the existing code for this minor feature.
In order to keep it as is, remaining undefined symbols are replaced
with dummy defined symbols.

llvm-svn: 240913
2015-06-28 19:35:15 +00:00
Rui Ueyama 9d72f09efd COFF: Remove a function that doesn't do much itself. NFC.
llvm-svn: 240901
2015-06-28 03:05:38 +00:00
Rui Ueyama 06cf3df2d5 COFF: Handle LINK environment variable.
If LINK is defined and not empty, it's supposed to contain
command line options.

llvm-svn: 240900
2015-06-28 02:35:31 +00:00
Rui Ueyama b0a360bf15 COFF: Remove useless "explicit".
llvm-svn: 240899
2015-06-28 02:00:33 +00:00
Rui Ueyama 50be6edfa6 COFF: Make doICF non-recursive. NFC.
llvm-svn: 240898
2015-06-28 01:35:59 +00:00
Rui Ueyama 871847e32d COFF: Fix ICF correctness bug.
When comparing two COMDAT sections, we need to take section values
and associative sections into account. This patch fixes that bug.
It fixes a crash bug of llvm-tblgen when linked with /opt:lldicf.

One thing I don't understand yet is that this logic seems to be
too strict. MSVC linker is able to create more compact executables
(which of course work correctly). With this ICF algorithm, LLD is
able to make executable smaller, but the outputs are larger than
MSVC's. There must be something I'm missing here.

llvm-svn: 240897
2015-06-28 01:30:54 +00:00
Chandler Carruth 52eb355765 [opt] Inline a trivial lookup function into the header.
This function is actually *very* hot. It is hard to see currently
because the call graph is very recursive, but I'm working to remove that
and when I do this function becomes significantly higher on the profile
(up to 5%!) and so worth avoiding the call overhead.

No specific perf gain I can measure yet (below the noise), but likely to
have more impact as we stop cluttering the call graph.

Differential Revision: http://reviews.llvm.org/D10788

llvm-svn: 240873
2015-06-27 03:40:10 +00:00
Chandler Carruth 2eb15fff94 Switch the new COFF linker's symbol table to use a DenseMap of
StringRefs. This uses the LLVM hashing rather than the standard library
and a closed addressed hash table rather than chaining.

This improves the Windows self-link of LLD by 4.4% (averaged over 10
runs, with well under 1% of variance on each).

There is still some room to improve here. Two things I clearly see in
the profile:

1) This is one of the biggest stress tests for the LLVM hashing code. It
   actually consumes something like 3-4% of the link time after the
   change.
2) The way that StringRef keys are handled in the DenseMap interface is
   pretty suboptimal. We pay the price of checking for empty and
   tombstone keys when we could only possibly be looking for a normal
   key. But fixing this requires invasive API changes.

So there is still some headroom here.

Differential Revision: http://reviews.llvm.org/D10684

llvm-svn: 240871
2015-06-27 02:05:40 +00:00
Rui Ueyama 77731b4909 COFF: Use vector::erase instead of reallocating entire vector. NFC.
llvm-svn: 240862
2015-06-26 23:59:13 +00:00
Rui Ueyama 605e1f6b6c COFF: Avoid vector reallocation. NFC.
llvm-svn: 240859
2015-06-26 23:51:45 +00:00
Rui Ueyama 81ba1353ce COFF: Remove dead code.
llvm-svn: 240846
2015-06-26 22:14:41 +00:00
Rui Ueyama 810551a694 COFF: Add base relocation for delay-import table.
Because the address table of the delay-import table contains
absolute address, it needs to be added to the base relocation
table.

llvm-svn: 240844
2015-06-26 22:05:32 +00:00
Rui Ueyama 382dc96e29 COFF: Fix delay-import tables.
There were a few issues with the previous delay-import tables.

 - "Attribute" field should have been 1 instead of 0.
   (I don't know the meaning of this field, though.)
 - LEA and CALL operands had wrong addresses.
 - Address tables are in .didat (which is read-only).
   They should have been in .data.

llvm-svn: 240837
2015-06-26 21:40:15 +00:00
Peter Collingbourne baf5f87b6c Fix MSVC build.
llvm-svn: 240818
2015-06-26 19:20:09 +00:00
Peter Collingbourne be54955bba COFF: Implement /lldmap flag.
This flag can be used to produce a map file, which is essentially a list
of objects linked into the final output file together with the RVAs of
their symbols. Because our format differs from MSVC's we expose it as a
separate flag.

Differential Revision: http://reviews.llvm.org/D10773

llvm-svn: 240812
2015-06-26 18:58:24 +00:00
Rui Ueyama 7383562bc9 COFF: Align DLL import thunks on 16-byte boundaries.
llvm-svn: 240806
2015-06-26 18:28:56 +00:00
Rui Ueyama 5740abd7d3 COFF: Fix README.
llvm-svn: 240802
2015-06-26 17:59:12 +00:00
Rui Ueyama 3afe908294 COFF: Update README with the latest performance numbers.
llvm-svn: 240759
2015-06-26 04:26:02 +00:00
Rui Ueyama 32f8e1cb4e COFF: Change symbol resolution order for entry and /include.
We were resolving entry symbols and /include'd symbols after all other
symbols are resolved. But looks like it's too late. I found that it
causes some program to fail to link.

Let's say we have an object file A which defines symbols X and Y in an
archive. We also have another file B after A which defines X, Y and
_DLLMainCRTStartup in another archive. They conflict each other, so
either A or B can be linked.

If we have _DLLMainCRTStartup as an undefined symbol, file B is always
chosen. If not, there's a chance that A is chosen. If the linker
find it needs _DllMainCRTStartup after that, it's too late.

This patch adds undefined symbols to the symbol table as soon as
possible to fix the issue.

llvm-svn: 240757
2015-06-26 03:44:00 +00:00
Rui Ueyama ccde19d77e COFF: Fix local absolute symbols.
Absolute symbols were always handled as external symbols, so if two
or more object files define the same absolute symbol, they would
conflict even if the symbol is private to each file.
This patch fixes that bug.

llvm-svn: 240756
2015-06-26 03:09:23 +00:00
Rui Ueyama a29948873f COFF: Don't read non-x64 object files.
Currently the new LLD supports only x86-64.

llvm-svn: 240749
2015-06-26 00:42:21 +00:00
Rui Ueyama f799edef28 COFF: Rename /opt:icf -> /opt:lldicf.
ICF implemented in LLD is so experimental that we don't want to
enable that even if /opt:icf option is passed. I'll rename it back
once the feature is complete.

llvm-svn: 240721
2015-06-25 23:26:58 +00:00
Rui Ueyama 68633f1719 COFF: Better error message for duplicate symbols.
Now the symbol table prints out not only symbol names but
also file names for duplicate symbols.

llvm-svn: 240719
2015-06-25 23:22:00 +00:00
Rui Ueyama 9b921e5dc9 COFF: Merge DefinedRegular and DefinedCOMDAT.
I split them in r240319 because I thought they are different enough
that we should treat them as different types. It turned out that
that was not a good idea. They are so similar that we ended up having
many duplicate code.

llvm-svn: 240706
2015-06-25 22:00:42 +00:00
Rui Ueyama 5817ebb0c8 COFF: Fix lexer for the module-definition file.
Previously it would hang if there's a stray punctuation (e.g. ?).

llvm-svn: 240697
2015-06-25 21:06:00 +00:00
Rui Ueyama b0c001c055 COFF: Remove dead code.
llvm-svn: 240682
2015-06-25 20:12:15 +00:00
Rui Ueyama fc510f4cf8 COFF: Devirtualize mark(), markLive() and isCOMDAT().
Only SectionChunk can be dead-stripped. Previously,
all types of chunks implemented these functions,
but their functions were blank.

Likewise, only DefinedRegular and DefinedCOMDAT symbols
can be dead-stripped. markLive() function was implemented
for other symbol types, but they were blank.

I started thinking that the change I made in r240319 was
a mistake. I separated DefinedCOMDAT from DefinedRegular
because I thought that would make the code cleaner, but now
we want to handle them as the same type here. Maybe we
should roll it back.

This change should improve readability a bit as this removes
some dubious uses of reinterpret_cast. Previously, we
assumed that all COMDAT chunks are actually SectionChunks,
which was not very obvious.

llvm-svn: 240675
2015-06-25 19:10:58 +00:00
Rui Ueyama f34c088515 COFF: Simplify. NFC.
llvm-svn: 240666
2015-06-25 17:56:36 +00:00
Rui Ueyama c6fcfbc98a COFF: Use std::equal to compare two lists of relocations.
llvm-svn: 240665
2015-06-25 17:51:07 +00:00
Rui Ueyama 02c302790f COFF: Don't use COFFHeader->NumberOfRelocations.
The size of the field is 16 bit, so it's inaccurate if the
number of relocations in a section is more than 65535.

llvm-svn: 240661
2015-06-25 17:43:37 +00:00
Rui Ueyama 88e0f9206b COFF: Fix a bug of __imp_ symbol.
The change I made in r240620 was not correct. If a symbol foo is
defined, and if you use __imp_foo, __imp_foo symbol is automatically
defined as a pointer (not just an alias) to foo.

Now that we need to create a chunk for automatically-created symbols.
I defined LocalImportChunk class for them.

llvm-svn: 240622
2015-06-25 03:31:47 +00:00
Rui Ueyama d766653534 COFF: Handle undefined symbols starting with __imp_ in a special way.
MSVC linker is able to link an object file created from the following code.
Note that __imp_hello is not defined anywhere.

  void hello() { printf("Hello\n"); }
  extern void (*__imp_hello)();
  int main() { __imp_hello(); }

Function symbols exported from DLLs are automatically mangled by appending
__imp_ prefix, so they have two names (original one and with the prefix).
This "feature" seems to simulate that behavior even for non-DLL symbols.

This is in my opnion very odd feature. Even MSVC linker warns if you use this.
I'm adding that anyway for the sake of compatibiltiy.

llvm-svn: 240620
2015-06-25 02:21:44 +00:00
Rui Ueyama 42aa00b34b COFF: Use COFFObjectFile::getRelocations(). NFC.
llvm-svn: 240614
2015-06-25 00:33:38 +00:00
Rui Ueyama cde92423d7 COFF: Cache raw pointers to relocation tables.
Getting an iterator to the relocation table is very hot operation
in the linker. We do that not only to apply relocations but also
to mark live sections and to do ICF.

libObject's interface is slow. By caching pointers to the first
relocation table entries makes the linker 6% faster to self-link.

We probably need to fix libObject as well.

llvm-svn: 240603
2015-06-24 23:03:17 +00:00
Rui Ueyama 49560c7a10 COFF: Move code for ICF from Writer.cpp to ICF.cpp.
llvm-svn: 240590
2015-06-24 20:40:03 +00:00
Rui Ueyama ddf71fc370 COFF: Initial implementation of Identical COMDAT Folding.
Identical COMDAT Folding (ICF) is an optimization to reduce binary
size by merging COMDAT sections that contain the same metadata,
actual data and relocations. MSVC link.exe and many other linkers
have this feature. LLD achieves on per with MSVC in terms produced
binary size with this patch.

This technique is pretty effective. For example, LLD's size is
reduced from 64MB to 54MB by enaling this optimization.

The algorithm implemented in this patch is extremely inefficient.
It puts all COMDAT sections into a set to identify duplicates.
Time to self-link with/without ICF are 3.3 and 320 seconds,
respectively. So this option roughly makes LLD 100x slower.
But it's okay as I wanted to achieve correctness first.
LLD is still able to link itself with this optimization.
I'm going to make it more efficient in followup patches.

Note that this optimization is *not* entirely safe. C/C++ require
different functions have different addresses. If your program
relies on that property, your program wouldn't work with ICF.
However, it's not going to be an issue on Windows because MSVC
link.exe turns ICF on by default. As long as your program works
with default settings (or not passing /opt:noicf), your program
would work with LLD too.

llvm-svn: 240519
2015-06-24 04:36:52 +00:00
Peter Collingbourne bd3a29d063 COFF: Remove unused field SectionChunk::SectionIndex.
llvm-svn: 240512
2015-06-24 00:12:36 +00:00
Peter Collingbourne 2ed4c8f55d COFF: Add some error checking to SymbolTable::addCombinedLTOObject().
llvm-svn: 240511
2015-06-24 00:12:34 +00:00
Peter Collingbourne c7b685d997 COFF: Ignore debug symbols.
Differential Revision: http://reviews.llvm.org/D10675

llvm-svn: 240487
2015-06-24 00:05:50 +00:00
Rui Ueyama 6a60be7749 COFF: Add names for logging/debugging to COMDAT chunks.
Chunks are basically unnamed chunks of bytes, and we don't like
to give them names. However, for logging or debugging, we want to
know symbols names of functions for COMDAT chunks. (For example,
we want to print out "we have removed unreferenced COMDAT section
which contains a function FOOBAR.")

This patch is to do that.

llvm-svn: 240484
2015-06-24 00:00:52 +00:00
Rui Ueyama 0d2e999050 COFF: Make link order compatible with MSVC link.exe.
Previously, we added files in directive sections to the symbol
table as we read the sections, so the link order was depth-first.
That's not compatible with MSVC link.exe nor the old LLD.

This patch is to queue files so that new files are added to the
end of the queue and processed last. Now addFile() doesn't parse
files nor resolve symbols. You need to call run() to process
queued files.

llvm-svn: 240483
2015-06-23 23:56:39 +00:00
Peter Collingbourne bf0aa08bba COFF: Fix null pointer dereference.
llvm-svn: 240447
2015-06-23 20:02:31 +00:00
Benjamin Kramer 44b0723069 Add missing dependencies for the CMake shared lld build.
llvm-svn: 240445
2015-06-23 19:54:57 +00:00
David Blaikie 6521ed964b Update for LLVM API change to return by InputArgList directly (rather than by pointer) from ParseArgs
llvm-svn: 240347
2015-06-22 22:06:52 +00:00
David Blaikie 008181933d Fix missed formatting in prior commit (mostly 80 cols violation and some whitespace around *)
llvm-svn: 240346
2015-06-22 22:06:48 +00:00
Rui Ueyama 617f5ccb5c COFF: Separate DefinedCOMDAT from DefinedRegular symbol type. NFC.
Before this change, you got to cast a symbol to DefinedRegular and then
call isCOMDAT() to determine if a given symbol is a COMDAT symbol.
Now you can just use isa<DefinedCOMDAT>().

As to the class definition of DefinedCOMDAT, I could remove duplicate
code from DefinedRegular and DefinedCOMDAT by introducing another base
class for them, but I chose to not do that to keep the class hierarchy
shallow. This amount of code duplication doesn't worth to define a new
class.

llvm-svn: 240319
2015-06-22 19:56:01 +00:00
Rui Ueyama 610962061f Fix typo.
llvm-svn: 240298
2015-06-22 17:26:27 +00:00
Rui Ueyama a77336bd5d COFF: Support delay-load import tables.
DLLs are usually resolved at process startup, but you can
delay-load them by passing /delayload option to the linker.

If a /delayload is specified, the linker has to create data
which is similar to regular import table.
One notable difference is that the pointers in a delay-load
import table are originally pointing to thunks that resolves
themselves. Each thunk loads a DLL, resolve its name, and then
overwrites the pointer with the result so that subsequent
function calls directly call a desired function. The linker
has to emit thunks.

llvm-svn: 240250
2015-06-21 22:31:52 +00:00
David Blaikie b2b1c7c3e1 ArrayRef-ify Driver::parse and related functions.
llvm-svn: 240236
2015-06-21 06:32:10 +00:00
David Blaikie 8da889f1a5 ArrayRef-ify ParseArgs
llvm-svn: 240235
2015-06-21 06:32:04 +00:00
Rui Ueyama 1a109285c2 COFF: Use short varaible name. NFC.
llvm-svn: 240232
2015-06-21 04:10:54 +00:00
Rui Ueyama 4d769c3a57 COFF: Support exception table.
.pdata section contains a list of triplets of function start address,
function end address and its unwind information. Linkers have to
sort section contents by function start address and set the section
address to the file header (so that runtime is able to find it and
do binary search.)

This change seems to resolve all but one remaining test failures in
check{,-clang,-lld} when building the entire stuff with clang-cl and
lld-link.

llvm-svn: 240231
2015-06-21 04:00:54 +00:00
Rui Ueyama e3a335076a COFF: Combine add{Object,Archive,Bitcode,Import} functions. NFC.
llvm-svn: 240229
2015-06-20 23:10:05 +00:00
Rui Ueyama 5e31d0b2e9 COFF: Fix common symbol alignment.
llvm-svn: 240217
2015-06-20 07:25:45 +00:00
Rui Ueyama efb7e1aa29 COFF: Fix a common symbol bug.
This is a case that one mistake caused a very mysterious bug.
I made a mistake to calculate addresses of common symbols, so
each common symbol pointed not to the beginning of its location
but to the end of its location. (Ouch!)

Common symbols are aligned on 16 byte boundaries. If a common
symbol is small enough to fit between the end of its real
location and whatever comes next, this bug didn't cause any harm.

However, if a common symbol is larger than that, its memory
naturally overlapped with other symbols. That means some
uninitialized variables accidentally shared memory. Because
totally unrelated memory writes mutated other varaibles, it was
hard to debug.

It's surprising that LLD was able to link itself and all LLD
tests except gunit tests passed with this nasty bug.

With this fix, the new COFF linker is able to pass all tests
for LLVM, Clang and LLD if I use MSVC cl.exe as a compiler.
Only three tests are failing when used with clang-cl.

llvm-svn: 240216
2015-06-20 07:21:57 +00:00
Peter Collingbourne 74ecc89c46 COFF: Take reference to argument vector using std::vector::data() instead of operator[](0).
This avoids undefined behaviour caused by an out-of-range access if the
vector is empty, which can happen if an object file's directive section
contains only whitespace.

llvm-svn: 240183
2015-06-19 22:40:05 +00:00
Rui Ueyama f00df0af2d COFF: Fix precedence between LIB and /libpath.
/libpath should take precedence over LIB.
Previously, LIB took precedence over /libpath.

llvm-svn: 240182
2015-06-19 22:39:48 +00:00
Rui Ueyama 165b254e06 COFF: Add search paths in the correct order.
Previously, we added search paths in reverse order.

llvm-svn: 240180
2015-06-19 21:44:32 +00:00
Rui Ueyama 29792a82a9 COFF: Cache Archive::Symbol::getName(). NFC.
getName() does strlen() on the symbol table, so it's not very fast.
It's not as bad as r239332 because the number of symbols exported
from archive files are fewer than object files, and they are usually
shorter, though.

llvm-svn: 240178
2015-06-19 21:25:44 +00:00
Rui Ueyama 573bf7de9c COFF: Continue reading object files until converge.
In this linker model, adding an undefined symbol may trigger chain
reactions. It may trigger a Lazy symbol to read a new file.
A new file may contain a directive section, which may contain various
command line options.

Previously, we didn't handle chain reactions well. We visited /include'd
symbols only once, so newly-added /include symbols were ignored.
This patch fixes that bug.

Now, the symbol table is versioned; every time the symbol table is
updated, the version number is incremented. We repeat adding undefined
symbols until the version number does not change. It is guaranteed to
converge -- the number of undefined symbol in the system is finite,
and adding the same undefined symbol more than once is basically no-op.

llvm-svn: 240177
2015-06-19 21:12:48 +00:00
Rui Ueyama 4d2834bd7b COFF: Don't add new undefined symbols for /alternatename.
Alternatename option is in the form of /alternatename:<from>=<to>.
It's effect is to resolve <from> as <to> if <from> is still undefined
at end of name resolution.

If <from> is not undefined but completely a new symbol, alternatename
shouldn't do anything. Previously, it introduced a new undefined
symbol for <from>, which resulted in undefined symbol error.

llvm-svn: 240161
2015-06-19 19:23:43 +00:00
Rui Ueyama ce86c9962c COFF: Add /nodefaultlib and /merge for .drectve.
llvm-svn: 240077
2015-06-18 23:22:39 +00:00
Rui Ueyama 08d5e1875f COFF: Handle /include in .drectve.
We don't want to insert a new symbol to the symbol table while reading
a .drectve section because it's going to be too complicated.
That we are reading a directive section means that we are currently
reading some object file. Adding a new undefined symbol to the symbol
table can trigger a library file to read a new file, so it would make
the call stack too deep.

In this patch, I add new symbol names to a list to resolve them later.

llvm-svn: 240076
2015-06-18 23:20:11 +00:00
Rui Ueyama e8d56b5258 COFF: Allow identical alternatename options.
Alternatename option is in the form of /alternatename:<from>=<to>.
It is an error if there are two options having the same <from> but
different <to>. It is *not* an error if both are the same.

llvm-svn: 240075
2015-06-18 23:04:26 +00:00
Rui Ueyama 562daa8148 COFF: Unknown options in .drectve section is an error.
We skip unknown options in the command line with a warning message
being printed out, but we shouldn't do that for .drectve section.
The section is not visible to the user. We should handle unknown
options as an error.

llvm-svn: 240067
2015-06-18 21:50:38 +00:00
Rui Ueyama 75b098b29d COFF: Handle /failifmismatch in the same manner as other options.
No functionality change intended.

llvm-svn: 240061
2015-06-18 21:23:34 +00:00
Rui Ueyama 223fe1b9e7 COFF: Fix unsafe memory access.
llvm-svn: 240046
2015-06-18 20:29:41 +00:00
Rui Ueyama b95188cb2c COFF: Add /implib option.
llvm-svn: 240045
2015-06-18 20:27:09 +00:00
Rui Ueyama ea63a28364 COFF: Handle both / and \ as path separator.
llvm-svn: 240042
2015-06-18 20:16:26 +00:00
Rui Ueyama 2edb35a264 COFF: Handle /alternatename in .drectve section.
llvm-svn: 240037
2015-06-18 19:09:30 +00:00
Rui Ueyama 23ed96d95f COFF: Rename a function. NFC.
llvm-svn: 240031
2015-06-18 17:29:50 +00:00
Peter Collingbourne 8b2492f2a0 COFF: Implement DLL symbol exports for bitcode files.
Differential Revision: http://reviews.llvm.org/D10530

llvm-svn: 239994
2015-06-18 05:22:15 +00:00
Rui Ueyama ae36985af7 COFF: Fix entry point inference bug.
Previously, LLD couldn't find a default entry point if it's
defined by a library.

llvm-svn: 239982
2015-06-18 00:40:33 +00:00
Rui Ueyama 24c5fd0419 COFF: Support /manifest{,uac,dependency,file} options.
The linker has to create an XML file for each executable.
This patch supports that feature.

You can optionally embed an XML file to an executable as .rsrc
section. If you choose to do that (by passing /manifest:embed
option), the linker has to create a textual resource file
containing an XML file, compile that using rc.exe to a binary
resource file, conver that resource file to a COFF file using
cvtres.exe, and then link that COFF file. This patch implements
that feature too.

llvm-svn: 239978
2015-06-18 00:12:42 +00:00
Rui Ueyama a9c8838f69 COFF: Simplify. NFC.
Executor is a convenience class to run an external command.

llvm-svn: 239945
2015-06-17 21:01:56 +00:00
Rui Ueyama 151d862d97 COFF: Create import library files.
On Windows, we have to create a .lib file for each .dll.
When linking against DLLs, the linker doesn't use the DLL files,
but instead read a list of dllexported symbols from corresponding
lib files.

A library file containing descriptors of a DLL is called an
import library file.

lib.exe has a feature to create an import library file from a
module-definition file. In this patch, we create a module-definition
file and pass that to lib.exe.

We eventually want to create an import library file by ourselves
to eliminate dependency to lib.exe. For now, we just use the MSVC
tool.

llvm-svn: 239937
2015-06-17 20:40:43 +00:00
Rui Ueyama 1f373704e3 COFF: Support module-definition files.
Module-definition files (.def files) are yet another way to
specify parameters to the linker. You can write a list of dllexported
symbols in module-definition files instead of using /export command
line option. It also supports a few more directives.

The parser code is taken from lib/Driver/WinLinkModuleDef.cpp
with the following modifications.

 - variable names are updated to comply with the LLVM coding style.
 - Instead of returning parsing results as "directive" objects,
   it updates Config object directly.

llvm-svn: 239929
2015-06-17 19:19:25 +00:00
Rui Ueyama 97dff9ee3a COFF: Support creating DLLs.
DLL files are in the same format as executables but they have export tables.
The format of the export table is described in PE/COFF spec section 5.3.

A new class, EdataContents, takes care of creating chunks for export tables.
What we need to do is to parse command line flags for dllexports, and then
instantiate the class to create chunks. For the writer, export table chunks
are opaque data -- it just add chunks to .edata section.

llvm-svn: 239869
2015-06-17 00:16:33 +00:00
Rui Ueyama 6592ff8c93 COFF: Add miscellaneous boolean flags.
llvm-svn: 239864
2015-06-16 23:13:00 +00:00
Rui Ueyama e25147626c COFF: Simplify SymbolBody::compare(SymbolBody *Other).
We are currently handling all combinations of SymbolBody types directly.
This patch is to flip this and Other if Other->kind() < this->kind()
to reduce number of combinations. No functionality change intended.

llvm-svn: 239745
2015-06-15 19:06:53 +00:00
Rui Ueyama bc2cc7d0b8 COFF: Fix .reloc section attributes.
llvm-svn: 239738
2015-06-15 18:03:47 +00:00
Rui Ueyama 6200b6d593 COFF: Update README.
llvm-svn: 239734
2015-06-15 16:25:11 +00:00
Rui Ueyama f3770d3edb COFF: Use ulittle32_t::operator|=. NFC.
llvm-svn: 239717
2015-06-15 03:03:23 +00:00
Rui Ueyama 095409e9e8 COFF: Add a brief description about LTO.
llvm-svn: 239714
2015-06-15 02:46:18 +00:00
Rui Ueyama 59e9578f20 COFF: Fix resource table size.
The size field shouldn't include trailing padding.

llvm-svn: 239712
2015-06-15 01:35:56 +00:00
Rui Ueyama 588e832d0a COFF: Support base relocations.
PE/COFF executables/DLLs usually contain data which is called
base relocations. Base relocations are a list of addresses that
need to be fixed by the loader if load-time relocation is needed.

Base relocations are in .reloc section.

We emit one base relocation entry for each IMAGE_REL_AMD64_ADDR64
relocation.

In order to save disk space, base relocations are grouped by page.
Each group is called a block. A block starts with a 32-bit page
address followed by 16-bit offsets in the page. That is more
efficient representation of addresses than just an array of 32-bit
addresses.

llvm-svn: 239710
2015-06-15 01:23:58 +00:00
Rui Ueyama 9a03362a08 COFF: Change const name. NFC.
llvm-svn: 239707
2015-06-14 22:21:29 +00:00
Rui Ueyama 669236fef3 COFF: Set Chunk to OutputSection backreference in addChunk().
When we add a chunk to an OutputSection, we always want to create
a backreference from an OutputSection to a Chunk. To make sure
we always do, do that in addChunk(). NFC.

llvm-svn: 239706
2015-06-14 22:16:47 +00:00
Rui Ueyama 4108f3f393 COFF: Add an assertion. NFC.
r239458 changed callee side of this function, so Live can never be
true when this function is called.

llvm-svn: 239705
2015-06-14 22:01:39 +00:00
Rui Ueyama 2bf6a12238 COFF: Support Windows resource files.
Resource files are data files containing i18n messages, icon images, etc.
MSVC has a tool to convert a resource file to a regular COFF file so that
you can just link that file to embed resources to an executable.

However, you can directly pass resource files to the linker. If you do that,
the linker invokes the tool automatically. This patch implements that feature.

llvm-svn: 239704
2015-06-14 21:50:50 +00:00
Rafael Espindola 9bd82e9952 Update for llvm api change.
llvm-svn: 239671
2015-06-13 12:50:13 +00:00
Davide Italiano d106ab263a [COFF] Spell the namespace correctly.
llvm-svn: 239641
2015-06-12 21:37:55 +00:00
Peter Collingbourne 1b6fd1f5fd COFF: Symbol resolution for common and comdat symbols defined in bitcode.
In the case where either a bitcode file and a regular file or two bitcode
files export a common or comdat symbol with the same name, the linker needs
to pick one of them following COFF semantics. This patch implements a design
for resolving such symbols that pushes most of the work onto either LLD's
regular mechanism for resolving common or comdat symbols or the IR linker's
mechanism for doing the same.

We modify SymbolBody::compare to always prefer non-bitcode symbols, so that
during the initial phase of symbol resolution, the symbol table always contains
a regular symbol in any case where we need to choose between a regular and
a bitcode symbol. In SymbolTable::addCombinedLTOObject, we force export
any bitcode symbols that were initially pre-empted by a regular symbol,
and later use SymbolBody::compare to choose between the regular symbol in
the symbol table and the regular symbol from the combined LTO object file.

This design seems to be sound, so long as the resolution mechanism is defined
to be commutative and associative modulo arbitrary choices between symbols
(which seems to be the case for COFF).

Differential Revision: http://reviews.llvm.org/D10329

llvm-svn: 239563
2015-06-11 21:49:54 +00:00
Rui Ueyama 8b33f59bfd COFF: De-virtualize and inline garbage collector functions.
isRoot, isLive and markLive functions are called very frequently.
Previously, they were virtual functions. This patch make them
non-virtual.

Also this patch checks chunk liveness before calling its mark().
Previously, we did that at beginning of markLive(), so the virtual
function would return immediately if it's live. That was inefficient.

llvm-svn: 239458
2015-06-10 04:21:47 +00:00
Peter Collingbourne bd1cb792d3 COFF: Implement /lib using LibDriver.
Differential Revision: http://reviews.llvm.org/D10347

llvm-svn: 239436
2015-06-09 21:52:48 +00:00
Rui Ueyama efba7812cc COFF: Split SymbolTable::addCombinedLTOObject. NFC.
llvm-svn: 239418
2015-06-09 17:52:17 +00:00
Rui Ueyama 0e77d227f8 COFF: Update comment.
llvm-svn: 239413
2015-06-09 16:52:56 +00:00
Peter Collingbourne 73b75e3d0c COFF: Handle references from LTO object to lazy symbols correctly.
The code generator may create references to runtime library symbols such as
__chkstk which were not visible via LTOModule. Handle these cases by loading
the object file from the library, but abort if we end up having loaded any
bitcode objects.

Because loading the object file may have introduced new undefined references,
call reportRemainingUndefines again to detect and report them.

Differential Revision: http://reviews.llvm.org/D10332

llvm-svn: 239386
2015-06-09 04:29:54 +00:00
Peter Collingbourne d9e4e98cce COFF: Allow the combined LTO object to define new symbols.
The LLVM code generator can sometimes synthesize symbols, such as SSE
constants, that are not visible via the LTOModule interface. Allow such
symbols so long as they have definitions.

Differential Revision: http://reviews.llvm.org/D10331

llvm-svn: 239385
2015-06-09 02:53:09 +00:00
Peter Collingbourne df637ea289 COFF: Skip internal symbols in bitcode files.
Differential Revision: http://reviews.llvm.org/D10319

llvm-svn: 239338
2015-06-08 20:21:28 +00:00
Rui Ueyama 57fe78d339 COFF: Read symbol names lazily.
This change seems to make the linker about 10% faster.
Reading symbol name is not very cheap because it needs strlen()
on the string table. We were wasting time on reading non-external
symbol names that would never be used by the linker.

llvm-svn: 239332
2015-06-08 19:43:59 +00:00
Rui Ueyama f533d3e09d COFF: Avoid callign stable_sort.
MSVC profiler reported that this stable_sort takes 7% time
when self-linking. As a result, createSection was taking 10% time.
Now createSection takes 3%. This small change actually makes
the linker a bit but perceptibly faster.

llvm-svn: 239292
2015-06-08 08:26:28 +00:00
Rui Ueyama 7d80640f25 COFF: Use the empty string as the current directory instead of ".".
This is NFC but makes log message a bit nicer because it doesn't
append .\ (or ./ on Unix) to files in the current directory.

llvm-svn: 239290
2015-06-08 06:13:12 +00:00
Rui Ueyama eeae5ddbe2 COFF: Add more log messages.
llvm-svn: 239289
2015-06-08 06:00:10 +00:00
Rui Ueyama 5b2588ae8c COFF: Print out log messages to stdout.
llvm-svn: 239288
2015-06-08 05:43:50 +00:00
Rui Ueyama 80141a4bcd COFF: Check for auxiliary symbol's type.
We forgot to check for auxiliary symbol's type. So we sometimes read
garbage as associative section definitions.

Associative sections are considered as not live themselves by the
garbage collector because they are live only when associaited sections
are live.

By reading more data (or garbage) as associative section definitions,
we treated more sections as non-GC-roots, that caused the linker to
discard too many sections by mistake. That caused another mysterious
bug (such as some global constructors don't run at all for some reason.)

llvm-svn: 239287
2015-06-08 05:00:42 +00:00
Rui Ueyama f811472b4c COFF: Simplify InputFile class.
Now that all InputFile subclasses have MemoryBufferRefs and
provides the same set of functions. Implement that in the base class.

llvm-svn: 239281
2015-06-08 03:27:57 +00:00
Rui Ueyama 9cf1abb8d4 COFF: Set non-1 alignment to common chunks.
I don't know what the right thing to do here, but at least 1 does
not seem like a correct value. If we do not align common chunks at
all, a small program which calls puts() from global dtors crashes
mysteriously in a kernel32's function.

I believe the crash was caused by symbols overlapping each other,
and my guess is that alignment has something to do with that, but
I am not 100% sure. Needs investigating.

llvm-svn: 239280
2015-06-08 03:17:07 +00:00
Rui Ueyama b4f791b510 COFF: Fix memory leak.
llvm-svn: 239272
2015-06-08 00:09:25 +00:00
Rui Ueyama a6cd6c0cd8 COFF: Fix typo.
This change doesn't change its functionality since the value
passed here is converted to uint16_t immediately.

llvm-svn: 239271
2015-06-07 23:10:50 +00:00
Rui Ueyama aace577e3e COFF: Simplify. NFC.
llvm-svn: 239270
2015-06-07 23:02:50 +00:00
Rui Ueyama b51f67a175 COFF: Use llvm:🆑:ExpandReponseFiles.
llvm-svn: 239269
2015-06-07 23:00:29 +00:00
Rui Ueyama 0c35b38fd2 COFF: Add a glossary to README.
llvm-svn: 239268
2015-06-07 22:42:52 +00:00
Rui Ueyama c6b87363a1 COFF: Use named constants instead of sizeof().
llvm-svn: 239267
2015-06-07 22:00:28 +00:00
Rui Ueyama b3aa5e71a0 COFF: Remove dead code.
/include'ed symbols are already added to Config->GCRoots.
Marking symbols twice doesn't have any effect.

llvm-svn: 239266
2015-06-07 21:58:34 +00:00
Rui Ueyama 94df713199 COFF: Make local variables local.
llvm-svn: 239244
2015-06-07 03:55:28 +00:00
Rui Ueyama e2cbfeae5c COFF: Add /opt:noref option.
This option disables dead-stripping.

llvm-svn: 239243
2015-06-07 03:17:42 +00:00
Rui Ueyama 115d7c1036 COFF: Support resonpse files.
llvm-svn: 239242
2015-06-07 02:55:19 +00:00
Rui Ueyama 4b22fa7437 COFF: Move Windows-specific code from Chunk.{cpp,h} to DLL.{cpp,h}.
llvm-svn: 239239
2015-06-07 01:15:04 +00:00
Rui Ueyama ad66098c20 COFF: Fix default output file path.
Default output filename is the same as the first object file's
name with its extension replaced with ".exe".

llvm-svn: 239238
2015-06-07 00:20:32 +00:00
Rui Ueyama 4a9fbbca9f COFF: Add comments and move main function to the top. NFC.
llvm-svn: 239237
2015-06-06 23:32:08 +00:00
Rui Ueyama cc608e4f35 COFF: Rename writeHeader -> writeHeaderTo.
Chunk has writeTo function which takes uint8_t *Buf.
writeHeaderTo feels more consistent with that because this member
function also takes uint8_t *Buf.

llvm-svn: 239236
2015-06-06 23:19:38 +00:00
Rui Ueyama 929d8c52b1 COFF: Inline a constant that is used only once.
llvm-svn: 239235
2015-06-06 23:19:36 +00:00
Rui Ueyama e56f9c0883 Remove redundant `using namespace`.
llvm-svn: 239234
2015-06-06 23:11:39 +00:00
Rui Ueyama 55168c9f70 COFF: Add .didat section.
llvm-svn: 239233
2015-06-06 23:07:01 +00:00
Rui Ueyama 458df98869 COFF: Update comments.
llvm-svn: 239232
2015-06-06 22:56:55 +00:00
Rui Ueyama c6ea057d7f COFF: Move .idata constructor from Writer to Chunk.
Previously, half of the constructor for .idata contents was in Chunks.cpp
and the rest was in Writer.cpp. This patch moves the latter to Chunks.cpp.
Now IdataContents class manages everything for .idata section.

llvm-svn: 239230
2015-06-06 22:46:15 +00:00
Rui Ueyama 743afa0736 COFF: Merge Chunk::applyRelocations with Chunk::writeTo.
In this design, Chunk is the only thing that knows how to write
its contents to output file as well as how to apply relocations
there. The writer shouldn't know about the details.

llvm-svn: 239216
2015-06-06 04:07:39 +00:00
Peter Collingbourne ace2f091fd COFF: Read linker directives from bitcode files.
Differential Revision: http://reviews.llvm.org/D10285

llvm-svn: 239212
2015-06-06 02:00:45 +00:00
Rui Ueyama 8854d8a6f1 COFF: Add /failifmismatch option.
llvm-svn: 239073
2015-06-04 19:21:24 +00:00
Rui Ueyama 2ba7908cf9 Add comments.
llvm-svn: 239072
2015-06-04 19:21:22 +00:00
Rui Ueyama eb262ce4b6 COFF: /include'd symbols must be preserved.
Not only entry point symbol but also symbols specified by /include
option must be preserved, as they will never be dead-stripped.

http://reviews.llvm.org/D10220

llvm-svn: 239005
2015-06-04 02:12:16 +00:00
Rui Ueyama 2d7627198f Fix typo.
llvm-svn: 238937
2015-06-03 16:50:41 +00:00
Rui Ueyama bda72a4af4 COFF: Change OutputSections' type from vector<unique_ptr<T>> to vector<T*>.
This is mainly for readability. OutputSection objects are still owned
by the writer using SpecificBumpPtrAllocator.

llvm-svn: 238936
2015-06-03 16:44:00 +00:00
Rui Ueyama 652052b82c COFF: Update README.
Avoid saying this is based on sections because it's not very accurate.
That we don't split section into smaller chunks of data does not mean
that the linker is built on top of that.
In reality, most part of the code do not care about underlying data,
so they are neither based on "atoms" nor sections.
The symbol table only cares about symbol names and their types.
The writer handles list of chunks, which look like just blobs,
and the writer doesn't care what those chunks are backed by.
The only thing that interact with sections is SectionChunk, which is
abstracted away as one type of Chunk.

llvm-svn: 238902
2015-06-03 05:39:13 +00:00
Rui Ueyama 07e661f8cd COFF: SymbolTable to manage symbols using BumpPtrAllocator.
llvm-svn: 238901
2015-06-03 05:39:12 +00:00
Rui Ueyama 1db1ef9ab4 Use reinterpret_cast instead of const_cast and C-style cast.
llvm-svn: 238786
2015-06-01 21:49:21 +00:00
Rui Ueyama 81b030cbf6 COFF: Remove BitcodeFile::BitcodeFile(StringRef Filename).
In r238690, I made all files have only MemoryBufferRefs. This change
is to do the same thing for the bitcode file reader. Also updated
a few variable names to match with other code.

llvm-svn: 238782
2015-06-01 21:19:43 +00:00
Rui Ueyama fd99e01b91 COFF: Support import-by-ordinal DLL imports.
Symbols exported by DLLs can be imported not by name but by
small number or ordinal. Usually, symbols have both ordinals
and names, and in that case ordinals are called "hints" and
used by the loader as hints.

However, symbols can have only ordinals. They are called
import-by-ordinal symbols. You need to manage ordinals by hand
so that they will never change if you choose to use the feature.
But it's supposed to make dynamic linking faster because
it needs no string comparison. Not sure if that claim still
stands in year 2015, though. Anyways, the feature exists,
and this patch implements that.

llvm-svn: 238780
2015-06-01 21:05:27 +00:00
Rui Ueyama c2abdd9152 COFF: Use Chunk instead of its derived classes.
I'm adding ordinal-only (nameless) imports to the import table.
The chunk for that type is going to be different from LookupChunk.
Without this change, we cannot add objects of the new type to the
vectors.

llvm-svn: 238779
2015-06-01 21:05:24 +00:00
Peter Collingbourne 60c1616613 COFF: Initial implementation of link-time optimization.
This implementation is known to work in very simple cases (see new test case).

Differential Revision: http://reviews.llvm.org/D10115

llvm-svn: 238777
2015-06-01 20:10:10 +00:00
Denis Protivensky 6833690402 COFF: Fix warnings found by gcc
llvm-svn: 238734
2015-06-01 09:26:32 +00:00
Denis Protivensky 1c43df7481 COFF: Better noexcept specification with LLVM_NOEXCEPT
This is a follow-on to r238732

llvm-svn: 238733
2015-06-01 09:08:11 +00:00
Denis Protivensky c44223ed96 COFF: Add noexcept to std::error_category::name
This fixes build error with gcc.

llvm-svn: 238732
2015-06-01 08:12:44 +00:00
Rui Ueyama 5b25edddfe COFF: Fix the import table Hint/Name field.
llvm-svn: 238719
2015-06-01 03:55:04 +00:00
Rui Ueyama 68216c680d Fix comments.
llvm-svn: 238718
2015-06-01 03:55:02 +00:00
Rui Ueyama 78aefcb238 COFF: Fix /include. Included symbols are GC-roots.
llvm-svn: 238717
2015-06-01 03:42:54 +00:00
Rui Ueyama 8fd9fb9857 COFF: Define an error category for the linker.
Instead of returning non-categorized errors, return categorized errors.
All uses of make_dynamic_error_code are removed.

Because we don't have error reporting mechanism, I just chose to print out
error messages to stderr, and then return an error object. Not sure if
that's the right thing to do, but at least it seems practical.

http://reviews.llvm.org/D10129

llvm-svn: 238714
2015-06-01 02:58:15 +00:00
Rui Ueyama 360bace8eb COFF: Add /alternatename option.
Previously, this feature was implemented using a special type of
undefined symbol, in addition to an intricate way to make the resolver
read a virtual file containing that renaming symbols.

Now the feature is directly handled by the symbol table.
The symbol table has a function, rename(), to rename symbols, whose
definition is 4 lines long. Symbol renaming is naturally modeled using
Symbol and SymbolBody.

llvm-svn: 238696
2015-05-31 22:31:31 +00:00
Rui Ueyama 711cd2d7c8 COFF: Detect file type by file magic.
llvm-svn: 238691
2015-05-31 21:17:10 +00:00
Rui Ueyama d7c2f5847a COFF: Make the Driver own all MemoryBuffers. NFC.
Previously, a MemoryBuffer of a file was owned by each InputFile object.
This patch makes the Driver own all of them. InputFiles now have only
MemoryBufferRefs. This change simplifies ownership managment
(particularly for ObjectFile -- the object owned a MemoryBuffer only when
it's not created from an archive file, because in that case a parent
archive file owned the entire buffer. Now it owns nothing unconditionally.)

llvm-svn: 238690
2015-05-31 21:04:56 +00:00
Rui Ueyama f4784cce54 COFF: /libpath should not take precedence over the current directory.
llvm-svn: 238683
2015-05-31 20:20:37 +00:00
Rui Ueyama 0613747e1a COFF: Add /libpath option.
llvm-svn: 238682
2015-05-31 20:10:11 +00:00
Rui Ueyama e042fa9aa5 COFF: Add /include option.
It does not involve notions of virtual archives or virtual files,
nor store a list of undefined symbols somewhere else to consume them later.
We did that before. In this patch, undefined symbols are just added to
the symbol table, which now can be done in very few lines of code.

llvm-svn: 238681
2015-05-31 19:55:40 +00:00
Rui Ueyama d21b00bd7c COFF: Add /nodefaultlib option.
llvm-svn: 238679
2015-05-31 19:17:14 +00:00
Rui Ueyama 54b71daec4 COFF: Refactor functions to find files from search paths.
llvm-svn: 238678
2015-05-31 19:17:12 +00:00
Rui Ueyama a9cbbf885f COFF: Create LinkerDriver class.
Previously the main linker routine is just a non-member function.
We store some context information to the Config object.
This patch makes it belong to Driver.

llvm-svn: 238677
2015-05-31 19:17:09 +00:00
Rui Ueyama 80b5689d91 COFF: Use range-based for loop.
llvm-svn: 238675
2015-05-31 16:10:50 +00:00
Rui Ueyama d68ff34ad2 Fix unsafe memory access.
llvm-svn: 238669
2015-05-31 03:57:30 +00:00
Rui Ueyama 3ee0fe4c2c COFF: Implement subsystem inference.
llvm-svn: 238668
2015-05-31 03:55:46 +00:00
Rui Ueyama 5cff68599d COFF: Infer entry symbol name if /entry is not given.
`main` is not the only main function in Windows. You can choose one
from these four -- {w,}{WinMain,main}. There are four different entry
point functions for them, {w,}{WinMain,main}CRTStartup, respectively.
The linker needs to choose the right one depending on which `main`
function is defined.

llvm-svn: 238667
2015-05-31 03:34:08 +00:00
Rui Ueyama e00d651071 Use initializer instead of memset to zero out.
llvm-svn: 238662
2015-05-30 19:28:58 +00:00
Rui Ueyama bfb4aa1791 COFF: Support long section name.
Section names were truncated to 8 bytes because the section table's
name field is 8 byte long. This patch creates the string table to
store long names.

llvm-svn: 238661
2015-05-30 19:09:50 +00:00
Peter Collingbourne 246ccc5f51 COFF: Move machine type auto-detection to SymbolTable.
The new mechanism is less code, and fixes the case where all inputs
are archives.

Differential Revision: http://reviews.llvm.org/D10136

llvm-svn: 238618
2015-05-29 21:47:36 +00:00
Rui Ueyama 15cc47ee81 COFF: Add /subsystem option.
llvm-svn: 238571
2015-05-29 16:34:31 +00:00
Rui Ueyama b9dcdb5fc9 COFF: Add /version option.
llvm-svn: 238570
2015-05-29 16:28:29 +00:00
Rui Ueyama c377e9aefe COFF: Add /heap option.
llvm-svn: 238569
2015-05-29 16:23:40 +00:00
Rui Ueyama b41b7e5a69 Add /stack option.
llvm-svn: 238568
2015-05-29 16:21:11 +00:00
Rui Ueyama 804a8b6361 COFF: Add /base option.
llvm-svn: 238567
2015-05-29 16:18:15 +00:00
Rui Ueyama 5c726433d2 COFF: Add /help option.
llvm-svn: 238565
2015-05-29 16:11:52 +00:00
Rui Ueyama 3d3e6fba6e COFF: Add /machine option.
llvm-svn: 238564
2015-05-29 16:06:00 +00:00
Rui Ueyama 7c4fcdd559 COFF: Move Windows-specific function under Windows-specific marker.
llvm-svn: 238563
2015-05-29 15:49:09 +00:00
Rui Ueyama c9bfe32010 COFF: Fill imort table HintName field.
Currently we set the field to zero, but as per the spec, we should
set numbers we read from import library files. The loader uses the
values as starting offsets for binary search when looking up imported
symbols from DLL.

llvm-svn: 238562
2015-05-29 15:45:35 +00:00
Rui Ueyama 322b2c413d COFF: Return an error_code directly.
llvm-svn: 238486
2015-05-28 20:39:29 +00:00
Rui Ueyama 3500f6667a COFF: Split Driver.cpp to Driver.cpp and DriverUtils.cpp. NFC.
The previous implementation's driver file is cluttered by lots of
small functions, and it was hard to find important functions.
Make a separate file to prevent that issue.

llvm-svn: 238482
2015-05-28 20:30:06 +00:00
Rui Ueyama d52824d361 Rename InputFile::Name -> InputFile::Filename.
Other local variables shadowed the member variable.
Rename to make that a bit longer.

llvm-svn: 238478
2015-05-28 20:16:25 +00:00
Rui Ueyama 9aefa0c6b9 Fix non-debug build.
llvm-svn: 238474
2015-05-28 20:04:51 +00:00
Rui Ueyama d6fefba447 COFF: Teach Chunk to write to a mmap'ed output file.
Previously Writer directly handles writes to a file.
Chunks needed to give Writer a continuous chunk of memory.
That was inefficent if you construct data in chunks because
it would require two memory copies (one to construct a chunk
and the other is to write that to a file).

This patch teaches chunk to write directly to a file.
From readability point of view, this is also good because
you no longer have to call hasData() before calling getData().

llvm-svn: 238464
2015-05-28 19:45:43 +00:00
Rui Ueyama 411c636081 COFF: Add a new PE/COFF port.
This is an initial patch for a section-based COFF linker.

The patch has 2300 lines of code including comments and blank lines.
Before diving into details, you want to start from reading README
because it should give you an overview of the design.

All important things are written in the README file, so I write
summary here.

- The linker is already able to self-link on Windows.

- It's significantly faster than the existing implementation.
  The existing one takes 5 seconds to link LLD on my machine,
  while the new one only takes 1.2 seconds, even though the new
  one is not multi-threaded yet. (And a proof-of-concept multi-
  threaded version was able to link it in 0.5 seconds.)

- It uses much less memory (250MB vs. 2GB virtual memory space
  to self-host).

- IMHO the new code is much simpler and easier to read than
  the existing PE/COFF port.

http://reviews.llvm.org/D10036

llvm-svn: 238458
2015-05-28 19:09:30 +00:00