Original commit description:
[LLD] Remove global state in lld/COFF
This patch removes globals from the lldCOFF library, by moving globals
into a context class (COFFLinkingContext) and passing it around wherever
it's needed.
See https://lists.llvm.org/pipermail/llvm-dev/2021-June/151184.html for
context about removing globals from LLD.
I also haven't moved the `driver` or `config` variables yet.
Differential Revision: https://reviews.llvm.org/D109634
This reverts commit a2fd05ada9.
Original commits were b4fa71eed3
and e03c7e367a.
This test checks that timers are working and printing as expected.
I also seem to have changed the order of the timers in my globals refactoring
patch, so I fixed it here.
Differential Revision: https://reviews.llvm.org/D109904
This patch removes globals from the lldCOFF library, by moving globals
into a context class (COFFLinkingContext) and passing it around wherever
it's needed.
See https://lists.llvm.org/pipermail/llvm-dev/2021-June/151184.html for
context about removing globals from LLD.
I also haven't moved the `driver` or `config` variables yet.
Differential Revision: https://reviews.llvm.org/D109634
We currently complain "could not open /LTCG: no such file or directory",
which isn't very useful. We could emit a warning when we see this flag, but
just ignoring it seems fine.
Final missing part of PR38799.
Differential Revision: https://reviews.llvm.org/D108799
If multiple /manifestdependency: flags are passed, they are
naively deduped, but after that each of them should have an
effect, instead of just the last one.
Also, /manifestdependency: flags are allowed in .drectve sections
(from `#pragma comment(linker, ...`). To make the interaction between
/manifestdependency: flags enabling manifest by default but
/manifest:no overriding this work, add an explict ManifestKind::Default
state to represent no explicit /manifest flag being passed.
To make /manifestdependency: flags from input file .drectve sections
work with /manifest:embed, delay embedded manifest emission until
after input files have been read.
Differential Revision: https://reviews.llvm.org/D108628
When enable CSPGO for ThinLTO, there are profile cfg mismatch warnings that will cause lld-link errors (with /WX)
due to source changes (e.g. `#if` code runs for profile generation but not for profile use)
To disable it we have to use an internal "/mllvm:-no-pgo-warn-mismatch" option.
In contrast clang uses option ”-Wno-backend-plugin“ to avoid such warnings and gcc has an explicit "-Wno-coverage-mismatch" option.
Add "lto-pgo-warn-mismatch" option to lld COFF/ELF to help turn on/off the profile mismatch warnings explicitly when build with ThinLTO and CSPGO.
Differential Revision: https://reviews.llvm.org/D104431
When enable CSPGO for ThinLTO, there are profile cfg mismatch warnings that will cause lld-link errors (with /WX).
To disable it we have to use an internal "/mllvm:-no-pgo-warn-mismatch" option.
In contrast clang uses option ”-Wno-backend-plugin“ to avoid such warnings and gcc has an explicit "-Wno-coverage-mismatch" option.
Add this "lto-pgo-warn-mismatch" option to lld to help turn on/off the profile mismatch warnings explicitly when build with ThinLTO and CSPGO.
Reviewed By: tejohnson
Differential Revision: https://reviews.llvm.org/D104431
In PGO, a C++ external linkage function `foo` has a private counter
`__profc_foo` and a private `__profd_foo` in a `comdat nodeduplicate`.
A `__attribute__((weak))` function `foo` has a weak hidden counter `__profc_foo`
and a private `__profd_foo` in a `comdat nodeduplicate`.
In `ld.lld a.o b.o`, say a.o defines an external linkage `foo` and b.o
defines a weak `foo`. Currently we treat `comdat nodeduplicate` as `comdat any`,
ld.lld will incorrectly consider `b.o:__profc_foo` non-prevailing. In the worst
case when `b.o:__profd_foo` is retained and `b.o:__profc_foo` isn't, there will
be dangling reference causing an `undefined hidden symbol` error.
Add SelectionKind to `Comdat` in IRSymtab and let linkers ignore nodeduplicate comdat.
Differential Revision: https://reviews.llvm.org/D106228
C++23 will make these conversions ambiguous - so fix them to make the
codebase forward-compatible with C++23 (& a follow-up change I've made
will make this ambiguous/invalid even in <C++23 so we don't regress
this & it generally improves the code anyway)
LLD on 32-bit Windows would frequently fail on large projects with
an exception "thread constructor failed: Exec format error". The stack
trace pointed to this usage of std::async, and looking at the
implementation in libc++ it seems using std::async with
std::launch::async results in the immediate creation of a new thread
for every call. This could result in a potentially unbounded number
of threads, depending on the number of input files. This seems to
be hitting some limit in 32-bit Windows host.
I took the easy route, and only use threads on 64-bit Windows, not all
Windows as before. I was thinking a more proper solution might
involve using a thread pool rather than blindly spawning any number
of new threads, but that may have other unforeseen consequences.
Reviewed By: rnk
Differential Revision: https://reviews.llvm.org/D105506
If linking directly against a DLL without an import library, the
DLL export symbols might not contain stdcall decorations.
If we have an undefined symbol with decoration, and we happen to have
a matching undecorated symbol (which either is lazy and can be loaded,
or already defined), then alias it against that instead.
This matches what's done in reverse, when we have a def file
declaring to export a symbol without decoration, but we only have
a defined decorated symbol. In that case we do a fuzzy match
(SymbolTable::findMangle). This case is more straightforward; if we
have a decorated undefined symbol, just strip the decoration and look
for the corresponding undecorated symbol name.
Add warnings and options for either silencing the warning or disabling
the whole feature, corresponding to how ld.bfd does it.
(This feature works for any symbol decoration mismatch, not only when
linking against a DLL directly; ld.bfd also tolerates it anywhere,
and also fixes up mismatches in the other direction, like
SymbolTable::findMangle, for any symbol, not only exports. But in
practice, at least for lld, it would primarily end up used for linking
against DLLs.)
Differential Revision: https://reviews.llvm.org/D104532
GNU ld.bfd supports linking directly against DLLs without using an
import library, and some projects have picked up on this habit.
(There's no one single unsurmountable issue with using import
libraries, but this is a regularly surfacing missing feature.)
As long as one is linking by name (instead of by ordinal), the DLL
export table contains most of the information needed. (One can
inspect what section a symbol points at, to see if it's a function
or data symbol. The practical implementation of this loops over all
sections for each symbol, but as long as they're not very many, that
should hopefully be tolerable performance wise.)
One exception where the information in the DLL isn't entirely enough
is on i386 with stdcall functions; depending on how they're done,
the exported function name can be a plain undecorated name, while
the import library would contain the full decorated symbol name. This
issue is addressed separately in a different patch.
This is implemented mimicing the structure of a regular import library,
with one InputFile corresponding to the static archive that just adds
lazy symbols, which then are fetched when they are needed. When such
a symbol is fetched, we synthesize a coff_import_header structure
in memory and create a regular ImportFile out of it.
The implementation could be even smaller by just creating ImportFiles
for every symbol available immediately, but that would have the
drawback of actually ending up importing all symbols unless running
with GC enabled (and mingw mode defaults to having it disabled for
historical reasons).
Differential Revision: https://reviews.llvm.org/D104530
Commit 728cc0075e made comdat symbols
from LTO objects be treated as any regular comdat symbol. This works
great for symbols that actually are IMAGE_COMDAT_SELECT_ANY, but
if the symbols have a less trivial selection type that require comparing
either the section chunk size or contents, we can't check that before
actually doing the LTO compilation.
Therefore bring back one aspect of handling from before; that comdat
resolution with a leader from an LTO symbol is essentially skipped,
like it was before 728cc0075e.
Differential Revision: https://reviews.llvm.org/D104605
This reverts commit e1adf90826.
This appears to affect the way that C++ mangled symbols appear in the
import library when using a .def file that names a C++ free function
with no name decoration. I will follow up with a reduced test case
shortly.
This is run every time around in the main linker loop. Once a match
has been found, stop trying to rematch such a symbol.
Not sure if this has any actual measurable performance impact though
(SymbolTable::findMangle() iterates over the whole symbol table for
each call and does fuzzy matching on top of that) but this makes the
code more reassuring to read at least. (This is in practice run for def
files listing undecorated stdcall functions to be exported.)
Differential Revision: https://reviews.llvm.org/D104529
The following class isn't part of the export table; there's a
second correctly placed comment about the things that actually
belong to the export table.
Make sure that comdat symbols also have a non-null dummy
SectionChunk associated.
This requires moving around an existing FIXME regarding comdats in
LTO.
Differential Revision: https://reviews.llvm.org/D103012
Before this patch, the maximum size of the GHASH table was 2^31 buckets. However we were storing the bucket index into a TypeIndex which has an input limit of (2^31)-4095 indices, see this link. Any value above that limit will improperly set the TypeIndex's high bit, which is interpreted as DecoratedItemIdMask. This used to cause bad indices on extraction when calling TypeIndex::toArrayIndex().
Differential Revision: https://reviews.llvm.org/D103297
Ghashing is probably going to be faster in most cases, even without
precomputed ghashes in object files.
Here is my table of results linking clang.pdb:
-------------------------------
| threads | GHASH | NOGHASH |
-------------------------------
| j1 | 51.031s | 25.141s |
| j2 | 31.079s | 22.109s |
| j4 | 18.609s | 23.156s |
| j8 | 11.938s | 21.984s |
| j28 | 8.375s | 18.391s |
-------------------------------
This shows that ghashing is faster if at least four cores are available.
This may make the linker slower if most cores are busy in the middle of
a build, but in that case, the linker probably isn't on the critical
path of the build. Incremental build performance is arguably more
important than highly contended batch build link performance.
The -time output indicates that ghash computation is the dominant
factor:
Input File Reading: 924 ms ( 1.8%)
GC: 689 ms ( 1.3%)
ICF: 527 ms ( 1.0%)
Code Layout: 414 ms ( 0.8%)
Commit Output File: 24 ms ( 0.0%)
PDB Emission (Cumulative): 49938 ms ( 94.8%)
Add Objects: 46783 ms ( 88.8%)
Global Type Hashing: 38983 ms ( 74.0%)
GHash Type Merging: 5640 ms ( 10.7%)
Symbol Merging: 2154 ms ( 4.1%)
Publics Stream Layout: 188 ms ( 0.4%)
TPI Stream Layout: 18 ms ( 0.0%)
Commit to Disk: 2818 ms ( 5.4%)
--------------------------------------------------
Total Link Time: 52669 ms (100.0%)
We can speed that up with a faster content hash (not SHA1).
Differential Revision: https://reviews.llvm.org/D102888
The COFF driver produces an ABSOLUTE relocation base for an ADDR32
relocation type and the system is 64 bits (machine=AMD64). The
relocation information won't be added in the output and could
produce an incorrect address access during run-time. This change
set checks if the relocation type is IMAGE_REL_AMD64_ADDR32 and
if so, adds the relocated symbol as IMAGE_REL_BASED_HIGHLOW base.
Differential Revision: https://reviews.llvm.org/D96619
Previously we simply didn't check this. Prereq to make the test suite
pass with ghash enabled by default.
Differential Revision: https://reviews.llvm.org/D102885
These symbols are long, and they tend to cause the PDB file size to
overflow. They are generally not necessary when debugging problems in
user code.
This change reduces the size of chrome.dll.pdb with coverage from
6,937,108,480 bytes to 4,690,210,816 bytes.
Differential Revision: https://reviews.llvm.org/D102719
Handle PDB writing errors like any other error in LLD: emit an error and
continue. This allows the linker to print timing data and summary data
after linking, which can be helpful for finding PDB size problems. Also
report how large the file would have been.
Example output:
lld-link: error: Output data is larger than 4 GiB. File size would have been 6,937,108,480
lld-link: error: failed to write PDB file ./chrome.dll.pdb
Summary
--------------------------------------------------------------------------------
33282 Input OBJ files (expanded from all cmd-line inputs)
4 PDB type server dependencies
0 Precomp OBJ dependencies
33396931 Input type records
... snip ...
Input File Reading: 59756 ms ( 45.5%)
GC: 7500 ms ( 5.7%)
ICF: 3336 ms ( 2.5%)
Code Layout: 6329 ms ( 4.8%)
PDB Emission (Cumulative): 46192 ms ( 35.2%)
Add Objects: 27609 ms ( 21.0%)
Type Merging: 16740 ms ( 12.8%)
Symbol Merging: 10761 ms ( 8.2%)
Publics Stream Layout: 9383 ms ( 7.1%)
TPI Stream Layout: 1678 ms ( 1.3%)
Commit to Disk: 3461 ms ( 2.6%)
--------------------------------------------------
Total Link Time: 131244 ms (100.0%)
Differential Revision: https://reviews.llvm.org/D102713
LLD already produces a nice error message when sections exceed 4GB, and
this setRVA assertion causes LLD to crash instead of diagnosing the
error properly.
No test because we don't want slow tests that create 4GB files.
Since c579a5b1d9 we don't traverse
.eh_frame when doing GC. But the exception handling personality
function needs to be included, and is only referenced from within
.eh_frame.
Differential Revision: https://reviews.llvm.org/D102138
As this isn't handled as a regular relocation, the normal handling of
maybeReportRelocationToDiscarded in Chunks.cpp doesn't apply here.
This would have caught the issue fixed by
82de4e0753.
Differential Revision: https://reviews.llvm.org/D102115
The Halide project uses `#pragma comment(linker, "/STACK:...")` to set
the stack size high enough for our embedded compiler to run in end-user
programs on Windows.
Unfortunately, lld-link.exe breaks on this when embedded in a COFF
object, despite supporting the flag on the command line. MSVC's link.exe
supports this fine. This patch extends support for this to lld-link.exe
for better compatibility with MSVC projects.
Differential Revision: https://reviews.llvm.org/D99680
This is a followup to 2b01a417d7ccb001ccc1185ef5fdc967c9fac8d7;
previously the RVAs of the exported symbols from comdats were left
zero.
Thanks to Kleis Auke Wolthuizen for the fix suggestion and pointing
out the omission.
Differential Revision: https://reviews.llvm.org/D101615
When looking for the "all" symbols that are supposed to be exported,
we can't look at the live flag - the symbols we mark as to be
exported will become GC roots even if they aren't yet marked as live.
With this in place, building an LLVM library with BUILD_SHARED_LIBS
produces the same set of symbols exported regardless of whether the
--gc-sections flag is specified, both with and without being built
with -ffunction-sections.
Differential Revision: https://reviews.llvm.org/D101522
This is a different approach from D98993 that should achieve most of the
same benefit. The two changes are:
1. Sort the list of associated child sections by section name
2. Do not consider associated sections to have children themselves
This fixes the main issue, which was that we sometimes considered an
.xdata section to have a child .pdata section. That lead to slow links
and larger binaries (less xdata folding).
Otherwise, this should be NFC: we go back to ignoring .debug/.gljmp and
other metadata sections rather than only looking at pdata/xdata. We
discovered that we do care about other associated sections, like ASan
global registration metadata.
Problem:
On SystemZ we need to open text files in text mode. On Windows, files opened in text mode adds a CRLF '\r\n' which may not be desirable.
Solution:
This patch adds two new flags
- OF_CRLF which indicates that CRLF translation is used.
- OF_TextWithCRLF = OF_Text | OF_CRLF indicates that the file is text and uses CRLF translation.
Developers should now use either the OF_Text or OF_TextWithCRLF for text files and OF_None for binary files. If the developer doesn't want carriage returns on Windows, they should use OF_Text, if they do want carriage returns on Windows, they should use OF_TextWithCRLF.
So this is the behaviour per platform with my patch:
z/OS:
OF_None: open in binary mode
OF_Text : open in text mode
OF_TextWithCRLF: open in text mode
Windows:
OF_None: open file with no carriage return
OF_Text: open file with no carriage return
OF_TextWithCRLF: open file with carriage return
The Major change is in llvm/lib/Support/Windows/Path.inc to only set text mode if the OF_CRLF is set.
```
if (Flags & OF_CRLF)
CrtOpenFlags |= _O_TEXT;
```
These following files are the ones that still use OF_Text which I left unchanged. I modified all these except raw_ostream.cpp in recent patches so I know these were previously in Binary mode on Windows.
./llvm/lib/Support/raw_ostream.cpp
./llvm/lib/TableGen/Main.cpp
./llvm/tools/dsymutil/DwarfLinkerForBinary.cpp
./llvm/unittests/Support/Path.cpp
./clang/lib/StaticAnalyzer/Core/HTMLDiagnostics.cpp
./clang/lib/Frontend/CompilerInstance.cpp
./clang/lib/Driver/Driver.cpp
./clang/lib/Driver/ToolChains/Clang.cpp
Reviewed By: MaskRay
Differential Revision: https://reviews.llvm.org/D99426
Amy Huang